critical analysis for your literature review · what is the purpose of a lit. rev.? •a literature...

META ANALISIS Rosseni Din

1

2

SG-Meta-Analysis-booklet-with-tables-and-references-1.pdf

SG-Meta-Analysis-booklet-with-tables-and-references-1.pdf

7

CRITICAL

REVIEW

Critical analysis for your

literature review Serge Gabarre

Wacana Siswazah 7.03.15

8 / 56

Colour codes

• Presentation

• Samples

• Practice

9 / 56

What is the purpose of a lit. rev.?

• A literature review seeks to describe the current

knowledge pertaining to a particular domain.

• Ensure that your research is novel and relevant to the

field of study.

• In all cases the researcher follows three steps

1. Search for articles (books, chapters in book, reports…)

2. Read the articles

3. Decide to keep or reject each article

10 / 56

The two types of literature reviews

• 1. A review can be conducted linearly by first dividing the

main topic into different segments and by addressing

each issue sequentially.

• Problems: This narrative review of the literature could

be subjected to bias from the researcher as a selection

of articles is always a personal interpretation of

relevance (Guzzo et al. 1987).

11 / 56

The two types of literature reviews

• 2. A critical review / meta-analysis may be used to

evaluate each article using specific criteria.

• Advantages: It offers undeniable advantages as it

grants social science researchers a clear view of all

research findings in their specific area based on specific

criteria (Rosenthal & DiMatteo 2001)

• Advantages: Conducting a critical review incites

researchers to adopt a rigorous survey and review of all

available documents.

12 / 56

Conducting a critical review

CRITICAL

=

CRITERIA

13 / 56

Some criteria you may use

• What are the main questions and problems that have

been addressed so far?

• What are the origins and definitions of the topic?

• What are the political standpoints?

• What are the key sources?

• What are the key theories concepts and ideas?

• What are the major issues and debate about the topic?

• What are the epistemological and ontological grounds for

the discipline?

• How is knowledge on the topic structured and organised?

14 / 56

How is a critical review conducted ?

• Guzzo et al. (1987) as well as Egger et al. (1997)

recommended approaching the task of conducting a

critical review in the same fashion as a quantitative study.

15 / 56

The three steps of conducting a study

• According to Egger et al. (1997)

1. “Formulation of the problem to be addressed,

2. Collection and analysis of the data, and

3. Reporting of the results” (1533)

16 / 56

Step 1: Formulation of the problem to be

addressed • The researcher should clearly mention the objectives of

the study.

• Formulate a research question

17 / 56

Sample RQ

• Purpose of the study:

The primary reason for conducting the present critical

review is to investigate the extent of the current

knowledge on learning a foreign language with social

networking sites on mobile phones.

• RQ:

To what extent do recent and respected publications deal

with the theme of learning a foreign language with social

networking sites and mobile devices?

18 / 56


addressed • What is the RQ for your critical review?

19 / 56


addressed • Variables need to be extracted from the RQ

20 / 56

Extracting variables

• To what extent do recent and respected publications deal

with the theme of learning a foreign language with social

networking sites and mobile devices?

1. Recent publications

2. Respected publications

3. Learning a foreign language

4. Learning with social networking sites

5. Learning with mobile devices

21 / 56

Extracting variables

• What are the variables which may be extracted from your

RQ?

22 / 56

Step 2: Collection and analysis of the data

• All reviews of the literature should be exhaustive and at

the same time “deal with a bounded domain of studies”

(Guzzo et al. 1987: 418).

• A thorough search of all pertinent articles must be

conducted before these are subsequently selected for

relevance.

23 / 56

Sample collection of the data

• Articles were also sourced from seven publishers’

website:

• Elsevier

• Emerald

• Routledge

• Sage

• Springer

• Taylor & Francis

• Wiley

24 / 56


• Keywords related to the theme of SNS, m-Learning, and

language learning with technology, were used to find

articles on four online services:

1. the ERIC database

2. the EBSCO server

3. the JSTOR service

4. the Google Scholar website

25 / 56


• When a relevant article was located in a journal, archival

issues were systematically searched to identify additional

articles.

26 / 56


• Looking at the list of references in each article.

• For Jacso (2005), these cited references represent more

than a part of the article as they reveal well-structured

connections with other scholarly papers.

27 / 56


• The use of online hyperlinks to citing articles also yielded

more relevant references.

28 / 56

Analysis of the data

• Slavin (1984) explained that in order to conduct a rigorous

critical review, only articles of a good quality should be

included.

• Slavin further explained that through a “garbage in-

garbage out” (1984: 6) process, including studies of a

lesser quality would reduce the worthiness of the critical

review.

29 / 56

Analysis of the data

• It is necessary to apply several excluding filters in order to

identify the most relevant documents

30 / 56

Sample analysis of the data

• Seven excluding filters were applied in order to obtain

more relevant documents.

• These inclusions and exclusions were adapted from Wu

et al.’s (2012) publication.

31 / 56


• Excluding filter 1

• First, all publications prior to the year 2010 were

excluded.

• Klavans and Boyack (2007) noted that the time frame

used to analyse new scientific literature should reflect the

stability of science in that particular field. A long time

frame assumes that science is stable, whereas a short

one presupposes that the field is rapidly evolving.

32 / 56



• Second, all publications which did not explicitly describe a

peer review process were removed from the selection.

• This decision was based on Jefferson et al.’s (2002)

consideration that scientific knowledge is deemed fit only

after having been reviewed by peer researchers.

• Consequently, this second filter excluded all conference

proceedings and chapters in books.

33 / 56



• Third, studies which were not in relation to education were excluded.

• This criterion was used to exclude all articles which evaluated the impact of either mobile phones or SNSs on society without taking into account the element of learning.

• Using this filter enabled the critical review to be centred solely on education without being diverted from the focal point of this study.

34 / 56



• Fourth, studies which only featured language learning and

technology without investigating either m-Learning or

learning with SNSs were excluded.

• Computer assisted language learning (CALL) is not a new

phenomenon and its impact on education has been

extensively researched as attested by Zhao (2003).

35 / 56



• Fifth, studies which only described the users or the

frequency of usage were excluded.

• With this criterion, studies which reported on analyses of

students’ profiles were not included in the meta-analysis.

• Even though some such studies occurred in universities,

they did not bring new findings on teaching or learning

applications.

36 / 56



• Sixth, studies which were not published in journals

referenced by the Institute for Scientific Information (ISI)

or Elsevier’s Scopus were excluded.

• This decision was taken to ensure a constant level of

quality in the articles. Although Google Scholar offers a

convenient way to access online papers, the quality of the

results remains inconsistent, as noted by Falagas et al.

(2008).

37 / 56



• Seventh, articles which reported on other studies were

rejected.

• This filter was used to remove all articles which dealt with

reviews of the literature, and did not include new empirical

findings.

38 / 56


• Applying the exclusion filters reduced the number of

articles. The literature review became more manageable

• Several hundred articles ► 59

39 / 56

Reporting of the results

• The selected articles need to be organised based on the

variables which were extracted from the RQ.

• Reminder:






40 / 56


• Construct a table with

• the variables on x axis, and

• the articles on the y axis

41 / 56







42 / 56







43 / 56







44 / 56


• Additional variables may be added

• Data should be sorted according to the most relevant

variables

45 / 56


46 / 56


47 / 56


48 / 56


49 / 56


• Present any gap which may visually appear in your tables.

50 / 56


• The first finding that is reported in this critical review relates to the lack of articles dealing with all three aspects of the current research: mobile learning, SNS-Learning, and language learning.

• This clearly represents a gap in the current knowledge of learning a foreign language with these two technologies.

51 / 56


• Hardly any article dealing with

SNS-learning dealt with

language learning

• This finding highlights a

second knowledge gap which

strongly warrants additional

research in this field.

52 / 56


• Use descriptive statistics to report the findings of the

critical review.

53 / 56


• Articles were almost equally distributed between these

two technologies with research on m-Learning accounting

for 53% of all academic journals, while those on SNSs

accounted for 47%.

• Unexpectedly, a large majority (93%) of articles which

reported on language learning also dealt with mobile

learning.

54 / 56


• The Asian continent figures prominently among the

articles selected for the meta-analysis with 52% of all

publications where a geographical setting was identified.

• On the other hand, Northern America (20%) and Europe

(18%) account together for just over a third of all articles.

55 / 56


• Additionally, the analysis of the literature revealed a

scarcity of articles reporting findings from research

conducted in Malaysia. Indeed, from a total of 59 articles,

only two (3%) reported on the Malaysian context.

• As such, this finding represents a third gap in the

literature.

56 / 56


• Describe the implications of your findings from the critical

review.

57 / 56


• The main implication that resulted from the critical review

was the discovery of three gaps in the literature.

• Although a single knowledge gap would be sufficient

justification for conducting a research, the accumulation of

three strengthens the need for additional investigations.

58 / 56


• Having identified a manageable quantity of relevant

articles, it is possible to report on them without any bias.

59 / 56


60 / 56


61 / 56

THANK YOU Q&A

62 / 56

Application

1. Look through your articles and identify five as relevant.

2. Analyse them according to the criteria you have

identified

3. Present your findings

63 / 56

References

• Egger, M., Smith, G. D., & Phillips, A. N. 1997. Meta-analysis: Principles and procedures. British Medical Journal. 315(7121): 1533-1537.

• Falagas, M. E., Pitsouni, E. I., Malietzis, G. A., & Pappas, G. 2008. Comparison of PubMed, Scopus, Web of Science, and Google Scholar: Strengths and weaknesses. Federation of American Societies for Experimental Biology Journal. 22: 338-342.

• Guzzo, R. A., Jackson, S. E., & Katzell, R. A. 1987. Meta-analysis analysis. Research in Organizational Behavior. 9: 407-442.

• Jacso, P. 2005. As we may search – Comparison of major features of the Web of Science, Scopus, and Google Scholar citation-based and citation-enhanced databases. Current Science. 89(9): 1537-1547. http://cs-test.ias.ac.in/cs/index.php [5 May 2012].

• Jefferson, T., Wager, E., & Davidoff, F. 2002. Measuring the quality of editorial peer review. Journal of the American Medical Association. 287(21): 2786-2790.

64

References

• Klavans, R., & Boyack, K. 2007. Is there a convergent structure of science? A comparison of maps using the ISI and Scopus databases. Proceedings of the 11th International Conference of the International Society for Scientometrics and Informetrics.

• Rosenthal, R., & DiMatteo, M. R. 2001. Meta-analysis: Recent developments in quantitative methods for literature reviews. Annual Review of Psychology. 52(1): 59-82.

• Slavin, R. E. 1984. Meta-analysis in education: How has it been used? Educational Researcher. 13(8): 6-15.

• Wu, W.-H., Wu, Y.-C. J., Chen, C.-Y., Kao, H.-Y., Lin, C.-H., & Huang, S.-H. 2012. Review of trends from mobile learning studies: A meta-analysis. Computers & Education. 59: 817-827.

• Zhao, Y. 2003. Recent developments in technology and language learning: A literature review and meta-analysis. CALICO Journal. 21(1): 7-27. https://calico.org/journalTOC.php?current=1 [23 March 2012].

65

critical analysis for your literature review · what is the purpose of a lit. rev.? •a literature...

Documents