META ANALISIS Rosseni Din
1
2
3
4
5
6
7
CRITICAL
REVIEW
Critical analysis for your
literature review Serge Gabarre
Wacana Siswazah 7.03.15
8 / 56
Colour codes
• Presentation
• Samples
• Practice
9 / 56
What is the purpose of a lit. rev.?
• A literature review seeks to describe the current
knowledge pertaining to a particular domain.
• Ensure that your research is novel and relevant to the
field of study.
• In all cases the researcher follows three steps
1. Search for articles (books, chapters in book, reports…)
2. Read the articles
3. Decide to keep or reject each article
10 / 56
The two types of literature reviews
• 1. A review can be conducted linearly by first dividing the
main topic into different segments and by addressing
each issue sequentially.
• Problems: This narrative review of the literature could
be subjected to bias from the researcher as a selection
of articles is always a personal interpretation of
relevance (Guzzo et al. 1987).
11 / 56
The two types of literature reviews
• 2. A critical review / meta-analysis may be used to
evaluate each article using specific criteria.
• Advantages: It offers undeniable advantages as it
grants social science researchers a clear view of all
research findings in their specific area based on specific
criteria (Rosenthal & DiMatteo 2001)
• Advantages: Conducting a critical review incites
researchers to adopt a rigorous survey and review of all
available documents.
12 / 56
Conducting a critical review
CRITICAL
=
CRITERIA
13 / 56
Some criteria you may use
• What are the main questions and problems that have
been addressed so far?
• What are the origins and definitions of the topic?
• What are the political standpoints?
• What are the key sources?
• What are the key theories concepts and ideas?
• What are the major issues and debate about the topic?
• What are the epistemological and ontological grounds for
the discipline?
• How is knowledge on the topic structured and organised?
14 / 56
How is a critical review conducted ?
• Guzzo et al. (1987) as well as Egger et al. (1997)
recommended approaching the task of conducting a
critical review in the same fashion as a quantitative study.
15 / 56
The three steps of conducting a study
• According to Egger et al. (1997)
1. “Formulation of the problem to be addressed,
2. Collection and analysis of the data, and
3. Reporting of the results” (1533)
16 / 56
Step 1: Formulation of the problem to be
addressed • The researcher should clearly mention the objectives of
the study.
• Formulate a research question
17 / 56
Sample RQ
• Purpose of the study:
The primary reason for conducting the present critical
review is to investigate the extent of the current
knowledge on learning a foreign language with social
networking sites on mobile phones.
• RQ:
To what extent do recent and respected publications deal
with the theme of learning a foreign language with social
networking sites and mobile devices?
18 / 56
Step 1: Formulation of the problem to be
addressed • What is the RQ for your critical review?
19 / 56
Step 1: Formulation of the problem to be
addressed • Variables need to be extracted from the RQ
20 / 56
Extracting variables
• To what extent do recent and respected publications deal
with the theme of learning a foreign language with social
networking sites and mobile devices?
1. Recent publications
2. Respected publications
3. Learning a foreign language
4. Learning with social networking sites
5. Learning with mobile devices
21 / 56
Extracting variables
• What are the variables which may be extracted from your
RQ?
22 / 56
Step 2: Collection and analysis of the data
• All reviews of the literature should be exhaustive and at
the same time “deal with a bounded domain of studies”
(Guzzo et al. 1987: 418).
• A thorough search of all pertinent articles must be
conducted before these are subsequently selected for
relevance.
23 / 56
Sample collection of the data
• Articles were also sourced from seven publishers’
website:
• Elsevier
• Emerald
• Routledge
• Sage
• Springer
• Taylor & Francis
• Wiley
24 / 56
Sample collection of the data
• Keywords related to the theme of SNS, m-Learning, and
language learning with technology, were used to find
articles on four online services:
1. the ERIC database
2. the EBSCO server
3. the JSTOR service
4. the Google Scholar website
25 / 56
Sample collection of the data
• When a relevant article was located in a journal, archival
issues were systematically searched to identify additional
articles.
26 / 56
Sample collection of the data
• Looking at the list of references in each article.
• For Jacso (2005), these cited references represent more
than a part of the article as they reveal well-structured
connections with other scholarly papers.
27 / 56
Sample collection of the data
• The use of online hyperlinks to citing articles also yielded
more relevant references.
28 / 56
Analysis of the data
• Slavin (1984) explained that in order to conduct a rigorous
critical review, only articles of a good quality should be
included.
• Slavin further explained that through a “garbage in-
garbage out” (1984: 6) process, including studies of a
lesser quality would reduce the worthiness of the critical
review.
29 / 56
Analysis of the data
• It is necessary to apply several excluding filters in order to
identify the most relevant documents
30 / 56
Sample analysis of the data
• Seven excluding filters were applied in order to obtain
more relevant documents.
• These inclusions and exclusions were adapted from Wu
et al.’s (2012) publication.
31 / 56
Sample analysis of the data
• Excluding filter 1
• First, all publications prior to the year 2010 were
excluded.
• Klavans and Boyack (2007) noted that the time frame
used to analyse new scientific literature should reflect the
stability of science in that particular field. A long time
frame assumes that science is stable, whereas a short
one presupposes that the field is rapidly evolving.
32 / 56
Sample analysis of the data
• Excluding filter 2
• Second, all publications which did not explicitly describe a
peer review process were removed from the selection.
• This decision was based on Jefferson et al.’s (2002)
consideration that scientific knowledge is deemed fit only
after having been reviewed by peer researchers.
• Consequently, this second filter excluded all conference
proceedings and chapters in books.
33 / 56
Sample analysis of the data
• Excluding filter 3
• Third, studies which were not in relation to education were excluded.
• This criterion was used to exclude all articles which evaluated the impact of either mobile phones or SNSs on society without taking into account the element of learning.
• Using this filter enabled the critical review to be centred solely on education without being diverted from the focal point of this study.
34 / 56
Sample analysis of the data
• Excluding filter 4
• Fourth, studies which only featured language learning and
technology without investigating either m-Learning or
learning with SNSs were excluded.
• Computer assisted language learning (CALL) is not a new
phenomenon and its impact on education has been
extensively researched as attested by Zhao (2003).
35 / 56
Sample analysis of the data
• Excluding filter 5
• Fifth, studies which only described the users or the
frequency of usage were excluded.
• With this criterion, studies which reported on analyses of
students’ profiles were not included in the meta-analysis.
• Even though some such studies occurred in universities,
they did not bring new findings on teaching or learning
applications.
36 / 56
Sample analysis of the data
• Excluding filter 6
• Sixth, studies which were not published in journals
referenced by the Institute for Scientific Information (ISI)
or Elsevier’s Scopus were excluded.
• This decision was taken to ensure a constant level of
quality in the articles. Although Google Scholar offers a
convenient way to access online papers, the quality of the
results remains inconsistent, as noted by Falagas et al.
(2008).
37 / 56
Sample analysis of the data
• Excluding filter 7
• Seventh, articles which reported on other studies were
rejected.
• This filter was used to remove all articles which dealt with
reviews of the literature, and did not include new empirical
findings.
38 / 56
Sample analysis of the data
• Applying the exclusion filters reduced the number of
articles. The literature review became more manageable
• Several hundred articles ► 59
39 / 56
Reporting of the results
• The selected articles need to be organised based on the
variables which were extracted from the RQ.
• Reminder:
1. Recent publications
2. Respected publications
3. Learning a foreign language
4. Learning with social networking sites
5. Learning with mobile devices
40 / 56
Reporting of the results
• Construct a table with
• the variables on x axis, and
• the articles on the y axis
41 / 56
Reporting of the results
1. Recent publications
2. Respected publications
3. Learning a foreign language
4. Learning with social networking sites
5. Learning with mobile devices
42 / 56
Reporting of the results
1. Recent publications
2. Respected publications
3. Learning a foreign language
4. Learning with social networking sites
5. Learning with mobile devices
43 / 56
Reporting of the results
1. Recent publications
2. Respected publications
3. Learning a foreign language
4. Learning with social networking sites
5. Learning with mobile devices
44 / 56
Reporting of the results
• Additional variables may be added
• Data should be sorted according to the most relevant
variables
45 / 56
Reporting of the results
46 / 56
Reporting of the results
47 / 56
Reporting of the results
48 / 56
Reporting of the results
49 / 56
Reporting of the results
• Present any gap which may visually appear in your tables.
50 / 56
Reporting of the results
• The first finding that is reported in this critical review relates to the lack of articles dealing with all three aspects of the current research: mobile learning, SNS-Learning, and language learning.
• This clearly represents a gap in the current knowledge of learning a foreign language with these two technologies.
51 / 56
Reporting of the results
• Hardly any article dealing with
SNS-learning dealt with
language learning
• This finding highlights a
second knowledge gap which
strongly warrants additional
research in this field.
52 / 56
Reporting of the results
• Use descriptive statistics to report the findings of the
critical review.
53 / 56
Reporting of the results
• Articles were almost equally distributed between these
two technologies with research on m-Learning accounting
for 53% of all academic journals, while those on SNSs
accounted for 47%.
• Unexpectedly, a large majority (93%) of articles which
reported on language learning also dealt with mobile
learning.
54 / 56
Reporting of the results
• The Asian continent figures prominently among the
articles selected for the meta-analysis with 52% of all
publications where a geographical setting was identified.
• On the other hand, Northern America (20%) and Europe
(18%) account together for just over a third of all articles.
55 / 56
Reporting of the results
• Additionally, the analysis of the literature revealed a
scarcity of articles reporting findings from research
conducted in Malaysia. Indeed, from a total of 59 articles,
only two (3%) reported on the Malaysian context.
• As such, this finding represents a third gap in the
literature.
56 / 56
Reporting of the results
• Describe the implications of your findings from the critical
review.
57 / 56
Reporting of the results
• The main implication that resulted from the critical review
was the discovery of three gaps in the literature.
• Although a single knowledge gap would be sufficient
justification for conducting a research, the accumulation of
three strengthens the need for additional investigations.
58 / 56
Reporting of the results
• Having identified a manageable quantity of relevant
articles, it is possible to report on them without any bias.
59 / 56
Reporting of the results
60 / 56
Reporting of the results
61 / 56
THANK YOU Q&A
62 / 56
Application
1. Look through your articles and identify five as relevant.
2. Analyse them according to the criteria you have
identified
3. Present your findings
63 / 56
References
• Egger, M., Smith, G. D., & Phillips, A. N. 1997. Meta-analysis: Principles and procedures. British Medical Journal. 315(7121): 1533-1537.
• Falagas, M. E., Pitsouni, E. I., Malietzis, G. A., & Pappas, G. 2008. Comparison of PubMed, Scopus, Web of Science, and Google Scholar: Strengths and weaknesses. Federation of American Societies for Experimental Biology Journal. 22: 338-342.
• Guzzo, R. A., Jackson, S. E., & Katzell, R. A. 1987. Meta-analysis analysis. Research in Organizational Behavior. 9: 407-442.
• Jacso, P. 2005. As we may search – Comparison of major features of the Web of Science, Scopus, and Google Scholar citation-based and citation-enhanced databases. Current Science. 89(9): 1537-1547. http://cs-test.ias.ac.in/cs/index.php [5 May 2012].
• Jefferson, T., Wager, E., & Davidoff, F. 2002. Measuring the quality of editorial peer review. Journal of the American Medical Association. 287(21): 2786-2790.
64
References
• Klavans, R., & Boyack, K. 2007. Is there a convergent structure of science? A comparison of maps using the ISI and Scopus databases. Proceedings of the 11th International Conference of the International Society for Scientometrics and Informetrics.
• Rosenthal, R., & DiMatteo, M. R. 2001. Meta-analysis: Recent developments in quantitative methods for literature reviews. Annual Review of Psychology. 52(1): 59-82.
• Slavin, R. E. 1984. Meta-analysis in education: How has it been used? Educational Researcher. 13(8): 6-15.
• Wu, W.-H., Wu, Y.-C. J., Chen, C.-Y., Kao, H.-Y., Lin, C.-H., & Huang, S.-H. 2012. Review of trends from mobile learning studies: A meta-analysis. Computers & Education. 59: 817-827.
• Zhao, Y. 2003. Recent developments in technology and language learning: A literature review and meta-analysis. CALICO Journal. 21(1): 7-27. https://calico.org/journalTOC.php?current=1 [23 March 2012].
65