litereture review article

7/17/2019 Litereture Review Article

http://slidepdf.com/reader/full/litereture-review-article 1/34

Features

Scholars Before Researchers: On the Centrality

of the Dissertation Literature Review in ResearchPreparation

by David N. Boote and Penny Beile

A thorough, sophisticated literaturereview is the foundation and inspirationfor substantial, useful research. Thecomplex nature of education researchdemands such thorough, sophisticatedreviews. Although doctoral education is a

ey means for improving educationresearch, the literature has given shortshrift to the dissertation literature review. This article suggests criteria to evaluate

the !uality of dissertation literaturereviews and reports a study thatexamined dissertations at threeuniversities. Ac!uiring the sills andnowledge re!uired to be educationscholars, able to analy"e and synthesi"ethe research in a #eld of speciali"ation,should be the focal, integrative activity of predissertation doctoral education. $uch

scholarship is a prere!uisite for increasedmethodological sophistication and forimproving the usefulness of educationresearch.

e have all heard the %oe before&as we move through graduate school,we learn more and more about less andless until we now everything aboutnothing. 't is expected that someoneearning a doctorate has a thorough andsophisticated understanding of an area of research and scholarship. (nfortunately,

many doctoral dissertations in educationbelie the %oe, their authors failing tomaster the literature that is supposed tobe the foundation of their research. 'ftheir dissertation literature reviews areany indication, many of these now)doctors now bits and pieces of adisorgani"ed topic. *et we cannot blamethem for their failure to demonstratewhat we, the education researchcommunity, have not clearly articulatedor valued.

Ac!uiring the sills and nowledge

re!uired to be education scholars shouldbe the focal, integrative activity of predissertation doctoral education.Preparing students to analy"e andsynthesi"e research in a #eld of speciali"ation is crucial to understandingeducational ideas. $uch preparation isprere!uisite to choosing a productivedissertation topic and appropriatingfruitful methods of data collection andanalysis.

'n this article, we #rst argue that athorough, sophisticated review ofliterature is even more important in

education research, with its messy,complex problems, than in most other#elds and

+ducational esearcher, -ol. /,No. 0, pp. 123

disciplines. 4e then argue that currentinitiatives and faculty focuses haveignored the centrality of the literaturereview in research preparation, in turnweaening the !uality of education



research. This oversight has its roots, webelieve, in a too)narrow conception of theliterature review&as merely anexhaustive summary of prior research&and a misunderstanding of its role inresearch. By building on the extantliterature that supports the centrality of

the literature review, we o5er a practicalframewor from which to analy"e the!uality of doctoral dissertation reviews of the literature. 4e end by furtherdeveloping our understanding of theliterature review and indicating somemeans of improving the situation.

The Role and Purpose of theLiterature Review in EducationResearch

A substantive, thorough, sophisticatedliterature review is a precondition fordoing substantive, thorough,sophisticated research. 67ood8 researchis good because it advances ourcollective understanding. To advance ourcollective understanding, a researcher orscholar needs to understand what hasbeen done before, the strengths andweanesses of existing studies, and whatthey might mean. A researcher cannot

perform signi#cant research without #rstunderstanding the literature in the #eld.Not understanding the prior researchclearly puts a researcher at adisadvantage. $hulman argues thatgenerativity &along with discipline,publication, and peer review&is one of the hallmars of scholarship 92:::, p.20;120<. =e de#nes generativity as theability to build on the scholarship and

research of those who have come beforeus. 7enerativity grants our wor integrityand sophistication. To be useful andmeaningful, education research must becumulative> it must build on and learnfrom prior research and scholarship onthe topic.

*et the messy, complicated nature of problems in education maes generativityin education research more di?cult thanin most other #elds and disciplines9Berliner, ;@@;< and demands that wedevelop more sophisticated literaturereviews. 'n traditional disciplinaryresearch, where a researcher iscommunicating with a well)de#nedaudience about commonly acceptedproblems and where disciplinary research

often is based on a canon of sharednowledge, the researchers literaturereview is somewhat easier to construct.=owever, in education research we areoften faced with the challenge ofcommunicating with a diverse audience,and it is very di?cult for us to assumeshared nowledge, methodologies, oreven commonly agreed)upon problems9Boote 7audelli, ;@@;<. Cew of us worwithin a sub#eld of education research

that approaches 6normal science8 9T. $.uhn, 2:E@<& there are very few clear,cumulative research programs ineducation. Because such well)formedresearch communities are the exceptionrather than the rule, it is all the moreimportant that novice educationresearchers learn the craft ofconstructing a foundation on which theirresearch can be built.

As the foundation of any researchpro%ect, the literature review shouldaccomplish several important ob%ectives.'t sets the broad context of the study,clearly demarcates what is and what isnot within the scope of the investigation,and %usti#es those decisions. 't alsosituates an existing literature in a broaderscholarly and historical context. 't should

not only report the claims made in theexisting literature but also examinecritically the research methods used tobetter understand whether the claims arewarranted. $uch an examination of theliterature enables the author todistinguish what has been learned andaccomplished in the area of study andwhat still needs to be learned andaccomplished. Foreover, this type ofreview allows the author not only to



summari"e the existing literature but alsoto synthesi"e it in a way that permits anew perspective. Thus a good literaturereview is the basis of both theoretical andmethodological sophistication, therebyimproving the !uality and usefulness of subse!uent research.

't is a broadly held assumption thatsuccessful doctoral candidates need to be6comprehensive and up to date inreviewing the literature8 9Barry, 2::E<and that their dissertations demonstratethis prowess. Cor most educationresearchers, the doctoral dissertation isthe capstone to formal academic trainingand, as such, should be high !uality andcomprehensive and should reGectemerging research. The academic

community ought to be able to assumethat a dissertation literature reviewindicates a doctoral candidates ability tolocate and evaluate scholarly informationand to synthesi"e research in his or her#eld.

Despite the assumption thatdissertation literature reviews arecomprehensive and up)to)date, the dirtysecret nown by those who sit ondissertation committees is that most

literature reviews are poorlyconceptuali"ed and written. Hur secret ismade public by editors and reviewerswho openly lament the inade!uacy of literature reviews in manuscriptssubmitted for %ournal publication 9Alton)Iee, 2::J> 7rant 7raue, 2:::>IeKompte, lingner, Kampbell, Fen,;@@<. Crom Alton)Iees compilation of reviews of manuscripts submitted toTeaching and Teacher Education, we can

begin to see the problems associatedwith research by scholars who do notnow the literature in their #elds. Cor the3J manuscripts submitted for review overa 2)year period, she identi#ed 0:distinct criticisms in the 2/; reviews,which she divided into 2 broadcategories. Fethodological issues weremost common, but reviewers alsoidenti#ed theoretical shortcomings 92times<, inade!uacies in literature review

9;:<, parochial focus 9;<, failure to addto the international literature 9;2<, andfailure to lin #ndings to literature 9;@<.

'n short, it appears that either many of the authors who submit manuscripts tothis international %ournal do not now theliterature in their #elds or else their

nowledge of their #elds does not informthe presentation of their manuscripts.Foreover, a better understanding of theresearch in their #eld might have aidedthem with the other methodologicalproblems that the reviewers identi#ed.4e speculate that the shortcomings thateditors and reviewers bemoan stem frominsu?cient preparation in doctoralprograms. Fethodological training cannotoccur in a vacuum, and increased training

in research methods alone will not lead tobetter research. 'nstead, we mustrecogni"e the centrality of the literaturereview in doctoral research preparationand broaden our understanding of whatliterature reviewing entails.

Education Research and DoctoralPreparation

There is an emerging consensus that theperceived lac of !uality in education

research stems from problems withdoctoral preparation and that improvingdoctoral education is ey to improvingeducation research. 'nitiatives by severalfoundations have spawned a small buthealthy literature analy"ing the problemsof doctoral education, describing revisedprograms, and maing

recommendations.2

4hen considering the criteria andstandards used to evaluate a dissertation,

we need to eep in mind that mostpeople with doctorates in education donot go on to pursue research careers.Fost teach, administer, or lead9Passmore, 2:J@<. *et anyone earning adoctorate ought to be a steward of the#eld of education 9ichardson, ;@@<,with all the rights and responsibilitiesthereto appertaining. Hne of ourresponsibilities&whether we become aresearcher, teacher, administrator, or



leader&is to now the literature in our#eld. And the best avenue for ac!uiringnowledge of the literature 9beyondtaing courses and comprehensiveexaminations< is the dissertationliterature review. *et it is apparent that for many, if not

most, doctoral candidates anddissertation committees, the literaturereview is of secondary importance. Thiswas not always the case. (ntil the 2:thcentury, the doctorate was primarily ateaching degree, certifying that one hada thorough and sophisticated grasp of aparticular #eld of study. As such, thedissertation and accompanying oralexamination served primarily to assessones suitability as a scholar and a

teacher. 7raduate education in the(nited $tates developed in a period when7erman universities were ascendant andwhen 6America copied the 7ermanversion of advanced studies . . .unfortunately the period when the Berlinpositivists were in the ascendancy89Berelson, 2:0@, p. 2;> see also $torr,2:0:<. As a result, the (.$. doctorate wasdesigned to focus on research training,and the dissertation became a vehicle for

demonstrating research prowess.;

Konsistent with the assumption that thedoctorate is primarily for researchtraining, the limited (.$. literature oneducation doctorates has focusedprimarily on methodological andepistemological issues and to a lesserextent on the core and canon of

education nowledge. Barger and Duncan92:J0< raise di?cult !uestions about theassumption that doctoral candidatesshould be expected to do creativescholarly wor, and outline what they feelare the psychological, theoretical1methodological, and institutional contextsre!uired for creative wor. Based on acollection of anecdotes and experiences,$choenfeld 92:::< identi#es a number of di?culties and dilemmas facing doctoral

education. Among these arespeciali"ation that leads tocompartmentali"ation, theori"ing thatleads to super#ciality, and simplisticapproaches to methodology that hinder adeep understanding of what it means tomae and %ustify a claim abouteducational phenomena. $choenfeldsuggests that many graduates completetheir degrees unable to identify andframe worable research problems.

'n a theme issue of EducationalResearcher , several authors 9Fet", ;@@2>Page, ;@@2> Pallas, ;@@2< discussedsimilar challenges that they had faced ineducating doctoral students and methodsthat they had used in their programs toaddress those challenges. Hn the basis of his experience coordinating and teachingin a doctoral program, Iabaree 9;@@<outlined some of the general problemsfacing doctoral education. =e framed theproblems in terms of a clash betweenschool and university cultures that occurswhen we as teachers to shift from anormative to an analytic way of thining,from a personal to an intellectualrelationship with educational phenomena,from a particular to a universal

perspective, and from an experimental toa theoretical disposition.

An important exception to the emphasison methodology is ichardson 9;@@<,who develops the concept of doctors of education as stewards of both the #eld of study and the enterprise of education.$he uses this conceptual framewor toargue for the nowledge, sills, anddispositions that doctoral programs ineducation should inculcate. The (.$. literature on the education

doctorate is reminiscent of the earlyresearch on learning to teach 94ideen,Fayer)$mith, Foon, 2:::<> with little orno support from solid data, the authorsrely on their personal prestige to discussthe problems of practice and maerecommendations for improvement. Butlie the literature on learning to teach,



the literature on learning to researchmust move from anecdotes,generali"ations, and reports of programs

to systematic investigation andrecommendations based on evidence.

The Literature Review: A NecessaryChore?

The perceived lac of importance of thedissertation literature review is seen inthe paucity of research and publicationsdevoted to understanding it. Doctoralstudents seeing advice on how toimprove their literature reviews will #ndlittle published guidance worth heeding.+very introductory educational, social,and behavioral research textboocontains a chapter or section on

reviewing prior research as part of theresearch process 9e.g. Babbie, 2::J>Kreswell, ;@@;> Craenel 4allen, ;@@>7ay Airasian, ;@@@> FcFillan $chumacher, ;@@2<. These chapterstypically indicate the importance of theliterature review, albeit in vague terms,and brieGy summari"e techni!ues forsearching electronic databases andmethods for abstracting prior research.4e infer from these chapters and

sections that the authors of thesetextboos acnowledge the importance of the literature review, at least in asalutatory way, but place a much greateremphasis on an understanding of methods of data collection and analysis.

'n accordance with other textboos,Kreswell 92::/< suggests that theliterature review should meet threecriteriaL 6to present results of similarstudies, to relate the present study to the

ongoing dialogue in the literature, and toprovide a framewor for comparing theresults of a study with other studies8 9p.E<. To accomplish these criteria Kreswell9;@@;< recommends a #ve)step processL6identifying terms to typically use in yourliterature search> locating literature>reading and checing the relevance of the literature> organi"ing the literatureyou have selected> and writing aliterature review8 9p. J0<. 4ith guidelines

lie these, graduate students could beforgiven for thining that writing aliterature review is no more complicatedthan writing a high school term paper. These chapters and sections in

introductory research textboos are notthe extent of the literature, but othersources devoted to the topic of literaturereviewing and dissertation writing arelittle more sophisticated 9e.g., 7alvan,;@@/> Iester, ;@@;> Fauch, 2::J>Nicerson, 2::> Hgden, 2::> Pan,;@@<. Fore advanced researchtextboos and handboos ignore thesub%ect, focusing entirely on methods of data collection, interpretation, andphilosophical issues. 'n other words, withthe very few exceptions noted below,most graduate students receive little orno formal training in how to analy"e andsynthesi"e the research literature in their#eld, and they are unliely to #nd itelsewhere.

Bruces 92::/< phenomenographic

study of metaphors for the literaturereview sheds further light on thelimitations of published criteria forreviewing literature. $he found thatresearch students perceive themselves in!uite diverse metaphorical relationshipswith the literature in their #eld, rangingfrom listing, searching, and surveying toacting as a vehicle for learning,facilitating, and reporting. Note that themetaphor of vehicle for learning,

facilitating, and reporting suggests moresophisticated, developed, and integratedliterature reviews. Although a literaturereview consistent with any of thesemetaphors can ful#ll textboo criteria,candidates may view themselves asanywhere from metaphorically standingaside and hapha"ardly cataloguing prior#ndings to critically analy"ing andsynthesi"ing the #eld of study. Brucesaccount o5ers further support for the



view that criteria published in populartextboos are too vague and do notprovide clear standards. That doctoral candidates would espouse

such naMve conceptions of literaturereviewing and perceive it as relatively lowin importance would seem to be a

product of the culture of doctoralprograms in education. aporo"het"92:JE< reported that doctoral candidatesfelt their library sills were inade!uate,while their faculty advisors admittedexpecting their candidates to possessadvanced bibliographic sills even thoughthe advisors themselves had littlenowledge of information retrieval.Caculty also raned the review)of)literature chapter the lowest in

importance when considered in relationto the other standard dissertationchapters. aporo"het" also reported thatmost dissertation chairs and studentssaw the literature review as a relativelyroutine activity that doctoral candidates

should be able to complete alone withlittle help from their advisors. AndIabaree 9;@@< reminds us that mostdoctoral students in education have littleformal training in education research andscholarship before they start theirdoctorate, with their undergraduate and

masters degrees usually in other #elds ordisciplines or focused almost entirely oneducation practice. Both aporo"het" andIabaree note that education doctoralstudents tend to be mature,accomplished professionals who arecommitted to improving educationpractice. *et these !ualities mae it moredi?cult for them to admit that they maylac library search and informationsynthesis sills and nowledge. 4e may

speculate that, for similar reasons, it isdif#cult for education faculty to admit tolacing such sills and nowledge&andthe sills and nowledge that they canclaim in that area probably are tacit andhence di?cult to teach.

A product of this doctoral program

culture is that the literature review is notvalued, and because it is not valued it israrely an explicit part of doctoralcurriculums. Perhaps the conGictingmessages and lac of formal trainingexplain why it is not unheard of foreducation doctoral candidates toresearch and write their literature reviewsafter they have decided on their researchproblems and methods. Thisunsystematic approach is not surprising

when we consider how di?cult it is to #nda clear articulation of the criteria andstandards for !uality in a literaturereview. $tudents often lac thenowledge and sills even to completethorough summaries of the existingliterature, let alone more sophisticatedforms of research synthesis. And,because literature review is not valued, itis also not evaluated. Dissertations passdespite their poor literature reviews, and

another generation of educationresearchers fails to learn that generativityis the core of scholarship&fails to learn

what it means to understand and %ustifyan educational idea in a thorough,sophisticated way.

Iibrarians have been aware of theseissues for some time and have o5eredmany suggestions for improving thesituation 9see Iibutti opala, 2::3, fora review<. *et many librarians su5er fromsome of the same naMve conceptionsabout the role of the dissertationliterature review as do doctoral

candidates and education faculty. Iibraryinstruction has tended to focus on themechanics of database search strategiesand on the varieties of informationavailable.

Bibliographic sills and nowledge arenecessary for ensuring that a researchercan locate and evaluate the availableliterature, but a literature review shouldnot be understood as merely anexhaustive summary of prior research.



'nstead, we need to understand that theability to write a thorough, sophisticatedliterature review is a form of scholarshipre!uiring a broad range of sills andnowledge&sills and nowledge that weought to expect of anyone earning adoctorate.

The Literature Review: urFoundation and !nspiration

Despite the scant attention paid toliterature reviewing in research textboosand programs, a few authors have clearlyarticulated its centrality in research.Kommenting on the importance of reviews, Iather 92:::< argued that asynthetic review should serve a criticalrole in gateeeping, policing, and leadingto new productive wor, rather thanmerely mirroring research in a #eld. 'n aneditorial in Review of EducationalResearch, IeKompte and colleagues9;@@< wrote on the importance of convincing emerging scholars that

state)of)the)art literature reviews arelegitimate and publishable scholarlydocuments. Too many new scholarsbelieve that empirical research is theonly 6real8 research> they avoid the

deep levels of investigation needed tocreate the inds of manuscripts soughtby RER. This leaves educationresearch without an integrative andcritical grounding in priorinvestigations and weaenssubse!uent wor. 9p. 2;/<

$trie and Posner 92:J, pp. 3013E<further suggest that a good syntheticreview has three characteristics. Cirst, it

clari#es and perhaps resolves theproblems within a #eld of study ratherthan glossing over those problems.$econd, it results in a 6progressiveproblem shift8 that yields a newperspective on the literature with moreexplanatory and predictive power than iso5ered by existing perspectives. Cinally,it satis#es the formal criteria of a goodtheory. $tandards such as consistency,parsimony, elegance, and fruitfulness

characteri"e a good synthesis. Iather92:::<, IeKompte et al. 9;@@<, and$trie and Posner seem to suggest thatreviewing the literature in a #eld perhapsdoes re!uire more training than is neededto write a high school term paper andthat learning to perform substantive

literature reviews should be part ofdoctoral education.

An interesting contrast to the (.$.literature is provided by several studiesfrom the (nited ingdom and Australia,where dissertations and research thesesare normally ad%udicated by outsideexaminers who submit written reports9Delamont, Atinson, Parry, ;@@@> Oohnston, 2::E> Nightingale, 2:J/>Pitethly Prosser, 2::3> 4inter,

7ri?ths, 7reen, ;@@@<. Noting thevagueness and ambiguity of commonlyespoused goals for dissertations, thesestudies analy"ed the examiners reportsto better understand the criteria andstandards of evaluation. Fany of thesestudies indicate that examiners oftenfound problems with literature reviews,although criticisms of methodology werea more serious concern.

=owever, a later study by Fullins and

iley 9;@@;< established a lin betweenmethodology and the literature review. They interviewed experienced Australiandissertation examiners and found thatexaminers typically started reviewing adissertation with the expectation that itwould pass> but a poorly conceptuali"edor written literature review oftenindicated for them that the rest of thedissertation might have problems. Hnencountering an inade!uate literature

review, examiners would proceed to looat the methods of data collection, theanalysis, and the conclusions much morecarefully. 'n that way, Fullins and ileyfound that for examiners there was atacit lin between candidates nowledgeof the #eld and their ability to dosubstantive, well)%usti#ed research.

'n contrast to the vague suggestions of the (.$. research methodology textboos,=art 92:::< suggests a much more



central role for the literature review indoctoral dissertations. By maintainingthat a candidate simply cannot dooriginal or substantial research without athorough understanding of the #eld, =artruns contrary to the assumption that datacollection and analysis constitute the

centerpiece of a doctoral dissertation. 'naddition to the commonly discussedcriteria of summari"ing similar studies,lining the dissertation research toongoing research in the #eld, andproviding a basis for comparing thedissertation #ndings to prior studies, =artcontends 9p. ;E< that a dissertationliterature review should clearly articulatewhat research needs be done in a #eldand why it is important, articulate the

practical signi#cance of the research,synthesi"e prior research to gain a newperspective on it, and critically analy"e

the research methods used. =e alsoanaly"es the most commonly discussedcriteria and subdivides them into moreunderstandable components. Byintroducing these new criteria and setting

higher expectations, he communicatesthe importance not only of the literaturereview in a dissertation but also of thecriteria and standards against which itshould be %udged.

't is important to emphasi"e here that=arts criteria&supported by Iather

92:::<, IeKompte et al. 9;@@<, and$trie and Posner 92:J<&give us !uite adi5erent conception from that seen inmost published accounts of dissertationliterature reviewing. =art clearlyarticulates that doctoral students must besuccessful scholars&able to criticallysynthesi"e ideas and methods in their#eld&before they are to have any chanceof being generative researchers. Kontrastthis with the most common conception,

which seems to entail a mechanicalprocess of summari"ing a supposedlyexhaustive collection of prior studies.Pi!ued by our own experiences withdoctoral students lac of expertise inliterature reviewing, we adapted =arts92:::< criteria to investigatesystematically how well doctoral studentswere learning the sills of scholarship.

"tandards and Criteria of aLiterature Review

'n our recent study 9Boote Beile, ;@@/<we used =arts 92:::< criteria to developa framewor from which to analy"eliterature reviews in doctoraldissertations in the #eld of education. 4ealso sought to understand whether the

criteria are reasonable for a dissertationliterature review. =arts criteria wereadapted and incorporated into our 2;)item scoring rubric, which can begrouped into #ve categories 9see Table 2<. The #rst category, 6Koverage,8 consists

of a single criterion that was not one of =arts. Kriterion A assessed how well theauthor of the dissertation %usti#ed criteriafor inclusion and exclusion from review.Kooper 92:J3< argues that

coverage is probably the most distinctaspect of literature reviewing. Theextent to which reviewers #nd andinclude relevant wors in their paper isthe single activity that sets thisexpository form apart from all others.=ow reviewers search the literatureand how they mae decisions aboutthe suitability and !uality of materialsinvolve methods and analyticprocesses that are uni!ue to this formof scholarship. 9p. 2;<

Although it is worth noting that Kooper isreferring here to literature reviewing as adistinct form of scholarship, we believethat the same expectation should beapplied to a literature review that is aprecursor to research.

4e are encouraged in this belief byKoopers 92:J3< observation that there



are interesting di5erences among theways that authors search the literatureand mae decisions about suitability and!uality. elative novices to a topic of thereview, measured by the number of previous publications on the topic, tendto be very explicit about their search

strategies and criteria and are more lielyto use databases and indexes to identifyand select research to review. elativeexperts, on the other hand, tend to notbe as explicit about their searchstrategies and criteria and often rely onpersonal communications with leadingresearchers as their main means of identifying relevant research. 4e mightinfer that part of the reason that relativeexperts on a topic do not need to %ustify

their criteria for inclusion and exclusion isthat readers will assume that the well)recogni"ed authors now the literature.Doctoral candidates are noviceresearchers almost by de#nition and donot have the luxury of being assumed tonow the literature. Cor that reason webelieve that the onus is on doctoralcandidates to convince their readers thatthey have thoroughly mined the existingliterature and purposefully decided what

to review. =%orland 92:JJ< provides avivid case study of what happens when adoctoral student lacs sophisticatedlibrary search sills> the case study showsthe e5ects on both the originality and theexhaustivity of the resulting dissertation. The dissertation that =%orland analy"edwould probably have been ade!uate formost dissertation committees, but thecandidates inability to mine the existingliterature led to many erroneous claims

about the state of nowledge in the #eld. *et library search sills are not enough.

Too often, coverage is interpreted bydoctoral students as exhaustive coverageof everything previously written abouttheir topic 9Bruce, ;@@2a<. This naMveapproach to searching and selecting priorresearch can mae it very di?cult forresearchers to critically synthesi"e theliterature in their #eld, especially whenthe literature is relatively small or large,

or when it is highly fragmentedempirically, conceptually, or ideologically.Bruce suggests that coverage should belooed at more broadly. =e proposeseight criteriaL topicality,comprehensiveness, breadth, exclusion,relevance, currency, availability, and

authority. Thus, for example, a studentreviewing the literature on a topic aboutwhich very little has been written mayneed to broaden the search to examineanalogous research in other #elds ortopics. A student reviewing the literatureon a topic about which a great deal hasbeen written may need, instead, to focuson the best available evidence or on asmaller number of ey conceptual pieces.4hatever the strategy adopted, the

burden is on the doctoral candidate toconvince the audience that inclusion hasbeen purposeful and thorough. KriterionA is included in our rubric to measure thedegree to which selection criteria areclearly %usti#ed in the dissertation. The second category, 6$ynthesis,8

consists of Kriteria B through 7 and isdesigned to gauge how well the authorsummari"ed, analy"ed, and synthesi"edthe selected literature on a topic. The

individual criteria as how well the author9B< distinguished what has been done inthe #eld from what needs to be done, 9K<placed the topic or problem in thebroader scholarly literature, 9D< placedthe research in the historical context of the #eld, 9+< ac!uired and enhanced thesub%ect vocabulary, 9C< articulatedimportant variables and phenomenarelevant to the topic, and 97< synthesi"edand gained a new perspective on the

literature. As Iather 92:::< and $trieand Posner 92:J< suggest, this endeavorshould enable the author to synthesi"ethe literature, gain a new perspective onit, and clarify what has been done andstill needs to be done. $uch a synthesisenables the dissertation author to clarifyand resolve inconsistencies and tensionsin the literature and thereby mae agenuine contribution to the state ofnowledge in the #eld, by developing



theories with more explanatory andpredictive power, clarifying the scope andlimitations of ideas, posing fruitfulempirical investigations, andoridentifying and pursuing unresolvedproblems. This ind of theori"ing iscentral to our conception of what it

means to earn a doctoral degree.Kriteria = and ' constitute the third

category, 6Fethodology.8 Kriterion =measures how well the author identi#edthe main methodologies and researchtechni!ues that have been used in the#eld, and analy"ed their advantages and

disadvantages./ Kriterion ' evaluates howwell the authors literature review relatedideas and theories to researchmethodologies 9a criterion not included in=arts Q2:::R list<. At minimum, an authorshould recogni"e how previousresearchers methodological choices

a5ected the research #ndings. Anysophisticated review of literature shouldalso consider the research methods usedin that literature and consider thestrengths and weanesses of thoseresearch methods in relation the

Ta#le $Literature Review "corin% Ru#ric

&

2. Koverage

A. Ousti#ed criteria f or

in

clusion and

Did not di

scuss the D



iscusse

d the liter

ature Ousti

#ed inclusi

on and excl

usion e

xclusion f r

om review.

criteria in

clusion or



include

d and of li

terature ex

clusion exc

luded

;. $ynthesis B. Distinguished what hasbeen Did not distinguishDiscussed what has andKritically examined thestate of done in the #eld

from what needs what hasand has has not been donethe #eld to be done. notbeen done

K.Pl

aced the to

pic or prob

lem in the

Topic not p



laced i

n $ome disc

ussion of T

opic clearl

y situated

in broa

der broader

scholarly

literature

broader sch



olarly

broader sch

olarly scho

larly liter

ature liter

ature l

iterature

D.

Placedtheresearchinthehistori)

=istoryoftopicnot$omemention of

his)Kriticallyexaminedhistorycalcon



textofthe#eld.discuss

edtoryoftopic of topic

+.

Ac!uired and

enhanced thesub)eyvocabulary noteyvocabularyde#nedDiscussed and

resolvedambigui%ectvocabulary.discussed tiesinde#niti

onsC.

Articulatedimportant

variables eyvariables andeviewedrelationshipsNotedambiguities in

literature andphenomenarelevanttophenomena notamongeyvariable

s andproposed newrelationships thetopic.discussed andphenomena

7. $ynthesi"ed and gained a newAccepted literature at $ome

criti!ue of H5ered new perspective

perspective on the literature. face valueliterature

. Fethodology =. 'denti#ed the main

methodolo)esearchmethods $ome discussionof re)Kriti!ued researchmethods 'ntroduced newmethods gies and researchtechni!ues not discussedsearch methods used to

address problems thathave been used in the#eld, to produce claimswith predominant and theiradvantages and methodsdisadvantages.

'.



elated

ideasand th

eoriesin th

eesearch m

ethods$omed

iscussi

on ofap)Kri

ti!ued appr

opriateness

of#eld tore



search

methodologi

es.notdiscu

ssed propri

atenessof re

)resear

ch methodst

owarrantsea

rch methods

toclaimswar



rantcla

ims

/. $igni#cance

O. at

ionali"ed t

he practical signi#)Pr

actical

signi#canc

e Practical

signi#canc

e Kriti!ued



practi

cal signi#c

ance of can

ce of the r

esearch pro

blem. o

f research

not discuss

ed research

discussed



. ationali"ed the scholarly$cholarly signi#cance$cholarly signi#canceKriti!ued scholarlysigni#cance of signi#canceof the research of researchnot discussed research

problem. discussed3. hetoric I. 4as written with a

coherent, Poorlyconceptual)$ome coherent4ell developed, coherentclear structure thatsupported the i"ed,hapha"ard structurereview.

NoteLThe column)head numbers

represent scores for rating dissertationliterature reviews on )point and /)pointscales 9endnote / explains our choice of the two types of scales<. Adapted fromDoing a Iiterature eviewL eleasing the$ocial $cience esearch 'magination9p.;E<, by Khristopher =art, 2:::, Iondon,$A7+ Publications. Kopyright 2::: by$A7+ Publications. Adapted withpermission.state of the #eld. 'n many cases, the

body of literature on a topic is limited bythe research methods used and advanceswithin the #eld can be traced bac toincreased methodological sophistication.-ery sophisticated literature reviewsmight recogni"e the methodologicalweanesses of a #eld of study andpropose new methodologies tocompensate for those weanesses.Kriteria = and ' measured how well thedoctoral candidates identi#ed the main

methodologies and research techni!uesused in the #eld, analy"ed theiradvantages and disadvantages, andrelated the ideas and theories in the #eldto the research methodologies. To scorewell, the candidates could %ustify theirown methodological choices and perhapseven suggest and %ustify new researchmethods. The fourth category, 6$igni#cance,8

includes Kriteria O and , which measure

how well the dissertation rationali"ed thepractical 9O< and scholarly 9< signi#canceof the research problem. 4e wouldexpect that, at minimum, a dissertationshould discuss both the scholarly and thepractical implications of the existingresearch on a topic and, preferably, note

any ambiguities or shortcomings in theliterature. $ome dissertations clearly aremore scholarly in their orientation andothers are more practical, but we preferthat any dissertation explain both thepractical and the scholarly signi#canceand limitations of prior research on thetopic 9ichardson, ;@@<. Thisexpectation acnowledges theimportance of lining research andpractice in the #eld of education.

The #nal category, 6hetoric,8 alsoconsists of a single item, Kriterion I,which measures whether the literaturereview was written with a coherent, clearstructure that supported the review. Thiscriterion, too, was not included in =artslist, but it emerged as an important oneas we read through our sample ofdissertations. Hnce an author hassummari"ed, analy"ed, and synthesi"edthe literature, he or she will want to mae

some claims about that literature. Thoseclaims should be articulated clearly, andthe writing should be organi"ed tosupport them. This may seem lie anobvious point, but our experience evenbefore undertaing this research gave usmany examples of literature reviews thatlaced rhetorical structure and were verypoorly written. Koopers 92:J3< study of graduate students reading literaturereviews reported that rhetorical structure

and organi"ation were ey determinantsin how inGuential and persuasive readersbelieved the review to be.

Fore generally, 7ranello 9;@@2< hasargued that focusing on the formalaspects of writing is a means ofincreasing students cognitivecomplexity, moving students from lowerto higher levels of Blooms taxonomy of the cognitive domain 9see also Iibutti opala, 2::3<. =aving to organi"e ones



thoughts is important not merely forpersuading an audience but for betterunderstanding what one is writing 9lein,2:::> D. uhn, 2::;> ivard, 2::/<.Kriterion I measures how well thedissertation articulated clear claimsbased on its analysis and synthesis of the

literature and supported those claimsthrough purposeful organi"ation andcogent writing.

Taen together, these twelve criteria andassociated standards set ambitiousexpectations for doctoral dissertationliterature reviews. A literature review thatmeets high standards on these criteriaindicates that the doctoral candidate hasa thorough, sophisticated understanding

of a #eld of study&a precondition forsubstantial, useful research.

Literature Review Analysis Findin%s

After developing our rubric, we initiallyexamined @ dissertations awarded in theyear ;@@@ from three state)fundedcolleges of education in the (nited$tates> we selected 2; of those

dissertations for full analysis. The threecolleges all o5ered doctoral degrees inaddition to signi#cant involvement withpreservice teacher education, had similarrates of acceptance to their graduateprograms, and had a comparable numberof faculty members. Hne college wasraned by US News & World Report among the top 23 (.$. colleges of education> one was among the top @>and one was not raned. Although we are

cautious regarding the methods used byUS News & World Report to ran schools,our sample represents something of thediversity among state)funded educationdoctoral programs. Crom these schoolswe chose a strati#ed random sample of thirty dissertations representing thegeneral topics of education leadership,educational psychology, instructional orlearning theory, and teacher education.Hur analysis of dissertation literaturereviews supports $choenfelds 92:::<contention that doctoral students maynot be learning what it means to mae

and %ustify educational claims.3

Hur #ndings raise interesting !uestionsabout doctoral candidates ability to writea thorough and sophisticated literaturereview and what doctoral dissertationcommittees will accept as ade!uate.Although our analysis of scores revealed

di5erences in !uality of dissertationliterature reviews among the institutions,mean scores across all the institutionswere surprisingly low. These results mustbe interpreted cautiously because of oursmall sample, but we cannot deny thatthe worst literature reviews we analy"ed

were mere dis%ointed summaries of ahapha"ard collection of literature. 4edeveloped a very clear sense that formany of these doctoral candidates,reviewing prior research on their topicwas a hollow exercise from which theylearned nothing of substance and whichcontributed little to their understandingof their research pro%ect. Foreover, thecommon failure to synthesi"e literature,criti!ue methodology, or explain scholarly

signi#cance supported $choenfelds92:::< assertion. $uch lac ofsophistication does not bode well for thestudents ability to stay abreast ofresearch in their #eld as teachers,administrators or leaders&let alone tolead productive research careers afterreceiving their doctorates. That said, we are happy to report that

the best literature reviews were thorough,critical examinations of the state of the

#eld that set the stage for the authorssubstantive research pro%ects. Thesehigh)!uality reviews lead us to believethat our criteria and standards are notunreasonable. They also support =arts92:::< claim that the criteria areimportant for doctoral studentsunderstanding of their #eld andprere!uisite for framing fruitful researchproblems and appropriating sophisticatedresearch methods 9ichardson, ;@@>



$choenfeld, 2:::<.Perhaps even more remarable than the

di5erences among institutions is therange of scores and amount of variationwithin each institution. The variancewithin all three institutions implies thatliterature reviews were not held to

consistent criteria or standards, or thatstandards for acceptable literaturereviews were of little or no importance.4e were not surprised by this #nding,considering that we suspect that

aporo"het" 92:JE< report is generallytrue and that most faculty supervisingdoctoral dissertations do not value therole of reviewing literature in a researchpro%ect. 4e are inclined to attribute thedi5erences to faculty expertise ande5ectiveness at communicating

scholarship expectations, and we canonly infer that some faculty have higher,albeit uncodi#ed, minimal standards foracceptable wor than do others.

The assumption that all doctoralcandidates are on the cutting edge ofcurrent research in their #eld was not

well supported by our study, nor can ourstudy %ustify the assumption that alldoctoral candidates have learned tocritically analy"e and synthesi"e researchin their #eld. Although we must stillexercise caution with our small samplefrom a limited number of schools, ourresults suggest that not everyone with adoctorate in education understands the

norms of scholarly communication or theprocesses of warranting scholarly claims. The existing literature and our analysis of dissertation literature reviews suggeststhat the criteria and standards for a high)

!uality literature review are not part ofthe formal curriculum or graduationexpectations of even nationally raneddoctoral programs. Doctoral faculty andprograms must pay more attention toexplicitly teaching and assessingstudents on the norms and methods ofscholarship and scholarly communication.

Re'nin% ur Conception of Literature Reviewin%

The primary purpose of this article is tohighlight the general weaness of dissertation literature reviews and, in sodoing, to argue their centrality inpreparing doctoral candidates to bebetter scholars and researchers. 4edeveloped our rubric as a research tool toassess suggested criteria but have

discovered that many graduate facultymembers and graduate students seemmore interested in using it as apedagogical tool to teach or learnliterature review sills. As we hear fromthese colleagues, however, it becomesclear to us that education researchershave !uite varied beliefs about theliterature review and its role in learninghow to do research.

Hne of the most common concerns

raised about our research is whether the

criteria and standards that we havedeveloped should apply to the two typesof doctoral degrees in education and tovarious dissertation formats. Cirst, shoulda literature review from an +d.D.dissertation and a literature review froma Ph.D. dissertation be assessed bymeans of the same criteriaS $ome mightsay, for example, that an +d.D.dissertation should be more concernedwith the practical implications ofresearch, whereas a Ph.D. dissertationshould be more concerned with itsscholarly importance. Although there ismuch debate about the role and purposeof each degree, we tae the position thatanyone earning a doctorate in educationought to now the literature in his or herarea of speciali"ation&indeed, it is !uiteunclear to us what, exactly, earning adoctorate might signify if one does notnow the literature in ones #eld. Cor that



reason, we did not di5erentiate between+d.D. and Ph.D. degrees in our study, andwe do not believe that the correspondingliterature reviews should be held todi5erent standards and criteria. The second concern often raised is

whether our criteria ought to apply to the

various formats of dissertations. Theconcern here seems to be that ourcriteria may inadvertently valori"e thetraditional #ve)chapter, empiricaldissertation and may be inappropriate toapply to other dissertation formats. Thiscertainly was not our intent, nor do wethin it will be the e5ect.

Paltridge 9;@@;< di5erentiates amongfour general dissertation formats, each of which is seen among education

dissertations. The traditional simpledissertation presents a single study in#ve chaptersL 'ntroduction, Iiteratureeview, Fethodology, esults, andKonclusions. Twenty)nine of the @dissertations that we sampled used thisformat.

4hat Paltridge 9;@@;< calls thetraditional complex format presentsseveral related studies, each presentingits own introduction, methods, results,

and conclusions. The traditional complexdissertations that he found tended to usea single literature review for all of thestudies. 4e did not have anydissertations of this type in our sample,but we are aware that they are oftenused in the #eld of education, especiallyfor behavioral)science)inGuenced topics. The topic-ased dissertation is also

often used in education, especially fortheoretical, philosophical, humanities)

based, and !ualitative dissertations9Paltridge, ;@@;<. 'n topic)baseddissertations, authors divide the largerwor into chapters that best support therhetorical structure and often do not useseparate chapters for the literaturereview, methodology, results, orconclusions. Hur sample contained onesuch dissertation, a narrative of aprofessional development collaboration inwhich the literature reviewed was

interspersed throughout the dissertation.4e had no di?culty applying the rubric tothis format> indeed, it scored fairly wellagainst our criteria and standards.

Cinally, the compilation of researcharticles format for dissertations,advocated in these pages by Due and

Bec 92:::<, presents a number of discrete articles often written in theformat of %ournal articles, framed withintroductory and concluding sections.+ach article must be complete unto itself,including its own literature review.Although this ind of dissertation hasbeen of increasing popularity in thesciences 9Dong, 2::J<, we are not certainhow often it is used in education. Hnenotable di5erence between article

compilations and other dissertationformats is that dissertations in thecompilation format tend to be written6more as experts writing for experts,than novices writing for admission to theacademy8 9Paltridge, ;@@;, p. 2;<. Assuch, they contain much less writing thatseems to serve the purpose of merelydisplaying the authors nowledge.

Nothing in our arguments or analysis isintended to %ustify the all)too)common

practice, in a #ve)chapter, traditionalsimple dissertation, of isolating theliterature review in the second chapter. Adoctoral candidate who has a thorough,sophisticated understanding of theliterature should clearly be expected todemonstrate an understandingthroughout the dissertation, fromintroduction through conclusion. 4e canonly speculate that the tendency toisolate the literature review reGects

doctoral candidates less)thanthoroughunderstanding of their literature andinability to see how the literature shouldinGuence their choice and %usti#cation of research topic, choice and %usti#cation of methods of data collection and analysis,and discussion of the #ndings,conclusions, and implications. The choiceof format for a dissertation ought to #t itsrhetorical structure, including thedecision whether to write the literature



review as a stand)alone chapter or tointegrate it throughout the dissertation.4hatever format the author chooses, a

thorough, sophisticated review ought tobe inGuential and evident in the entiredissertation.

4e would expect to #nd signi#cantdi5erences among the dissertationliterature reviews according to format,including presentation, format, degree of integration, and authorial voice. 4e cansee no reason, however, to suggest thatthe various dissertation formats ought tobe %udged against di5erent criteria intheir literature reviews. A dissertation of any format should demonstrate that theauthor thoroughly understands the

literature in his or her area of speciali"ation. The fact that ;: of the @dissertations we examined were of thetraditional)simple format made our datacollection easier but is not relevant to theapplication of the criteria.

A related concern seems to originatefrom the practice, perhaps commonamong students, of writing literaturereviews as part of dissertation proposalsand then using the same literature in the

dissertation with little revision. Thus theliterature review becomes a static artifactrather than a dynamic part of the entiredissertation. 'n contrast, we wouldnormally expect candidates to continuallyrevisit their understanding of theliterature throughout the dissertationexperience. This might mean rereadingthe literature in light of subse!uent#ndings or analysis, or reading newliterature to address emerging #ndings or

ideas. 4ithout viewing the literaturereview as a dynamic, integral part of theresearch process, we are much moreliely to #nd the problems that Alton)Iee92::J< identi#ed in the submissions toTeaching and Teacher Education.

$everal critics have also raisedconcerns about Kriterion A, 6Ousti#edcriteria for inclusion and exclusion fromreview,8 noting that it was not stipulatedby =art 92:::< and that even the best of

the dissertations that we analy"ed did notscore well on this criterion. Hf all of the

criteria we used in our study, we suspectthat this will be most contentious. Hurdecision to include this criterion wasbased on our experience woring withdissertation students and noting howhapha"ardly many approached theliterature search. 4e also are aware thatreview %ournals increasingly expectauthors to describe explicitly how theyidenti#ed research to indicate theconditions for inclusion and exclusion,

implicitly following Koopers 92:J3<recommendations. 4e continue tobelieve that it is important for allresearchers, especially novices, to beginto tae this more methodical approach toliterature reviews. 4e also contend thatits use holds the most potential forimproving the !uality of dissertationliterature reviews, as it forces candidatesto be more methodical.

Hther critics have worried that

providing detailed criteria to evaluate the!uality of a literature review will lead toyet more dissertations that are

formulaic.0 This outcome seems unlielyto us, but we acnowledge that clearcriteria alone will never lead to betterscholarship. As Bargar and Duncan92:J0< write, a 6thorough understandingand sincere commitment to problems of importance can and very often do lead topedestrian, unimaginative solutions8 9p.3<. Iess)successful researchers haveperhaps never learned to developproductive research !uestions becausethey have super#cial understanding of the problems of their #eld, they tend tofollow unproductive habits learned intheir dissertation research pro%ects, orthey misunderstand the changing normsand expectations of their researchcommunity.



4e need to stress that a good literaturereview is necessary but not su?cient forgood research. A good review of theliterature cannot guarantee either arigorous study or signi#cant #ndings. Oust because authors understands theresearch that others have done does not

mean that they will necessarily be able tocollect, analy"e, or interpret data well. 'tcertainly does not mean that theirinterpretation of prior research in the#eld will lead them to focus on researchproblems that will yield signi#cant andimportant studies. Hf course, even havinga signi#cant insight into the literature in a#eld does not guarantee that researcheswill then be able to do signi#cantresearch on the topic. But, again, it is

unliely if not impossible to do signi#cantresearch without productive insight intothe #eld.

't is productive insight thatdistinguishes a synthetic review, inIathers 92:::< sense, from the ploddingresearch summaries that characteri"emost dissertations. Productive insight cannever be routine. But we contend thatre!uiring doctoral candidates to engagein substantive, thorough, sophisticated

literature reviews creates and fostersconditions that will greatly increase thelielihood of their developing productiveinsight.

Loo(in% Forward

Doctoral students must be scholarsbefore they are researchers. Cirst andforemost, a dissertation shoulddemonstrate a thorough andsophisticated grasp of ones #eld of

study> secondarily and antecedently, itshould demonstrate the ability to doresearch that advances the collectiveunderstanding of important educationissues. +ducation research is di?cultbecause of the complex nature of thephenomena studied. 'n the face of perennial concerns about the !uality of education research and contemporarypressures to reform it, (.$. educationresearch %ournals have emphasi"ed

methods of data collection and analysisand related issues of epistemology. 'nturn, the emerging literature on preparingdoctoral students in education hasemphasi"ed methodologicalsophistication as the ey to improvingeducation research. *et to try to improve

education research by focusing onmethodological sophistication is to putthe cart before the horse.

esearchers cannot appropriatesophisticated research methods if theirunderstanding of the phenomena theyare investigating is rudimentary andunsystematic. To be able to identifyworable and potentially importantresearch problems 9ichardson, ;@@>$choenfeld, 2:::<, they must be able to

shift the problem to #nd perspectivesthat are progressively more explanatoryand insightful> they must become moresophisticated theoretically without beingsuper#cial 9$trie Posner, 2:J<. Thisre!uires much more than the meresummaries of existing literature that wefound in most of the dissertations weanaly"ed. Doctoral students should beexpected to move through Bloomsstages of cognitive development, from

comprehending to applying, to analy"ing,and on to synthesi"ing and evaluating97ranello, ;@@2<. Foreover, simplyexpecting students to meet these criteriaand standards is not enough&the sillsre!uired must be taught explicitly. *et the most obvious means of

improving the situation& adding a classon literature reviewing to doctoralprograms&is the least liely to bee5ective. $uch a curricular solution, as

Brit"man 92::2< noted about learning toteach, would leave the hardest tas&integrating and applying the lessons fromvarious classes in the doctoral program&to those who are least capable of doing it. That is, to review the literature in the waythat we have suggested here is a verycomplex tas that re!uires theintegration and application of a variety of sills and nowledge that few individualfaculty members have mastered.



Cor example, this approach to literaturereviewing re!uires advancedbibliographic methods for searching andlocating research from a variety of

sources, sills usually associated withlibrary instruction and the traditionaldomain of librarians. And, indeed, there isa healthy literature on the importance of doctoral faculty teaming with librarians,because neither alone typically have thesills and nowledge needed 9Bailey,2:J3> Bruce, ;@@2b> Kaspers Ienn,;@@@> 'sbell Broadsus, 2::3> Iibutti opala, 2::3<. *et even this approach is inade!uate,

because to do the ind of criticalsynthesis that we suggest also re!uiresdiverse sills on the part of doctoralfaculty. These include a substantiveunderstanding of the topic beingreviewed, the sills and nowledgere!uired to critically evaluate andsynthesi"e concepts, advancedunderstanding of writing and rhetoric,and the sophisticated understanding of research methodology that is re!uired to

critically evaluate methods used in priorstudies and suggest means of overcoming prior methodologicallimitations.

Cor reasons lie these, we suggest thata stand)alone class in literature reviewingis inade!uate. ather, literaturereviewing should be a central focus of predissertation coursewor, integratedthroughout the program. Demonstratingthe scholarly abilities re!uired for good

literature reviewing ought to be aprere!uisite for passing into candidacy.$uch a process will re!uire the %ointe5orts of not only sub%ect experts butalso librarians, writing teachers,methodologists, and perhaps others fromacross campus. The current interest in improving

doctoral education and educationresearch coincides with changes ininstruction on information access and

use. Hnly recently have librariesrede#ned library instruction, with itstraditional emphasis on mechanicalsearching sills, to include informationliteracy, which employs a moreconceptual approach to information use.

The Association of Kollege and esearchIibraries 9;@@@< de#nes informationliteracy as

an intellectual framewor forunderstanding, #nding, evaluating,and using information&activitieswhich may be accomplished in part byGuency with information technology, inpart by sound investigative methods,but most importantly, through criticaldiscernment and reasoning. 9pp. 1/<

The new focus of libraries on teachingstudents to critically engage withinformation o5ers the possibility ofsuccessful faculty1librarian collaboration,especially in the realm of graduateliterature reviewing and writing. 7reenand Bowser 9;@@< describe an initiativewhereby faculty and librarians guidedstudents in determining appropriatestudy subtopics, organi"ing literature

reviews, evaluating resources, andestablishing relationships with theliterature 9for other examples, see Bruce,;@@2> Kaspers Ienn, ;@@@> =eller)oss,2::0> $tein Iamb, 2::J> 4right, ;@@@<. Taing the idea a step further, 'sbell and

Broaddus 92::3< discuss the possibility of integrating writing instruction into theprocess. Fany doctoral students have notexplicitly studied writing and rhetoricsince their freshman composition classes.

Doctoral faculty expect doctoral studentsto possess not %ust bibliographic sills butalso advanced sills in the mechanics of writing and the art of rhetoric. *et thewriting in many of the dissertations weread was little more sophisticated thanthat of freshmen. Admittedly, this is partof a larger discussion, but the point isthat we cannot expect students to writemore sophisticated literature reviews if they lac sophisticated writing sills.



$tudents enter doctoral programs with arange of sills and abilities. $uccessfulsupervision and development of doctoralstudents entails integrating the expertiseof a range of university personnel.'ndividual faculty members cannot beresponsible for teaching the necessary

sills in isolation&these sills must beintegrated into the curriculum at theprogram level, and clearly communicatedand evaluated.

'f the dissertations we reviewed hadfailed to properly collect and analy"e datathe education community would blametheir authors or their dissertationcommittees. 4ith few exceptions, theprocedures and standards of datacollection and analysis used in

dissertations are well articulated andwidely disseminated. =owever, we arenot so fortunate in the case of literaturereviewing&the methods andexpectations of literature reviewing are atbest hapha"ardly described. Kooper92:J3, p. < asserts, 6$tudents ineducation . . . can tae #ve or sixstatistics or methods courses withoutever directly addressing the problemsand procedures of literature review.8 'f

Koopers claim remains true, the situationmust be remedied.

Hur concern is that by focusing onmethodological issues, the educationresearch community is addressing thesymptom rather than the cause. That is,researchers must understand priorresearch in their #eld, and its strengthsand weanesses, before they can beexpected to choose appropriate methodsof data collection and data analysis.

Foreover, sophisticated methods of datacollection and analysis are of little use if one is studying an unproductive problem. They are also of little use if one lacs thesophisticated understanding of theliterature needed to understand themeaning of the data. 'f doctoral programsand dissertation committees have not

been attending to the literature review asa ey component of a research pro%ect,we might #nd that increased attention tothis aspect of our tradecraft will in turnimprove the !uality and usefulness ofresearch.

Curther, if we, the education research

community, are to teach our doctoralstudents, then we must begin to valuethe literature review in our own wor.'magine if we were to devote one tenthas much energy, care, and thought tobeing better scholars as we do todeveloping our methods of datacollection and analysis. That we have notdone so is a symptom of the broaderculture of education research thatarti#cially distinguishes between

literature review, on the one hand, andmethods and analytic techni!ues, on theother.

As a result, empiricism andmethodological issues have beenascendant at the expense of scholarship,generativity, and theory building. Theori"ing is fundamental to researchand scholarship. 't is an understanding of the literature that leads to increasinglysophisticated in!uiry, connecting

research methods and claims with theirwarrants. That our doctoral candidatesoften graduate without a sophisticatedunderstanding of the literature in whichthey are supposed to be expert indicatesa failure not only of doctoral programsbut of the education research enterprisein general. Doctoral students need to seeus engaged in systematic analysis andsynthesis of the literature if they aregoing to value those activities for

themselves. Dissertation committeesmust hold the literature review tostandards at least as high as those formethodology&arguably higher. Hurfailure to do so will leave us withfragmented and dis%ointed research,unconnected to theory.

e!uiring doctoral students ineducation to approach the existing



literature in their #eld in the ways that wehave suggested is a means of inculcatingthe norms and practices of academicculture, with its emphasis on the analytic,the intellectual, the universal, and thetheoretical 9Iabaree, ;@@<. 4e suggestthat the standards and criteria of good

literature reviewing are part of the hiddencurriculum of good graduate programsand perhaps part of the tacit nowledgepassed on from mentors to candidates.Neither of these propositions can betested with our current data, and furtherresearch will be needed to understandthe pattern. 'ndeed, future research mayshow that a thorough, sophisticatedunderstanding of the #eld is whatseparates the best doctoral candidates

and education researchers from everyoneelse.

NTE"

The authors wish to than Aldrin$weeney, Bill 7audelli, Ann Austin, andthree anonymous reviewers for their veryhelpful suggestions.

2 Cor example, the Karnegie 'nitiativeon the Doctorate 9Karnegie Coundation,;@@< is investigating the structure of the

doctorate across six disciplines, includingeducation, and its implications for thefuture vigor, !uality, and integrity of the#eld. $imilar initiatives have been startedby the $pencer Coundation 9*oung, ;@@2<,the Pew Kharitable Trust 9;@@2<, the4oodrow 4ilson National CellowshipCoundation 9;@@2<, the National esearchKouncil 92:::<, and the National+ducation Association 9Iageman $hulman, 2:::<. 'n the Karnegie

Coundation initiative, for example,participating departments are fromnotable schools such as Ari"ona $tate(niversity> 'ndiana (niversity,Bloomington> Fichigan $tate (niversity>Hhio $tate (niversity> the (niversity of Kolorado, Boulder> the (niversity of Fichigan> the (niversity of NorthKarolina, Khapel =ill> the (niversity of $outhern Kalifornia> and 4ashington$tate (niversity. All of the participating

departments are housed in nationallyraned schools of education 9AmericasBest Kolleges, ;@@/<. +merging fromthese initiatives, recent chapters by$choenfeld 92:::< and $iddle 4aler92:::< and articles by Iabaree 9;@@<,Fet" 9;@@2<, Page 9;@@2<, Pallas 9;@@2<,

and ichardson 9;@@< have, in turn,addressed the problems of doctoraleducation and made suggestions forimprovement. This literature has initiated an important

dialogue in our #eld, but it is lacing forat least two reasons. Cirst, although theaforementioned initiatives suggesthelpful strategies for improving educationresearch, they ignore the fact that, of theapproximately E,@@@ recipients of

doctoral degrees awarded in educationevery year, only a small percentagegraduate from these nationally ranedinstitutions 9National Kenter for +ducation$tatistics, ;@@;<. 4e cannot assume thatthe experiences of doctoral students inthese resource)rich institutions arerepresentative of the experiences of mostdoctoral students. As a result, we need toloo carefully at any generali"ations andrecommendations that these initiatives

generate. $econd, although thereGections of leading scholars on doctoraleducation are valuable in initiatingconversations about improving doctoraleducation, they are only the beginningand must be followed by systematicexaminations of doctoral education.

; 't is worth noting that in the middle2:th century, (.$. universitiesmisunderstood the changes in 7ermanuniversities. At that time the new(.$. graduate research universitiesemphasi"ed the utility of empiricalresearch 9a theme that continues today<,whereas the 7erman universitiesemphasi"ed the importance of freelypursuing investigations, both empiricaland scholarly, without regard for theimmediate needs of society 9-eysey,2:03, p. 2;0<.

$peci#cally, =art 92:::< argued that



the dissertation literature review plays acentral role in

1 distinguishing what has been donefrom what needs to be done>

2 discovering important variablesrelevant to the topic>

3 synthesi"ing and gaining a new

perspective>4 identifying relationships between

ideas and practices>5 establishing the context of the topic

or problem>6 rationali"ing the signi#cance of the

problem>7 enhancing and ac!uiring the

sub%ect vocabulary>8 understanding the structure of the

sub%ect>

9 relating ideas and theory toapplications>

10 identifying the main methodologiesand research techni!ues that havebeen used>

11 placing the research in a historicalcontext to show familiarity

with state)of)the)art developments.9p. ;E< 4e found that these criteria gaveus a much more robust and thorough set

of criteria with which to evaluate the!uality of the dissertations we wereexamining. =owever, as we tried tooperationali"e the criteria, we found thatwe needed to combine some, rewriteothers, add several, and reorder the listto group similar ones. Cor example, wecombined =arts 92:::< Kriteria / and :into our Kriterion O, 6ationali"ed thepractical signi#cance of the researchproblem.8 Also, we omitted =arts

Kriterion J because it was too ambiguousto operationali"e and evaluate.

/ Kriterion = is the only one that ismeasured on a /)point scale. 4hile pilot)testing this rubric, we found that wecould not reliably use a /)point scale inmeasuring the other criteria, so weconverted all of the others to )pointscales. =owever, we needed a /)pointscale to include the possibility of anauthors proposing and %ustifying the

need for new research methods within a#eld of study.

3 A complete description of the researchmethodology can be found in Boote andBeile 9;@@/<. $ummary statistics areincluded here. Fean scores for theindividual items ranged from a low of

2.@J 9S! U .;:< on Kriterion A, 6Ousti#edcriteria for inclusion and exclusion fromreview8 to a high of ;. on threeseparate criteriaL 6Placed the research inthe historical context of the #eld8 9S! U .EJ<, 6Ac!uired and enhanced the sub%ectvocabulary8 9S! U ./:<, and 6Articulatedimportant variables and phenomenarelevant to the topic8 9S! U ./:<. 't wouldseem that, by common agreement, theselatter three criteria are expected in any

dissertation. And, indeed, these are themost common criteria listed in theintroductory research textboos. Hn theother hand, we were more troubled bythe low average scores in criteria such as6$ynthesi"ed and gained a newperspective on the literature8 9" U 2./;,S! U .0E<, 6'denti#ed the mainmethodologies and research techni!uesthat have been used in the #eld, andtheir advantages and disadvantages8 9"

U 2.:;, S! U .E:<, and 6ationali"ed thescholarly signi#cance of the researchproblem8 9" U 2.:;, S! U .E:<.

'nter)university di5erences alsorevealed potential concerns. $cores oneach of the twelve criteria were averagedto arrive at an overall !uality score forthe literature review. The mean of theliterature review !uality scores rangedfrom a low of 2./; to a high of ;.:2 9" U;.@:, S! U .3@<. By institution, the top)raned colleges literature review !ualityscores 9" U ;.2;, S! U .33< ranged from2.0E to ;.:2> the mid)tier colleges scores9" U ;./@, S! U ./2< ranged from 2./; to;.:2> and the nonraned colleges scores9" U 2.E, S! U .E< ranged from 2./; to;.;3. A rusal)4allis calculation, #9;< U.:@, p U .2/, revealed no statisticallysigni#cant di5erence among institutions,yet practical di5erences are indicated bythe nonraned colleges average



placement of .JJ, as compared with thetop)tiered colleges average of 0.E3 andthe mid)tier colleges average of J.JJ.

0 Passmore 92:J@< reminds us that 6thePh.D. did not, of course, create pedantry892:J@, p. 3<.

REFERENCE"

Alton)Iee, A. 92::J<. A troubleshooters

checlist for prospective authors derived from reviewers critical

feedbac. Teaching and TeacherEducation$ %9J<, JJE1J:@.

Americas Best Kolleges. 9;@@/<. USNews & World Report$ etrieved Oanuary/, ;@@3, fromhttpLwww.usnews.comusnewsranguide rghome.htm

Babbie, +. 92::J<. The practice of socialresearch 9Jth ed.<. Belmont, KAL4adsworth.

Bailey, B. 92::3<. Thesis practicum andthe librarians role. 'ournal of (cademic)irarianship$ %%$ E:1J2.

Barger, . ., Duncan, O. . 92:J0<.Kreativity in doctoral researchL Areasonable expectationS TheEducational *orum, +%92<, 1/.

Barry, K. A. 92::E<. 'nformation sills foran electronic worldL Training doctoralresearch candidates. 'ournal of ,nformation Science$ .9<, ;;31;J.

Berelson, B. 92:0@<. /raduate educationin the United States. New *orL Fc7raw)=ill.

Berliner, D. K. 9;@@;<. +ducationalresearchL The hardest science of all.Educational Researcher$ .%9J<, 2J1;@.

Boote, D. N., 7audelli, 4. 9;@@;<. Theuse and abuse of historical educationaltheoristsL Komments on recent Deweyscholarship0 ,nsights1 ( pulication of the 'ohn !ewey Society for the Study of Education and 2ulture$ .+9;<, :12.

Boote, D.N., Beile, P. 9;@@/, April<. The3uality of dissertation literaturereviews1 ( missing lin4 in research preparation. Paper presented at theannual meeting of the American+ducational esearch Association, $an

Diego, KA.Brit"man, D. 92::2< 5ractice ma4es practice1 ( critical study of learning toteach. AlbanyL $tate (niversity of New *or Press.

Bruce, K. $. 92::/<. esearch studentsearly experiences of the dissertationliterature review. Studies in #igher Education$ %69;<, ;2E1;;:.

Bruce, K. $. 9;@@2a<. 'nterpreting thescope of their literature reviewsL$igni#cant di5erences in researchstudents concerns. New )irary World$%79/<, 23J1200.

Bruce, K. $. 9;@@2b<. Caculty1librarianpartnerships in Australian highereducationL Kritical dimensions.Reference Service Review$ 6, 2@01223.

Karnegie Coundation. 9;@@<. 2arnegieinitiative on the doctorate. etrieved Oanuary /, ;@@3, fromhttpLwww.carnegiefoundation.orgK'Dindex.htm

Kaspers, O., Ienn, . 9;@@@<. The futureof collaboration between librarians andteaching faculty. 'n D. aspa D. 4ard9+ds.<, The collaorative imperative1)irarians and faculty wor4ing together in the information universe 9p. 2/J123/<. KhicagoL American IibraryAssociation.

Kooper, =. F. 92:J3<. ( taxonomy of literature reviews0 Paper presented atthe annual meeting of the American+ducational esearch Association,Khicago. +'K Document eproduction$ervices No. +D;3/3/2.

Kreswell, O. 4. 92::/<. Research design18ualitative and 3uantitative



approaches) Thousand Has, KAL $A7+.Kreswell, O. 4. 9;@@;<. Educational

research1 5lanning$ conducting$ andevaluating 3uantitative and 3ualitativeresearch. (pper $addle iver, NOL FerrillPrentice =all.

Delamont, $., Atinson, P., Parry, H.

9;@@@<. The doctoral experience1Success and failure in graduate school0New *orL Calmer.

Dong, *. . 92::J<. Nonnative graduatestudents thesis dissertation writing inscienceL $elf)reports by students andtheir advisors from two ($ institutions.English for Speci9c 5urposes$ %:9/<,0:1:@.

Due, N. ., Bec, $. 4. 92:::<.+ducation should consider alternative

forms for the dissertation. EducationalResearcher$ ;9<, 210.

Craenel, O. ., 4allen, N. +. 9;@@<.#ow to design and evaluate research ineducation 93th ed.<. BostonL Fc7raw)=ill=igher +ducation.

7alvan, O. I. 9;@@/<. Writing literaturereviews1 ( guide for students of thesocial and ehavioral sciences 9;nded.<. Ios AngelesL Pyrc"a.

7ay, I. ., Airasian, P. 4. 9;@@@<.

Educational research1 2ompetencies for analysis and application0 (pper $addleiver, NOL Ferrill.

7ranello, D. =. 9;@@2<. Promotingcognitive complexity in graduate writtenworL (sing Blooms taxonomy as apedagogical tool to improve literaturereviews. 2ounselor Education andSupervision$ 79/<, ;:;1@E.

7rant, K. A., 7raue, +. 92:::<.9e<-iewing a reviewL A case history of

the 6eview of +ducational esearch.8Review of Educational Research, <69/<,J/1:0.

=art, K. 92:::<. !oing a literature review1Releasing the social science researchimagination0 IondonL $A7+.

=%orland, B. 92:JJ<. 'nformation retrievalin psychologyL 'mplications of a casestudy. =ehavioral and Social Sciences)irarian$ <91/<, :10/.

'sbell, D., Broaddus, D. 92::3<.

Teaching writing and research asinseparableL A faculty1librarian teachingteam. Reference Services )irarian$ ;$31;3.

Oohnston, $. 92::E<. +xamining theexaminersL An analysis of examinersreports on doctoral theses. Studies in

#igher Education$ 9<, 1/E.lein, P. 92:::<. eopening in!uiry into

cognitive processes of learning)to)write.Educational 5sychology Review$ %%$;@1;E@.

uhn, D. 92::;<. Thining as argument.#arvard Educational Review$ <9;<,23312EJ.

uhn, T. $. 92:E@<. The structure of scienti9c revolutions 9;nd ed.<. KhicagoL(niversity of Khicago Press.

Iabaree, D. C. 9;@@<. The peculiarproblems of preparing educationalresearchers. Educational Researcher ,.92<, 21;;.

Iagemann, +. K., $hulman, I. $. 92:::<.,ssues in education research1 5rolemsand possiilities) $an CranciscoL Oossey)Bass.

Iather, P. 92:::<. To be of useL The worof reviewing. Review of EducationalResearch$ <692<, ;1E.

IeKompte, F. D., lingner, O. .,Kampbell, $. A., Fen, D. 4. 9;@@<.+ditors introduction. Review of Educational Research$ :.9;<$ 2;12;/.

Iester, O. D. 9;@@;<. The essential guide1Research writing across the disciplines.New *orL Iongman.

Iibutti, P., opala, F. 92::3<. Thedoctoral student, the dissertation, andthe libraryL A review of the literature.Reference )irarian$ ;$ 31;3.

Fauch, O. +. 92::J<. /uide to thesuccessful thesis and dissertation1 (handoo4 for students and faculty . New *orL F. Deer.

FcFillan, O. =., $chumacher, $. A.9;@@2<. Research in education1 (conceptual introduction 93th ed.<. New *orL Iongman.

Fet", F. =. 9;@@2<. 'ntellectual bordercrossing in graduate educationL A reportfrom the #eld. Educational Researcher$



.793<, 2;12J.Fullins, 7., iley, F. 9;@@;<. 6'ts a PhD,

not a Nobel Pri"e8L =ow experiencedexaminers assess research theses.Studies in #igher Education , :9/<, 0:1J0.

National Kenter for +ducation $tatistics.

9;@@;<. !igest of education statistics$77%0 4ashington, DKL (.$. Departmentof +ducation.

National esearch Kouncil. 92:::<.,mproving student learning1 ( strategic plan for education research and itsutili>ation) 4ashington, DKL NationalAcademy Press.

Nicerson, +.T. 92::<. The dissertationhandoo41 ( guide to successfuldissertations. Dubu!ue, 'AL

endall=unt.Nightingale, P. 92:J/<. +xamination of

research theses. #igher EducationResearch and !evelopment$ .$ 2E123@.

Hgden, +. =. 92::<. 2ompleting your doctoral dissertation or master?s thesis

in two semesters or less. Iancaster, PAL Technomic.

Page, . N. 9;@@2<. eshaping graduatepreparation in educational researchmethodsL Hne schools experience.Educational Researcher$ .793<, 2:1;3.

Pallas, A. F. 9;@@2<. Preparing educationdoctoral students for epistemologicaldiversity. Educational Researcher$ .793<,0122.

Paltridge, B. 9;@@;<. Thesis anddissertation writingL An examination of published advice and actual practice.English for Speci9c 5urposes$ %$ 2;312/.

Pan, F. I. 9;@@<. 5reparing literaturereviews1 8ualitative and 3uantitative

approaches0 Ios AngelesL Pyrc"a.Passmore, O. 92:J@<. The philosophy of

graduate education. 'n 4. . Cranena9+d.<, The philosophy and future of graduate education 9pp. /@13:<. AnnArborL (niversity of Fichigan Press.

Pew Kharitable Trust. 9;@@2<. Re-envisioning the 5h0!) etrieved Oanuary/, ;@@3, fromhttpLwww.grad.washington.eduenvision

Pitethly, A., Prosser, F. 92::3<.+xaminers comments on theinternational context of PhD theses. 'nK. FcNaught . Beattie 9+ds.<,Research into higher education1!ilemmas$ directions and diversions9pp.2;:120<. FelbourneL =igher+ducation esearch and Development$ociety of Australasia -ictoria.

ichardson, -. 9;@@<. The 5h0!0 ineducation0 etrieved Oanuary /, ;@@3,

fromhttpLwww.carnegiefoundation.orgK'DessaysK'DV educVichardson.pdf

ivard, I. P. 92::/<. A review of writing tolearn in scienceL 'mplications forpractice and research. 'ournal of Research in Science Teaching$ .%$ :0:1:J.

$choenfeld, A. =. 92:::<. The core, the

canon, and the development of researchsills. 'n +. K. Iagemann I. $.$hulman 9+ds.<, ,ssues in educationresearch1 5rolems and possiilities9pp.2001;;<. $an CranciscoL Oossey)Bass.

$chools of +ducation. 9;@@@, April 2@<. USNews & World Report$ '2/.$hulman, I. $. 92:::<. Professing

educational scholarship. 'n +. K.Iagemann I. $. $hulman 9+ds.<,

,ssues in education research1 5rolemsand possiilities 9pp.23:1203<. $anCranciscoL Oossey)Bass.

$iddle 4aler, -. 92:::<. Kulture andcommitmentL Khallenges for the futuretraining of education researchers. 'n +.K. Iagemann I. $. $hulman 9+ds.<,,ssues in education research1 5rolemsand possiilities 9pp. ;;/1;//<. $anCranciscoL Oossey)Bass.

$torr, . O. 92:0:<. The eginning of



graduate education in (merica. New *orL Arno Press and New *or Press.

$trie, ., Posner, 7. 92:J<. Types of synthesis and their criteria. 'n

$. 4ard and I. eed 9+ds.<, nowledgestructure and useL 'mplications for synthesis and

interpretation 9pp. /10;<.PhiladelphiaL Temple (niversityPress. -eysey, I. . 92:03<. Theemergence of the (mericanuniversity . KhicagoL (niversityof Khicago Press.

4ideen, F. C., Fayer)$mith, O., Foon, B.92:::<. A critical analysis of theresearch on learning to teachL Faingthe case for an ecological perspectiveon in!uiry. Review of Educational

Research$ <;9;<, [email protected], ., 7ri?ths, F., 7reen, .

9;@@@<. The 6academic8 !ualities ofpracticeL 4hat are the criteria for apractice)based PhDS Studies in #igher Education$ +92<, ;31E.

4oodrow 4ilson National CellowshipCoundation. 9;@@2<. The responsive5h!0 etrieved Oanuary /, ;@@3, from

httpLwww.woodrow.orgresponsivephd

*oung, I. O. 9;@@2<. Border crossings andother %ourneysL e)envisioning thedoctoral preparation of educationresearchers. Educational Researcher$.793<, 13.

aporo"het", I. +. 92:JE<. Thedissertation literature review1 #owfaculty advisors prepare their doctoralcandidates0 (npublished doctoral

dissertation, (niversity of Hregon,+ugene.

A*T+R"

DA-'D N. BHHT+ is an Assistant Professorof Kurriculum $tudies in the Kollege of +ducation, (niversity of Kentral Clorida,Hrlando, CI ;J20)2;3@>dbooteWmail.ucf.edu. =is research

interests include education researchstudies, teacher educators, mathematicsand physics problem solving, and moraleducation.

P+NN* B+'I+ is an Associate Iibrarian atthe (niversity of Kentral Clorida, P.H. Box

20;000, Hrlando, CI ;J20);000>pbeileWmail. ucf.edu. =er areas of speciali"ation are the production and useof scholarly information in education andthe information literacy abilities of education students. $he is completingthe development and validation of the'nformation Iiteracy Assessment $cale for+ducation 9'IA$)+D<.

Fanuscript receivedDecember 2@, ;@@

evisions received Oanuary/, April ;J, ;@@3 Accepted

Fay 0, ;@@3

litereture review article

Documents