introduction to education design research

An Introduction to Educational Design Research

SLO Netherlands institute for curriculum developmentSLO

PO box 20417500 CA EnschedeThe NetherlandsT +31(0)53 484 08 40F +31(0)53 430 76 92E [email protected]

SLO is the Netherlands institute for curriculum development. We are bridging the contexts of policy, research, and practice. Our expertise focuses on the development of curricular goals and content for various educational levels, from national policy to classroom practices. We closely collaborate with many different stakeholders from policy circles, schools (boards, principals, teachers), research, civic organizations, and the society at large. This allows us to design and validate relevant curriculum frameworks, to elaborate exemplary materials and to evaluate these in school practices. Our products and services support both policy makers and schools and teachers in making substantive curricular decisions and in elaborating these into relevant, inspiring and effective education.

ISBN: 978 90 329 2329 7

An IntroductIon to EducAtIonAl dEsIgn rEsEArchEditors: tjeerd Plom

p & nienke nieveen

Editors:Tjeerd Plomp & Nienke Nieveen

An Introduction to Educational Design Research

SLO Netherlands institute for curriculum development

Tjeerd Plomp & Nienke Nieveen (editors)

Proceedings of the seminar conducted at the East China Normal University, Shanghai (PR China), November 23-26, 2007

an introduction to educational design research2

Authors: JanvandenAkker BrendaBannan AnthonyE.Kelly NienkeNieveen TjeerdPlompEditors: TjeerdPlomp NienkeNieveen

Design: AxisMedia-ontwerpers,Enschede

Production: Netzodruk,Enschede

AN: 7.5115.183

ISBN: 9789032923297

Orderaddress: SLO P.O.Box2041 7500Enschede theNetherlands

www.slo.nl/organisatie/international/publications

Enschede,3rdprintMarch2010

Colophon

an introduction to educational design research 3

Contents

Preface 5

1. EducationalDesignResearch:anIntroduction 9 Tjeerd Plomp

2. CurriculumDesignResearch 37 Jan van den Akker

3. TheIntegrativeLearningDesignFramework:AnIllustratedExamplefrom theDomainofInstructionalTechnology 53 Brenda Bannan

4 WhenisDesignResearchAppropriate? 73 Anthony E. Kelly

5 FormativeEvaluationinEducationalDesignResearch 89 Nienke Nieveen

6 ReferencesandSourcesonEducationalDesignResearch 103 Tjeerd Plomp and Nienke Nieveen

Authorbiographies 125


PrefaceThisbookistheresultofaseminaroneducationaldesignresearchorganizedfromNovem-ber23-26,2007,byProfZhuZhiting(DepartmentofEducationalTechnology)oftheCollegeofEducationalSciencesattheEastChinaNormalUniversityinShanghai(PRChina).Theprimarygoaloftheseminarwasto introduce a group postgraduate students and lectu-ring staff in China to educational design research as a research approach.Thesecondgoaloftheseminarwastoprepare,basedonthecontributionsofanumberinternationalexperts,proceedings of the seminarwritteninsuchawaythattheycanbeusedinpostgraduateseminarsoneducationaldesignresearchacrossChina.

About75peoplewithbackgroundsmainlyininstructionaltechnology,curriculumandinstructionaldesignparticipatedintheseminar.Mostofthemwereworkinginteachereducation,inschoolsasinstructionaltechnologistand/orindistanceeducation.Althoughparticipantshad(throughtheirstudies)alreadyknowledgeandsomeexperienceinin-structionalorcoursedesignandinresearchmethods,theywereeagertobeintroducedtodesignresearchasarelativelynewresearchapproachforaddressingcomplexproblemsineducationalpractice.

TheseminarstaffconsistedofProfsBrenda BannanandEamonn Kelly(bothGeorgeMasonUniversity,Fairfax,VA,USA)andProfJan van den Akker(UniversityofTwenteandNationalInstituteforCurriculumDevelopment[SLO],Enschede,TheNetherlands),andthetwoedi-torsofthisbookDrNienke Nieveen(NationalInstituteforCurriculumDevelopment[SLO],Enschede)andProfTjeerd Plomp(UniversityofTwente,Enschede,TheNetherlands).Ascanbeseenfromthetableofcontentofthisbook,theyarereflectingthebackgroundofthepar-ticipants,astheyrepresentedexperienceinconductingdesignresearchinthedomainsofcurriculumdevelopment,instructionaltechnologyandmathematicsandscienceeducation.ExpertswereconsciouslyinvitedfrombothEurope(TheNetherlands)aswellastheUSA,soastoascertainthatvariationinbackgroundandperspectiveondesignresearchwasrepre-sentedinconductingtheseminar.

Thechaptersinthisbookarebasedonthepresentationsandthesmallgroupdiscussionsduringthisseminar.Althoughthebookdoesnotprovideahowtodoguidefordesigningandconductingdesignresearch,thechaptershavebeenwritteninsuchawaythattheyreflectboththeconceptualunderpinningandpracticalaspectsofthewhatandhowofdoingdesignresearch(chaptersbyPlomp,KellyandNieveen),aswellasprovidethereaderaninsightinthespecificsofdoingdesignresearchinthedomainofcurriculum(chapterbyVandenAkker)andinstructionaltechnology(chapterbyBannan).Toassistthereadersinfindingtheirwayintheabundanceofliteratureondesignresearch,wehaveaddedachapterwithreferencesandsourcesoneducationaldesignresearch.This


bibliographyisfarfromcompleteandreflectsverymuchthebackgroundandthebiasesoftheeditorsofthisbook.Yetwetrustthatthischapterwillassisttheinterestedreaderingettingintroducedtothisexcitingandpromisingresearchapproach.

WewanttothankProfZhuZhitingfromtheEastChinaNormalUniversityfortakingtheinitiativeforthisseminar.Similarlywewanttothankourcolleaguesforcontributingtothisbook.

Butaboveall,weliketoexpressourhopethatthisbookwillstimulateandsupportmany(future)researcherstoengagethemselvesineducationaldesignresearch.

JanvandenAkkerDirector General SLO

TjeerdPlompandNienkeNieveenEditors


1. EducationalDesignResearch: anIntroduction Tjeerd PlompIntroduction

Thepurposeofthischapteristoprovideanintroductiontoeducationaldesignresearchasaresearchapproachsuitabletoaddresscomplexproblemsineducationalpracticeforwhichnoclearguidelinesforsolutionsareavailable.Educationaldesignresearchisperceivedasthesystematicstudyofdesigning,developingandevaluatingeducationalinterventions,-suchasprograms,teaching-learningstrategiesandmaterials,productsandsystems-assolutionstosuchproblems,whichalsoaimsatadvancingourknowledgeaboutthecharacteristicsoftheseinterventionsandtheprocessestodesignanddevelopthem.

Theneedforaresearchapproachthataddressescomplexproblemsineducationalpracticehasbeenarguedbyresearchersinvariouscornersofthedomainofeducationfromthelackofrelevanceofmucheducationalresearchforeducationalpractice.Forexample,theDesign-BasedResearchCollective(2003:5)arguesthateducationalresearchisoftendivorcedfromtheproblemsandissuesofeverydaypracticeasplitthatresultedinacredibilitygapandcreatesaneedfornewresearchapproachesthatspeakdirectlytoproblemsofpracticeandthatleadtothedevelopmentofusableknowledge.Fromhisbackgroundinresearchinthedomainofcurriculum development and implementation,VandenAkker(1999:2)arguesthatmanytraditionalresearchapproachessuchasexperiments,surveys,correlationalanalyses,withtheiremphasisondescriptionhardlyprovideprescriptionsthatareusefulfordesignanddevelopmentproblemsineducation.Heclaimsthatanimportantreasonfordesignresearch1stemsfromthecomplexnatureoftheeducationalreformsworldwide.Ambitiousreformscannotbedevelopedatthedrawingtablesingovernmentoffices,butcallforsystematicresearchsupportingthedevelopmentandimplementationprocessesinavarietyofcontexts.Inhisreviewofthestateofeducationalresearchandmorespecificallyeducationaltechnologyresearch,Reeves(2006:57)concludesthatthereisalegacyofill-conceivedandpoorlyconductedresearchthatresultsinnosignificantdifferencesor,atbest,inmodesteffectsizes.Healsoarguesforthedomainofeducational technologythateducationaltechnologists,insteadofdoingmore(media)comparisonstudies,shouldundertaketypesofdesignresearch.Inotherwords,ReevesarguesthatinsteadofdoingmorestudiescomparingwhetherinacertaincontextmethodAisbetterthanmethodB,itisbetterto

1) whichhecallsdevelopmentresearchinhis1999publication


undertakedesignresearchaimedatdevelopinganoptimalsolutionforaproblemincontext.Inthefieldoflearning sciences,thebeliefthatcontextmattersleadtotheconclusionthatresearchparadigmsthatsimplyexamineslearningprocessesasisolatedvariableswithinlaboratorysettingswillnecessarilyleadtoanincompleteunderstandingoftheirrelevanceinmorenaturalisticsettings(Barab&Squire,2004;withreferencetoBrown,1992).Inthisfield,design-basedresearchwasintroducedwiththeexpectationthatresearcherswouldsystematicallyadjustvariousaspectsofthedesignedcontextsothateachadjustmentservedasatypeofexperimentationthatallowedtheresearcherstotestandgeneratetheoryinnaturalisticcontexts(Barab&Squire,2004:3).

Thesesourcesillustratetheneedfordesignresearchasanalternativeresearchapproach.Beforeelaboratingondesignresearchthispaperwillfirstdiscussmoregenerallypossiblefunctionsofresearchandhowresearchfunctionsarerelatedtoresearchapproaches.Thendesignresearchwillbedefinedandcharacterizedfromvariousperspectives,suchasthetypeofknowledgethedesignresearchersaimfor,thetypeofresearchquestionsthatcanbeaddressed,andtheoutputsofdesignresearch.Thiswillbefollowedbyasectioninwhichdifferentapproachestodesignresearchareintroducedandsectionsdiscussinghowdesignresearchcanorshouldbeconducted,withamorein-depthdiscussionofformativeevaluationasthemostprominentresearchactivityindesignresearch.Conductingdesignresearchputsresearchersinasituationinwhichtheyhavetofaceanumberofdilemmas.Thesewillbediscussedbeforeendingthechapterwithafewconcludingremarks.

Afinalnoteonterminology,followingVandenAkkeretal.(2006:4)weusedesign researchasacommonlabelforafamilyofrelatedresearchapproacheswhomayvarysomewhatingoalsandcharacteristicsexamplesaredesignstudies,designexperiments,design-basedresearch,developmentalresearch,formativeresearch,engineeringresearch.

Research functions research approaches

Beforeelaboratingonthemeaningofdesignresearch,itisimportanttopositiondesignresearchasaresearchapproachnexttootherresearchapproaches,whichisthepurposeofthissection.

Thekeyfocusinallscientificresearchisthesearchforunderstandingorforknowingwiththeaimofcontributingtothebodyofknowledgeoratheoryinthedomainofresearch.Otherbroad aimsofdoingeducationalresearcharetoprovideinsightsandcontributionsforimprovingpractice,andtoinformdecisionmakingandpolicydevelopmentinthedomainofeducation.


ResearchfunctionsIngeneral,wecandistinguishvariousresearch functions,eachreflectingcertaintypesofresearchquestions.Examplesofresearchfunctions(withexemplaryresearchquestionsfittingthefunction)are:1. to describe: e.g.whatistheachievementofChinesegrade8pupilsinmathematics;

whatbarriersdostudentsexperienceinthelearningofmathematicalmodelling2. to compare:e.g.whatarethedifferencesandsimilaritiesbetweentheChineseandthe

Netherlandscurriculumforprimaryeducation;whatistheachievementinmathematicsofChinesegrade8pupilsascomparedtothatincertainothercountries

3. to evaluate:e.g.howwelldoesaprogramfunctionintermsofcompetencesofgraduates;whatarethestrengthsandweaknessesofacertainapproach;etc

4. to explain or to predict:e.g.whatarethecausesofpoorperformanceinmathematics(i.e.insearchofatheorypredictingaphenomenonwhencertainconditionsorcharacteristicsaremet)

5. to design and develop:e.g.whatarethecharacteristicsofaneffectiveteachingandlearningstrategyaimedatacquiringcertainlearningoutcomes;howcanweimprovethemotivationoflearners.

Inmanyresearchprojectstheresearchquestionsaresuchthatinfactvariousresearchfunctionsdoapply.Forexample,iftheresearchquestionpertainstocomparingthemathematicsachievementofChinesegrade8pupilsascomparedtothatincertainothercountries,thenaspartofcomparingtheresearcherswillevaluatetheachievementofgrade8pupilsineachofthecountriesinvolved.Or,asanotherexample,ifonewantstodesign and developateaching-learningstrategyforacquiringthecompetencyofmathematicalmodelling(ingrade11&12),thenresearchersmayfirstwanttounderstandandcarefullydescribewhatbarriersstudentsexperiencewithmathematicalmodelling,whilstalsotheevaluationfunctionisimportantindeterminingwhethertheteaching-learningstrategythathasbeendevelopediseffective.Bothexamplesillustratethatusuallyaresearchprojecthasaprimaryresearchfunction,butthatotherresearchfunctionsarebeingappliedtoservetheprimaryresearchfunction.

Atthelevelofaresearch project,startingfromaresearchproblemorquestion,wearesupposedtohavethefollowingsequence:Research question => (primary) research function =>choice of research approach.Inthischapterwefocusonresearchwhichhasdesign and developastheprimaryresearchfunction.

ResearchapproachesMosttextbooksonresearchmethodologypresentanddiscussanumberofresearch approachesorstrategies(seee.g.Denscombe,2007).Usuallyeachresearchapproachcanbe


usedforrealizingmorethanoneresearchfunction.Withoutgoingintodetailhere,examplesofresearchapproachesandtheirpossibleresearchfunctionsare: survey: todescribe,tocompare,toevaluate case studies:todescribe,tocompare,toexplain experiments:toexplain,tocompare action research:todesign/developasolutiontoapracticalproblem ethnography:todescribe,toexplain correlational research:todescribe,tocompare evaluation research:todeterminetheeffectivenessofaprogramTextbooksonresearchmethodologyusuallydonotpresentanddiscussdesignresearch: design research: todesign/developanintervention(suchasprograms,teaching-learning

strategiesandmaterials,productsandsystems)withtheaimtosolveacomplexeducationalproblemandtoadvanceourknowledgeaboutthecharacteristicsoftheseinterventionsandtheprocessestodesignanddevelopthem.

Inlinewiththeremarkthatmorethanoneresearchfunctionmayhavetobeappliedtoaddressaresearchquestion,itshouldbenoticedthatinaresearchprojectmorethanoneresearchapproachmayhavetobeapplied.Forexample,ifthereisaneedtocomparehowwellChinesegrade8pupilsperforminmathematicsascomparedtoanumberofothercountries,theprimaryresearchfunctionistocompare,leadinginthiscasetoasurveyasthebestresearchapproach.However,aspartofthedevelopmentofavalidandreliablemathematicstest,theresearchersmaydocorrelationalresearchtodeterminewhetherthetestbeingdevelopedisvalid,i.e.correlateswithothermeasuresofmathematicsachievement.

Asafinalremark,itisimportantthatdesignresearchers,likeallresearchers,keepinmindthatalsofortheirresearchtheguidingprinciplesforscientificresearch(Shavelson&Towne,2002)apply,viz: Posesignificantquestionsthatcanbeinvestigated Linkresearchtorelevanttheory Usemethodsthatpermitdirectinvestigationofthequestion Provideacoherentandexplicitchainofreasoning Replicateandgeneralizeacrossstudies Discloseresearchtoencourageprofessionalscrutinyandcritique


What is design research?

Asstatededucationaldesignresearchisthe systematic study of designing, developing and evaluating educational interventions (such as programs, teaching-learning strategies and materials, products and systems) as solutions for complex problems in educational practice, which also aims at advancing our knowledge about the characteristics of these interventions and the processes of designing and developing them. Thetwofoldyieldofdesignresearch,viz.researchbasedinterventionsaswellasknowledgeaboutthem,canalsofoundindefinitionsofdesignresearchbyotherauthors.Forexample,thebroaddefinitionofBarabandSquire(2004)alsoencompassesmostvariationsofeducationaldesignresearch:aseriesofapproaches,withtheintentofproducingnewtheories,artefacts,andpracticesthataccountforandpotentiallyimpactlearningandteachinginnaturalisticsetting.

Byitsnature,designresearchisrelevantforeducationalpractice(andthereforealsoforeducationalpolicy)asitaimstodevelopresearch-basedsolutionsforcomplexproblemsineducationalpractice.Startingpointfordesignresearchareeducationalproblemsforwhichnooronlyafewvalidatedprinciples(howtodoguidelinesorheuristics)areavailabletostructureandsupportthedesignanddevelopmentactivities2.Informedbypriorresearchandreviewofrelevantliterature,researchersincollaborationwithpractitionersdesignanddevelopworkableandeffectiveinterventionsbycarefullystudyingsuccessiveversions(orprototypes)ofinterventionsintheirtargetcontexts,andindoingsotheyreflectontheirresearchprocesswiththepurposetoproducedesignprinciples.Manyexamplesoftheneedforinnovativeinterventionscanbegivenatsystemlevelandinstitutionallevel.Atsystemlevel,forexample,onemaywanttodevelopasystemfore-learningtoserveaspecifictargetgroupofstudentsinhighereducation,andatthelevelofschoolorclassroomonemaywant,forexample,toaddressthequestionofwhatareeffectivemethodsforcollaborativelearning.SeealsoGustafson&Branch(2002)whodevelopedataxonomyofinstructionaldevelopmentmodelsbasedonaselectedcharacteristics;theydistinguishbetweenmodelswithaclassroomorientation,productorientationandsystemorientation.

Theresearchprocessindesignresearchencompasseseducationaldesignprocesses.Itislikeallsystematiceducationalandinstructionaldesignprocesses-thereforecyclicalincharacter:analysis,design,evaluationandrevisionactivitiesareiterateduntilasatisfyingbalancebetweenideals(theintended)andrealizationhasbeenachieved.

2) seealsothechapterofKellyinthisbookwherehediscusseswhendesignresearchisappropriate.


Thisprocesscanbeillustratedinvariousways.Justafewexamplesarepresentedheretoshowhowdifferentauthorshavevisualizedtheresearchprocess.

Reeves(2006)depictsthedesignresearchapproachasfollows:

Figure 1: Refinement of Problems, Solutions, Methods, and Design Principles (Reeves, 2000, 2006)

McKenney(2001)illustratesinherstudythiscyclicalprocessasfollows:

Figure 2: Display of the CASCADE-SEA study (McKenney, 2001)

Identify and analyse problems by researchers & practitioners in collaboration

Development of prototype solutions: informed by state-

of-art theory, existing design principals & technology innova-

tions

Iterative cycles of testing & refinement

of solutions in practice

Reflection to produce design principles & enhance solution implementation in

practice

160

140

120

100

80

60

40

20

0

numbe

r of p

artic

ipan

ts

needs & contextanalysis

design, development & formative evaluation semi-summativeevaluation

literaturereview &conceptvalidation

sitevisits

prototype1

prototype2

cycle width is proportional to timeschale: = circa 6 months

prototype3

prototype4

finalevaluation

query


ThequeryasthelastphaseinMcKenneysdisplaycanbeinterpretedasthereflectionboxinthemodelofReeves(Figure1).AnotherexampleistheIntegrativeLearningDesignFrameworkthatBannan-Ritlandpresentsinchapter5ofthisbook(seealsoBannan-Ritland,2003).

Authorsmayvaryinthedetailsofhowtheypicturedesignresearch,buttheyallagreethatdesignresearchcomprisesofanumberofstagesorphases: preliminary research:needsandcontextanalysis,reviewofliterature,developmentofa

conceptualortheoreticalframeworkforthestudy prototyping phase:iterativedesignphase3consistingofiterations,eachbeingamicro-

cycleofresearch4withformativeevaluationasthemostimportantresearchactivityaimedatimprovingandrefiningtheintervention

assessment phase:(semi-)summativeevaluationtoconcludewhetherthesolutionorinterventionmeetsthepre-determinedspecifications.Asalsothisphaseoftenresultsinrecommendationsforimprovementoftheintervention,wecallthisphasesemi-summative.

Throughoutalltheseactivitiestheresearcherorresearchgroupwilldosystematic reflection and documentationtoproducethetheoriesordesignprinciples(aconcepttakenfromVandenAkker,1999seealsochapter2)asthescientificyieldfromtheresearch.Onemaystatethatthissystematicreflectionanddocumentationmakesthatsystematicdesignanddevelopmentofaninterventionbecomesdesignresearch.

Authorsaboutdesignresearchalsoagreeanumberofcharacteristicsofthistypeofresearch.ThesearesummarizedbyVandenAkkeretal.(2006:5): Interventionist:theresearchaimsatdesigninganinterventioninarealworldsetting; Iterative:theresearchincorporatescyclesofanalysis,designanddevelopment,

evaluation,andrevision; Involvement of practitioners:activeparticipationofpractitionersinthevariousstages

andactivitiesoftheresearch Process oriented:thefocusisonunderstandingandimprovinginterventions(ablack

boxmodelofinputoutputmeasurementisavoided); Utility oriented:themeritofadesignismeasured,inpartbyitspracticalityforusersin

realcontexts;and Theory oriented:thedesignis(atleastpartly)basedonaconceptualframeworkand

upontheoreticalpropositions,whilstthesystematicevaluationofconsecutiveprototypesoftheinterventioncontributestotheorybuilding.

3) itispossiblethatthedesign/developmentcomponentinasucharesearchprojectwillnotbeginfromscratchbutwiththeevaluationofanexistinginterventionwiththeaimofidentifyingtheneedforimprovement,whichthenisfollowedbyre-designandanumberofdesigncycles.

4) termtakenfromBannan-Ritland,chapter5


ThefeaturesandcharacteristicsofdesignresearcharenicelycapturedbyWademan(2005)inwhathecallstheGenericDesignResearchModel(Figure3).Hismodelclearlyillustratesthatthesuccessiveapproximationofpracticalproducts(whatwecallinterventions)isgoinghandinhandwiththesuccessiveapproximationoftheory(whichhealsocallsdesignprinciples).

Figure 3: Generic Design Research Model (Wademan, 2005)

Itisimportanttonotethatdesignresearchfollowsaholisticapproach,anddoesnotemphasizeisolatedvariables.VandenAkkeretal.(2006:5)pointtoitthatyetdesignresearchersdofocusonspecificobjectsandprocesses(interventions)inspecificcontexts,buttheytrytostudythoseasintegralandmeaningfulphenomena.Thiscontextboundnatureofmuchdesignresearchalsoexplainswhyitusuallydoesnotstrivetowardscontext-freegeneralizations.Ifanefforttogeneralizingismade,thenitisananalyticalgeneralization(incontrasttostatisticalgeneralizationwheretheresearchermaygeneralizefromsampletopopulation).(Seealsothesectionoutputsofdesignresearch)

Practitioner and User Participation

Researchers

Other Sources

Collaboratives

Practitioners

ConsultExperts &

Practitioners

FocusedLiteratureReview

AnalyzePromisingExamples

AnalyzePracticalContext

Tentative Product

Approaches

Tentative Design

Principles

Reflection

FormativeEvaluation

PracticalProduct/Results

Contributionto

Theory

Refinement of Problem, Solution and Method

Refinement of Design Theory

Preliminary investigation of Problem, Context,

& Approaches

Problem in Context

Phases

Prototyping & Assessment of Preliminary Products & Theories

Identification of Tentative Products & Design Principes

Problem Identification Tentative Products& Theories

Problem Resolution & Advancing Theory

Successive Approximation of Theory

Successive Approximation of Product

Redesign & Refinementof Products & Theories


A closer look at design research

Asstatedkeycharacteristicsofdesignresearcharethatitisresearchfocusedondesigninginterventionsintherealcontextofeducationortraining(interventionistcharacteristic)combinedwitheffortstounderstandandimproveinterventions(process orientation),utilizingstateofthearttheorieswhilstthefieldtestingandtheevaluationoftheconsecutiveprototypesshouldcontributetotheorybuilding(theory orientation).Inthissectionwewillhavealookatwhatitmeansthatresearchsupportseducationaldesignprocesses,andreverselythateducationaldesignprocessessupportresearch.Thisisfollowedbyabriefdiscussionofthetypeofresearchquestionindesignresearch.Possibleoutputsofdesignresearchwillbediscussedinthenextsection.

Aswealreadystated,oneoftheaimsofdesignresearchisdesigninganddevelopinganinterventionasan(innovative)solutiontoacomplexproblem,andthereforethestartingpointfordesignresearchareeducationalproblemsforwhichnooronlyafewvalidatedprinciples(howtodoguidelines)areavailabletostructureandsupportthedesignanddevelopmentactivities.Ontheotherhand,designresearchisresearchandthereforetheappropriateyieldfordesignresearch(apartfromausableandeffectiveintervention)isempiricallyfoundedtheory,i.e.thechallengefordesignresearchistocaptureandmakeexplicittheimplicitdecisionsassociatedwithadesignprocess,andtotransformthemintoguidelinesforaddressingeducationalproblems(seeEdelson,2006;101;alsoBarab&Squire(2003),andmanyotherauthors).Thisaspectreferstothetheory orientation,mentionedaboveasoneofthecharacteristicsofdesignresearch.VandenAkker(1999,2006,alsochapter2),Reeves(2006;seefigure1)andWademan(2005;seefigure3)usetheconceptofdesignprincipleswhentheyrefertothetheoreticalyieldsofdesignresearch,whereothersspeakofnewtheories(e.g.Barab&Squire,2003;Edelson,2006).

However,itisnotself-evidenthowthedesignofinterventionsmaycontributetotheorybuilding.WithreferencetothegenericmodelofWademen(Figure3)andtheexemplaryschemesofReeves(2006)inFigure1andMcKenney(2001)inFigure2,onemaystatethattheresearcher(orbetter:thecollectiveofresearchersandpractitioners)-basedonanalysisoftheproblemincontext,andutilizingrelevant,state-of-the-arttheoriesdesignsanddevelops(inaniterativeway)theinterventionwiththeaimthatafteranumberofcyclestheintendedoutcomesarerealized,i.e.asatisfyingsolutiontotheproblemidentified.Eachiterationorcycleisamicro-cycleofresearch,i.e.astepintheprocessofdoingresearchandwillincludesystematic reflectiononthetheoreticalaspectsordesignprinciplesinrelationshiptothestatusoftheintervention,resultingintheendindesignprinciplesortheoreticalstatements.


Inotherwords,intheendtheresearcher(orresearchgroup)willconcludeabouthisintervention:

Given my context, if I do then I expect .

Thiscanbedisplayedschematicallyas:

Twopointsareimportantinthisscheme: theoutcomesoftheinterventionareindicatedasY1,Y2,,Yn,becauseoftenan

interventionisdesignedtorealizemultipleoutcomes(e.g.betterachievement,improvedstudentattitude,increasedteachersatisfaction,etc).

theinterventionispresentedasinputprocess,becausedesigningaprocess(e.g.learningenvironment)hastotakeintoaccountalsotheinputsnecessarytomaketheprocessfunction(e.g.certaininstructionallearningmaterials,teacherdevelopment).

Sointheend,theresearchgrouphasnotonlyatitsdisposaltheinterventionresultinginthedesiredoutcomes,butalsobasedonasystematicreflectionandanalysisofthedatacollectedduringthiscyclicalprocessanunderstandingofthehowandwhyofthefunctioningoftheinterventionintheparticularcontextwithinitwasdeveloped.ThedesignresearcherwillsummarizethisunderstandingofthehowandwhyoftheinterventioninoneormoredesignprinciplesifwewouldusetheterminologyofVandenAkker(1999,2006)andReeves(2000,2006).Asotherauthors,e.g.Barak&Squire(2004)andEdelson(2006),useoftheoryastheyieldofdesignresearch,onemayalsospeakofinterventiontheoryordesigntheory(Wademan,2005;Figure3)asasecondgenerictermtorefertotheknowledgegeneratedfromthisresearchendeavour(seebelowforspecificexamples).

Indesignresearch,interventionsaredevelopedinacyclical processofsuccessiveprototypes:

Intervention XInput Process

OutcomesY1, Y2, ...., Yn

Intervention XInput Process

OutcomesY1, Y2, ...., Yn

design principles or intervention theory

Akeyideaisthatwheninacertaincycletheprototypeoftheinterventiondoesnotresultinthedesiredoutcomes,onemayconcludethatthedesignprinciples(orinterventiontheory)appliedarenot(yet)effective(or,inotherwords,thattheinterventiontheoryfails).Thishastoresultinare-designorrefinementoftheintervention,whichgoeshand-in-handwiththerefinementoftheinterventiontheoryordesigntheory.Whenafteranumberofiterationstheresearcher(orresearchgroup)concludesthatbasedontheanalysisoftheevaluationdatatherealizedoutcomesarecloseenoughtotheintendedoutcomesthenhecanbesatisfied:thedesignprinciplesappeartobeeffective.Or,inotherwords,theresearcher(orresearchgroup)hasdevelopedalocal(intervention)theory(i.e.forthecontextinwhichhe/sheworks):in context Z the intervention X (with certain characteristics) leads to outcomes Y1, Y2, , Yn.

Twoexamplesaregiventoillustratethisratherabstractphrasingoftheyieldofdesignresearch.TheDesign-BasedLearningResearchCollective(2003:5)statethatthedesignofinnovationsenablesustocreateleaningconditionsthatlearningtheorysuggestsareproductive,butthatarenotcommonlypracticedorarenotwellunderstoodinotherwordsincludedintheinnovationsisknowledgeabouthowtocreateconditionsforlearning.Thesecondexampleistakenfromscienceeducation.Lijnse(1995:192)arguesthatdesignresearch(hecallsitdevelopmentalresearch)isacyclicprocessoftheoreticalreflections,conceptualanalysis,small-scalecurriculumdevelopment,andclassroomresearchoftheinteractionofteaching-learningprocesses.Thefinal,empiricallybaseddescriptionandjustificationoftheseinterrelatedprocessesandactivitiesconstituteswhatwecallapossibledidacticalstructureforthetopicunderconsideration.Inotherwords,thelocaltheoryconsistsofadidacticalstructureforteaching-learningprocessesforacertaintopic.

Theresearchquestionindesign/developmentresearchBynowitisclearthatdesigninganddevelopinganinterventionisinitselfnotyetdesignresearch.Butonemayconductadesign/developmentprojectasaresearchprojectbyemployingrigorouslysocialscienceresearchmethodology.Astheresearcherisstrivingtofinddesignprinciples(oranintervention theory)thatarevalidinacertaincontext,theresearchquestioncanbephrasedas:

what are the characteristics of an for the purpose/outcome Y (Y1, Y2, , Yn) in context Z

Design/develop Implement/try-0ut

Evaluate (formative)



Examplesofresearchquestionsare:(i) whatarethecharacteristicsofaneffectivein-serviceprogrammeformathematics

teachersthroughwhichtheydeveloptheabilitytoapplystudent-centredpedagogicalmethods,and

(ii)whatarethecharacteristicsofanin-servicearrangementthatfacilitatestheimplementationofMBL5-supportedlessonactivitiesinphysicseducation(Tecle,2003)?

Obviously,notallresearchersareusingthistypeofphrasing,butthewordingofthemainresearchquestionindesignresearchalwaysimpliesasearchforcharacteristics.Anexampleis:Whatisanadequatelearningandteachingstrategyforgeneticsinuppersecondarybiologyeducationinordertocopewiththemaindifficultiesinlearningandteachinggenetics,andtopromotetheacquisitionofameaningfulandcoherentunderstandingofhereditaryphenomena?(Knippels,2002)

The outputs of design research

Wealreadyconcludedthatdesignresearchresultsininterventions(programs,products,processes)andindesignprinciplesorinterventiontheory.Athirdoutputofdesignresearchisprofessionaldevelopmentoftheparticipantsinvolvedintheresearch.Eachoftheseoutputsisbrieflydiscussed.

OndesignprinciplesorinterventiontheoryDesignresearchaimsatproducingknowledgeaboutwhetherandwhyaninterventionworksinacertaincontext.Intheprevioussectionthistypeofoutputhasbeencalleddesignprinciplesorinterventiontheory.Otherauthorsusetermslikedomainspecifictheories(Gravemeijer&Cobb,2006),designtheory(Wademan,2005;Figure3),heuristicsorjustlessonslearned(seeVandenAkkeretal.2006).Wewillusethetermdesign principlesintheremainingofthispaper.

Designprinciplesareheuristic statementsforwhichVandenAkker(1999)developedthefollowingformat:

If you want to design intervention X for the purpose/function Y in context Z, then you are best advised to give that intervention the characteristics A, B, and C [substantive emphasis], and to do that via procedures K, L, and M [procedural

emphasis], because of arguments P, Q, and R. (Van den Akker, 1999)

5) MBL=MicrocomputerBasedLaboratory.

Theheuristicprinciplesaremeanttosupportdesignersintheirtasks,butcannotguaranteesuccess-theyareintendedtoassist(inotherprojects)inselectingandapplyingthemostappropriate(substantiveandprocedural)knowledgeforspecificdesignanddevelopmenttasks.Substantiveknowledgeisknowledgeaboutessentialcharacteristicsofaninterventionandcanbeextracted(partly)fromaresultinginterventionitself.Proceduralknowledgereferstothesetofdesignactivitiesthatareconsideredmostpromisingindevelopinganeffectiveandworkableintervention.Asknowledgeisincorporatedininterventions,itisprofitablefordesignresearchersintheearlystageoftheirresearchtosearchforalreadyavailableinterventionsthatcanbeconsideredusefulexamplesorsourcesofinspirationfortheproblematstake.Carefulanalysisofsuchexamplesincombinationwithreviewingrelevantliterature)willgenerateideasforthenewdesigntask.Thevalueofknowledgeresultingfromadesignresearchprojectwillstronglyincreasewhenitisjustifiedbytheoreticalarguments,well-articulatedinprovidingdirections,andconvincinglybacked-upwithempiricalevidenceabouttheimpactofthoseprinciples.Itisforthisreasonthatauthors(e.g.VandenAkker1999,2006;Reeves,2000,2006)statethatthefinalstageofeachdesignresearchprojectshouldconsistofsystematicreflectionanddocumentationtoproducedesignprinciples.

GeneralizabilityindesignresearchHeuristicdesignprincipleswillbeadditionallypowerfuliftheyhavebeenvalidatedinthesuccessfuldesignofmoresimilarinterventionsinvariouscontexts.Chancesforsuchknowledgegrowthwillincreasewhendesignresearchisconductedintheframeworkofresearchprograms,becausethenprojectscanbuildupononeanother.Herewetouchonthequestiontowhatextentdesignprinciplescanbegeneralizedfromonecontexttoothers.ItisinthiscontextthatEdelson(2006)statesthatdesignresearchshouldresultingeneralizabletheory.Indesignresearch,likeincasestudiesandexperimentalstudies,thefindingscannotbegeneralizedtoalargeruniversethereisnostatisticalgeneralizationfromsampletopopulation,likecanbethecaseinsurveyresearch.Yin(2003)pointstoitthatincasestudiesandexperimentalstudies,theinvestigatorisstrivingtogeneralizeaparticularsetofresultstoabroadertheory.Thisisalsothecaseindesignresearch,theresearchershouldstrivetogeneralizedesignprinciplestosomebroadertheory.Yin(2003:37)pointstoitthatgeneralizationisnotautomatic.Designprinciplesmustbetestedthroughreplicationsofthefindingsinasecond,thirdormorecasesinvariouscontextswiththepurposethatthesameresultsshouldoccur.Oncesuchreplicationshavebeenmade,theresultsmightbeacceptedforamuchlargernumberofsimilarcontexts,eventhoughfurtherreplicationshavenotbeenperformed.Thisreplication logicisthe



samethatunderliestheuseofexperimentsandallowsexperimentalscientiststogeneralizefromoneexperimenttoanother:Yin(2003)callsthisanalyticalgeneralizability.Butawarningshouldbephrasedhere.Wheredesignprinciplesmayhavebeensupportedbyanumberofreplications,andanewcontextmaybesimilartotheonesfromwhichdesignprincipleshaveemerged,yeteachcontexthasuniquecharacteristicsthatjustifiesthatthedesignprinciplesshouldbeusedasheuristicstatements:theyprovideguidanceanddirection,butdonotgivecertainties.ItisinthiscontextthatReeves(2006)citesLeeCronbachoneofthemostinfluentialresearchersofthe20thcentury:When we give proper weight to local conditions, any generalization is a working hypothesis, not a conclusion.(Cronbach,1975:125)

OninterventionsDesignresearchbyitscharacteraimstobepracticallyrelevant.Itisinitiatedtodesignanddevelopinnovativeinterventionstomeetaneedfeltinacomplex,practicalsituationforwhichnoready-madesolutionsorguidelinesareavailable.Thereforedesignresearchersaimatdevelopinginterventions(suchasprograms,teaching-learningstrategiesandmaterials,productsandsystems)thatcanbeusedinpracticeandareempiricallyunderpinnedsolutionstotheproblemsidentified.

OnprofessionaldevelopmentOneofthefeaturesofdesignresearchisthecollaborationofresearchersandpractitioners.Thiscollaborationincreasesthechancethattheinterventionwillindeedbecomepracticalandrelevantfortheeducationalcontextwhichincreasestheprobabilityforasuccessfulimplementation.Buttheparticipationofpractitionersshouldalsobeseenasanimportantformofprofessionaldevelopment.Anextraspin-offmaybethatpractitionerswilldevelopanawarenessofhowresearchmaycontributetoimprovingtheirprofessionalcontext.

Design research differentiation

Designresearchisconductedthroughanumberofcyclesofdesignanddevelopmentresultingintheinitialimplementationoftheinterventioninalimitednumberofcontexts.Asstatedabove,designresearchhasusuallyanumberofstagesorphases(seealsoFigures1,2and3): needsandcontentanalysis prototypingphase(iterativecyclesofdesignandformativeevaluation) assessmentphase(semi-summativeevaluation)

Nieveenetal.(2006)suggestthatdesignresearchthathasresultedinavalidatedandeffectiveintervention(asasolutionfortheproblemunderstudy),andindesignprinciplescanbefollowedbyeffect studies(notnecessarilypartofthesameresearchproject)withan


emphasisonupscalingtheinterventiontoawidercontext,andindoingsoaimingatdesignprinciplestestedinawiderdomain.Effectstudiesmayrangefromsmall-scalelearningexperimentstolarge-scalecomparativetestingofimpact(e.g.viarandomizedcontrolledtrials).

Afurtherdifferentiationindesignstudiesispossibleonthebasisofvariationsingoalsofdesignresearchvizvalidationstudiesversusdevelopmentstudies(seeVandenAkker,Gravemeijeretal.,2006;chapters5and10).

Validation studieshaveafocusondesigninglearningenvironmentsortrajectorieswiththepurposetodevelopandvalidatetheoriesabouttheprocessoflearningandhowlearningenvironmentscanbedesigned.Validationstudiesaimatadvancinglearningandinstructiontheories,suchas(Gravemeijer&Cobb,2006): micro-theories:atthelevelofinstructionalactivities localinstructiontheories:atthelevelofinstructionalsequence; domain-specificinstructiontheories:atthelevelofpedagogicalcontentknowledge.Invalidationstudies,researchersdonotworkincontrolled(laboratoryorsimulated)settings,buttheychoosethenaturalsettingofclassroomastestbeds(althoughtheytendtoworkwithabove-averagenumberofteachingstaff).Usually,thestagesinvalidationstudiesare(Gravemeijer&Cobb,2006): environment preparation:elaboratingapreliminaryinstructionaldesignbasedonan

interpretativeframework; classroom experiment:testingandimprovingtheinstructionaldesignorlocal

instructionaltheoryanddevelopinganunderstandingofhowitworks; retrospective analysis:studyingtheentiredatasettocontributetothedevelopmentofa

localinstructionaltheoryand(improvementof)theinterpretativeframework.DiSessaandCobb(2004:83)warnthatdesignresearchwillnotbeparticularlyprogressiveinthelongrunifthemotivationforconductingexperimentsisrestrictedtothatofproducingdomainspecificinstructionaltheories.Butthepracticalcontributionliesindevelopingandimplementingspecificlearningtrajectoriesthatwereimplementedtotestthetheoreticalbasisofthedesign.(Nieveenetal,2006:153)

Development studiesaimtowardsdesignprinciplesfordevelopinginnovativeinterventionsthatarerelevantforeducationalpractice.Developmentstudiesintegratestate-of-the-artknowledgefrompriorresearchinthedesignprocessandfine-tuneeducationalinnovationsbasedonpilotinginthefield.Byunpackingthedesignprocess,designprinciplesthatcaninformfuturedevelopmentandimplementationdecisionsarederived.(Nieveenetal.,2006:153).Twomaintypesofdesignprinciplescanbedistinguished(VandenAkker,1999):1. proceduraldesignprinciples:characteristicsofthedesignapproach;2. substantivedesignprinciples:characteristicsofthedesign(=intervention)itself.


Figurethreesummarizesthecharacteristicsofaresearchcycleconsistingofdesignstudiesandeffectstudies(asdevelopedbyNieveenetal.;2006:155):

Design researchEffectiveness research

Validation studies Development studiesDesign aim To elaborate and

validate theoriesTo solve educational problems

-

Quality focus of design

Theoretical quality of design

Practicality of intervention

Effectiveness of intervention

Knowledge claim/ scientific output

Domain-specific instruction theories

Broadly applicable design principles

Evidence of impact of intervention

Methodological emphasis

Iterative design with small scale testing in research setting

Iterative development with formative evaluation in various user settings

Large scale, comparative field experiments

Practical contribution Specific learning trajectories for a specific classroom

Implemented interventions in several contexts/classrooms

Evidence-basedChange at large scale

Figure 4: Educational engineering research cycle (from Nieveen et al., 2006)

Itisimportanttonotethatthisdistinctionbetweenvalidationanddevelopmentstudiesisconceptuallyimportant,butthatinpracticemanyresearchprojecthaveaimsthatareacombinationofsolvingproblemsineducationalpracticeandelaboratingandvalidatingtheories(designprinciples).

Afurtherdifferentiationofdesignresearchisconceivable.Forexample,onecanimaginethatthedisseminationandimplementationofaparticularprogramissupportedbydesignresearchtheresultinginterventionisthesuccessfullydisseminatedandimplementedprogram,whilstthesystematicreflectionanddocumentationoftheprocessleadstoasetofproceduresandconditionsforsuccessfuldisseminationandimplementation(thedesignprinciples).

Asafinalnote,thedifferentiationbetweentypesofdesignresearch,suchasvalidationstudiesversusdevelopmentstudies,servesmainlyconceptualpurposes.Inpractice,designresearchersmaycombinethetwoorientationsintheirresearch.Forexample,startingfromacomplexandpersistentproblemine.g.scienceeducation,theresearchgroupmaydecidetoapplythedesignprinciples(localtheories)resultingfromotherstudiesintheirresearch.Indoingsotheyarenotonlydevelopinganintervention,butatthesametimeexploringthevalidityofdesignprinciples(theory)developedinanothercontextfortheirownproblemcontext.


Howisdesignresearchconducted?Designresearchisconductediterativelyasacollaborationofresearchersandpractitionersinareal-worldsetting.Onlythenthetwoprincipaloutputs(designprinciplesandempiricallyunderpinnedinnovativeinterventions)canberealized.Doingresearchinsuchasettingischallenginganddemandsacarefulresearchdesign.Itisthereforeimportanttoreflectnotonlyonthecyclical,iterativecharacterofthesystematicdesignoftheintervention,butalsobecauseitisresearch-tomakeexplicitthetenetsthatformthefoundationofthistypeofresearch(McKenneyetal.,2006)

McKenneyetal.(2006:77)definethreetenetstoshapedesignresearchforthecurriculumdomain(butthetenetsalsoapplytootherdomains): Rigorfordesignresearchtobeabletoresultinvalidandreliabledesignprinciples,the

researchhastomeetrigorousstandardsandapplytheguidingprinciplesforscientificresearchasmentionedbyShavelson&Towne(2002;mentionedabove).Muchliteratureisavailabletoguideresearchinnaturalsettingsthatofferssupporttoissueslikeinternalandexternalvalidity,reliabilityandutililizationoftheresearch.

Relevance:Designresearchaimstoberelevantforeducationalpractice(andpolicy).Anecessaryconditionforthisisthattheresearchgroupmusthaveagoodworkingknowledgeofthetargetsettingandbeinformedbyresearchanddevelopmentsactivitiestakingplaceinnaturalsettings(ortestbeds).

Collaboration:fordesignresearchtoberelevantforeducationalpractice,thedesignanddevelopmentactivitiesmustbeconductedincollaborationwithandnotjustforprofessionalsfromeducationalpractice.

Asexplainedinthebeginningofthischapter,designresearchiscyclicalandeachiterationorcyclecontributestosharpeningtheaimsandtobringingtheinterventionsclosertothedesireddesignoutcomesandresearchoutputs.AsisillustratedinFigures1-3,designresearchusuallygoesthroughseveralstageswhichNieveenetal.(2006:154)phraseasfollows(seealsop.15): preliminary research:thoroughcontextandproblemanalysisalongwiththe

developmentofaconceptualframeworkbasedonliteraturereview; prototyping stage:settingoutdesignguidelines,optimizingprototypesofthe

interventionthroughcyclesofdesign,formativeevaluation,andrevisionitisimportanttonotethateachcycleinthestudyisapieceofresearchinitself(i.e.havingitsresearchorevaluationquestiontobeaddressedwithaproperresearchdesign);

assessment stage (summative evaluation):oftenexplorestransferabilityandscaling,alongwith(usuallysmall-scaleevaluationof)effectiveness;and

systematic reflection and documentation:thisarecontinuousactivities(asillustratedinFigure3)thattakesplaceduringallcyclesintheresearchhowever,attheendthe


researcherportraystheentirestudytosupportretrospectiveanalysis,followedbyspecificationofdesignprinciplesandarticulationoftheirlinkstotheconceptualframework.

Itisbeyondthescopeofthischaptertodiscussindetailhowtoperformthesestages.Butanexceptionismadeforformativeevaluation,becausethisisthekeyresearchactivityindesignresearchaimedatimprovingthequalityoftheconsecutiveprototypesoftheintervention.

Formativeevaluationindevelopmentresearch6BasedonpriorworkNieveen(1999;seealsoChapter5)proposesfourgenericcriteriaforhighqualityinterventions(seeTable1).Sheexplainsthesecriteriaasfollows:Thecomponentsoftheinterventionshouldbebasedonstate-of-the-artknowledge(content validity)andallcomponentsshouldbeconsistentlylinkedtoeachother(construct validity).Iftheinterventionmeetstheserequirementsitisconsideredtobevalid.Anothercharacteristicofhigh-qualityinterventionsisthatend-users(forinstancetheteachersandlearners)considertheinterventiontobeusableandthatitiseasyforthemtousethematerialsinawaythatislargelycompatiblewiththedevelopersintentions.Iftheseconditionsaremet,wecalltheseinterventionspractical.Athirdcharacteristicofhighqualityinterventionsisthattheyresultinthedesiredoutcomes,i.e.thattheinterventioniseffective.

CriterionRelevance (also referred to as content validity)

There is a need for the intervention and its design is based on state-of-the-art (scientific) knowledge.

Consistency (also referred to as construct validity)

The intervention is logically designed.

Practicality The intervention is realistically usable in the settings for which it has been designed and developed.

Effectiveness Using the intervention results in desired outcomes.

Table 1: Criteria for high quality interventions (from Nieveen, 1999; Chapter 5)

Giventhecharacterofdesignresearch,thesefourcriteriamaygetdifferentemphasisindifferentstagesoftheresearchasisillustratedbyFigure5.Forexample,duringthepreliminaryresearchwheretheemphasisisonanalyzingtheproblemandreviewingtheliterature,thecriterionofrelevance(contentvalidity)isthemostdominant,withsomeattentionforconsistency(constructvalidity)andpracticality,whilstinthatstatenoattentionisyetgiventoeffectiveness.Ontheotherhand,intheprototypingstagemuch

6) SeealsoNieveenschapter5inthisbookinwhichshediscusseshowtodotheformativeevaluationindesignresearch


attentionhastobepaidintheformativeevaluationtothecriterionofpracticality,whilsteffectivenesswillbecomeincreasinglyimportantinlateriterations.Finally,inassessmentstageofsummativeevaluation,thefocuswillbeonpracticalityandeffectiveness(seeFigure5,andFigure2forthestages).

Stage Criteria Short description of activities1 Preliminary research Emphasis mainly on

content validity, not much on consistency and practicality

Review of the literature and of (passed and/or present) projects addressing questions similar to the ones in this study. This results in (guidelines for) a framework and first blueprint for the intervention.

2 Prototyping stage Initially: consistency (construct validity) and practicality. Later on mainly practicality and gradually attention for efficiency.

Development of a sequence of prototypes that will be tried out and revised on the basis of formative evaluations. Early prototypes can be just paper-based for which the formative evaluation takes place via expert judgments.

3 Assessment phase practicality and efficiency

Evaluate whether target users can work with intervention (practicality) and are willing to apply it in their teaching (relevance & sustainability). Also whether the intervention is effective.

Figure 5: Evaluation criteria related to stages in design research

Formativeevaluationtakesplaceinallphasesanditerativecyclesofdesignresearch.AsillustratedbyFigure5,formativeevaluationservesdifferentfunctions,or-inotherwords-isaimedatdifferentcriteria(orcombinationsofthese)inthevariousdevelopmentcycles,eachbeingamicro-cycleofresearchwithitsspecificresearch/evaluationquestionandrelatedresearch/evaluationdesign.OnemaysaythatformativeevaluationhasvariouslayersinadesignresearchprojectasisillustratedinFigure6,takenfromTessmer(1993):frommoreinformalintheearlystagesofaproject(self-evaluation,one-to-oneevaluation,expertreview)tosmallgroupevaluationaimedattestingthepracticalityandeffectiveness,toafullfieldtest(ifapplicable).Theresearch/evaluationdesignforeachcycleshouldreflectthespecificfocusandcharacterofthecycleseeChapter5byNieveenformoredetails.


Figure 6: Layers of formative evaluation (taken from Tessmer, 1993)

Figure6alsoillustratesthatmanypossiblemethodsofformativeevaluationcanbechosen,suchas7 expertreviewand/orfocusgroups(importanttoconsiderexpertsinwhat) self-evaluationorscreening(usingchecklistofimportantcharacteristicsordesign

specifications) one-to-oneevaluationorwalkthrough(withrepresentativeoftargetaudience) smallgroupormicro-evaluation fieldtestortry-outDesignresearchersshouldchooseforeachphaseandforeachprototypeformativeevaluationapproachesthataresuitableforthepurposeofthatparticularstageoftheresearch.Designresearchhastomeetcriteriaforgoodresearch.Itisthereforeimportantthatforeachdevelopmentcycletheresearcher(orresearchgroup)appliesthemethodologicalrulesfordoingresearch,i.e.foridentifyingthetargetaudienceandsampling,forinstrumentdevelopmentandapplytriangulationtoobtaingoodqualityinformation.But

7) seealsoChapter5byNieveen

HighResistanceto Revision

LowResistanceto Revision

Revise

Revise

Revise

Field TestUser Acceptance, Implementability

Organizational Acceptance

Small GroupEffectiveness, AppealImplementability

Expert ReviewContent, Design,Techical Quality

One-to-OneCiarity, AppealObvious Errors

Self-Evaluationobvious errors


giventhelayersofformativeevaluationindesignresearch,intheearlycyclesofdevelopmenttheevaluationdesigncanbelessrigorousthaninlaterphases.Figure7adaptedfromNieveen(1999)presentsanexamplethatillustrateshowvariousformativeevaluationmethodsareusedfortherespectiveprototypesinaprojectaimedatdevelopingacomputerassistedsupportsystemforcurriculumdevelopers.

prelimcomp.based

paper-based computer-based versions

final version

Users (n=5)

experts (n=3)

users (n=5)

experts (n=6)

users (n=4)

users (n=4)

users (n=17)

Validity content *) ea eainterface ea

Practicality content wt wt ea me to ftinterface wt wt ea me to ft

Effectiveness entire system to ft

*): Content refers to the content of the support system = primary attention of prototype and of formative evaluationMethods of formative evaluation: me = micro evaluation; wt = walk through; ea = expert appraisal; ft = field trial; to = try-out

Figure 7: Focus of design and formative evaluation of the prototypes for computer assisted support system for curriculum development (adapted from Nieveen, 1999)

Afinalnoteonthecriteriaofpracticalityandeffectiveness.Itmayoccurincertainstudiesthattheresearcher(orresearchcollaborative)cannotdoafinalfieldtrialoftheinterventionwiththefull(orasampleofthe)targetgroup,buthastorestricthimselftoexpertappraisaland/ormicro-evaluationofthefinalprototypeoftheintervention.Itisobviousthatinsuchasituationtheactual practicalityandtheactual effectivenessoftheinterventioncannotbedemonstrated,butonlyconclusionsabouttheexpected practicalityandtheexpected effectivenesscanbedrawn.Moreevaluationwillthenbeneededtodemonstratetheactual practicalityandtheactual effectiveness.ThiscanbeillustratedwithanexampleadaptedfromMafumiko(2006)whoconducteddesignresearchtoinvestigatewhethermicroscaleexperimentationcancontributetoimprovingthechemistrycurriculuminTanzania.HisresearchdesignhasbeensummarizedinFigure8.


Figure 8: Example of research research design (adapted from Mafumiko, 2006)

SupposearesearcherwouldrestricthimselftothedevelopmentofprototypesoftheinterventionasillustratedinFigure8,anddoesnotplantoinvestigatewhetherVersionIVworksinthetargetcontext.Insuchasituationthemosthecanconcludeiswhetherhisinterventionisexpectedtobepracticalandeffectiveforthetargetcontext.Onlywhenhewouldconductafieldtest,hewillbeinthepositiontodecideuponactualpracticalityandactualeffectiveness(whichiswhatMafumikodid).

Design research dilemmas

Designresearchisconductedinclosecollaborationwitheducationalpractice.Notonlytheproblemaddressedissituatedineducationalpractice,butakeyfeatureofthisresearchisthateducationalpractitionersareactivelyinvolved,oftenasmembersoftheresearchteam.Thisleadstoanumberofchallengesthataretypicalforthistypeofresearch.McKenneyetal.(2006:83,84)havediscussedsomeoftheseandprovidesuggestionsforhowtoaddressthem.Theirpointsarebrieflysummarizedhere.

1. theresearcherisdesignerandoftenalsoevaluatorandimplementer.Severalmeasurescanbetakentocompensateforthispotentialconflictofinterest: makeresearchopentoprofessionalscrutinyandcritiquebypeopleoutsidetheproject theresearcherappliesthefollowingruleofthumb:shiftfromadominanceofcreative

designerperspectiveintheearlystage,towardsthecriticalresearcherperspectiveinlaterstages(thisisreflectedinTessmerslayersofformativeevaluation,Figure6)

Summativeevaluation

Appraisel by3 experts

Tryout in3 classrooms

Panel session with experts

Field test infour schools

Tryout with teacher educ students

Version I Version II Version III Version IV

Development of prototypes

Design guidelines &specifications


haveagoodqualityofresearchdesign,e.g. - strong chain of reasoning(Krathwohl,1998)-themetaphorexpressestheideathat

eachpartoftheresearchdesignisequallyimportant - triangulationtoincreasethequalityofdataandofanalysistriangulationofdata

sourcesanddatacollectionmethodsshouldbeapplied,aswellasinvestigatortriangulationtoavoidtheinfluenceofanyspecificresearcher(seee.g.Denscombe,2007;136)

- empirical testingofboththeusabilityandtheeffectivenessoftheintervention - systematic documentation, analysisandreflectionofthedesign,development,

evaluationandimplementationprocessandtheirresults - haveattentionforvalidityandreliabilityofdataandinstruments - applyavarietyofmethodsandtactics:e.g.usepractitionersandotherresearchersas

criticalfriends;usemultipleobservers/ratersandcalculateinter-observer/raterreliability,etc.

2. real-worldsettingsbringreal-worldcomplicationsDesignresearchisconductedinreal-worldsettingsbecauseitaddressescomplexproblemsineducationalpractice.Oneoftheproblemsisthattheresearchercanbeaculturalstranger(Thijs,1999)inthesettingoftheresearchandthatparticipants(e.g.principals,teachersnotinvolvedintheresearch,etc)arehesitanttobecompletelyopentoaresearchercomingfromtheoutside.McKenneyetal.(2006:84)pointstotheimportanceofcollaborationandmutualbeneficialactivitiestogainparticipantstrustandthoroughunderstandingofthecontext(i.e.insiderperspective).Ontheotherhand,theyalsopointtotheadvantagestobeanoutsiderasthismayallowtheresearchertodevelopadegreeofobjectivityandfreedom(orforgiveness)forhonestythatisnotpermittedtothosewithinaparticulargroup(o.c.85)

3. adaptabilityDesignresearchiscyclicalandtakesplaceinreal-worldsettings.Eachcyclehastotakethefindingsofthepreviousonesintoaccount.Soontheonehandtheresearchdesignhastochange(ordevelop)fromonecycletotheother,whilstontheotherhandanever-changingresearchdesigncanbeweak.Inthiscontext,McKenneyetal.(2006:84)refertothenotionofevolutionaryplanning,i.e.aplanningframeworkthatisresponsivetofielddataandexperiencesasacceptablemomentsduringthecourseofthestudy.Thisisalreadyalludedtointhediscussionofformativeevaluation(seeFigure6fromTessmerandtheexampletakenfromNieveen,1999).Theneedforadaptabilitypertainsalsototheroleoftheresearcher.AccordingtoVandenAkker(2005,inMcKenneyetal.,2006),thesynergybetweenresearchandpracticecanbemaximizedwhenresearchersdemonstrateadaptabilityby:


(i) beingprepared,wheredesirable,totakeontheadditionalroleofdesigner,advisor,andfacilitator,withoutlosingsightoftheirprimaryroleasresearcher,

(ii) beingtolerantwithregardtotheoftenunavoidablyblurredroledistinctionsandremainingopentoadjustmentsintheresearchdesignifprojectprocesssodictates,

(iii)allowingthestudytobeinfluenced,inpart,bytheneedsandwishesofthepartners,duringwhatisusuallyalong-termcollaborativerelationship.

Suchadaptabilityrequiresstrongorganizationalandcommunicativecapabilitiesonbehalfoftheresearcher,aswellassoundunderstandingtheresearchprocesssothatcarefulchangesandchoicesthatmaximizevalueandminimizethreatstoqualityaremade.(McKenneyetal.,2006:84).

Toaddressthechallengesmentioned,McKenneyetal.(2006:85,86)presentafewguidelines for conducting design researchthatmayhelpresearchersmonitoringthescientificcharacterofhis/herresearch:- haveanexplicitconceptualframework(basedonreviewofliterature,interviewsof

experts,studyingotherinterventions)- developcongruentstudydesign,i.e.applyastrongchainofreasoningwitheachcycle

havingitsresearchdesign- usetriangulation(ofdatasource,datatype,method,evaluatorandtheory)toenhance

thereliabilityandinternalvalidityofthefindings- applybothinductiveanddeductivedataanalysis- usefull,context-richdescriptionsofthecontext,designdecisionsandresearchresults- membercheck,i.e.takedataandinterpretationsbacktothesourcetoincreasethe

internalvalidityoffindings.ItisbeyondthescopeofthispapertoelaborateontheseguidelinesfurtherseeMcKenneyetal.(2006;85,86)andresearchmethodologybooks.

Concluding remarks

Inthefieldofeducationthereismuchneedforresearchrelevantforeducationalpractice.Wehavearguedthatforcomplexpracticalproblemsandforresearchquestion(s)callingforthedesignanddevelopmentofaninterventiondesignresearchistheappropriateresearchapproach.

Givenitsfocusonpracticalproblemsanditsnatureofconductingtheresearchinareal-worldsettingwithactiveinvolvementofpractitioners,designresearchmaylooklikeactionresearch.Soonemaywonderhowdesignresearchisrelatedtoactionresearch.Indeed,actionresearchisalsodealingwithreal-worldproblems,aimingatimprovingpractice,cyclicalinnatureandparticipative(Denscombe,2007),buttheessentialdifferenceisthat


actionresearchisnotaimedatgeneratingdesignprinciplesithasaparticularnicheamongprofessionalswhowanttouseresearchtoimprovetheirpractices(o.c.:122).

Wediscussedhowdesignresearchersshouldstriveforgeneralizabledesignprinciplesinthemeaningofgeneralizingtoabroadertheory.Whendesignresearchisconductedwithintheframeworkofaprogramofresearchaddressingfundamentalproblemsineducationalpractice,itwillresultinaspecificbodyofknowledge,vizsubstantiveandproceduraldesignprinciplesthatmaycontributetoimproveeducation.Ontheotherhandmanyquestionsarestilltobeaddressedastherearemanytypesofpracticalproblemsandthereforemanytypesofresearchgoalsforwhichdesignresearchmaybethebestapproach(e.g.,Reeves(2000)mentionssixdifferenttypesofgoals).

VandenAkker,Gravemeijer,McKenneyandNieveen(2006)reportthepresentationsanddiscussionsataseminardedicatedtoeducationaldesignresearch.Theirbookpoints-nexttodiscussinganumberofapproachestodesignresearchbyGravemeijerandCobb(2006),Reeves(2006)andMcKenneyetal.(2006)-toissueslikeassessingthequalityofdesignresearchproposals(chaptersbyPhillips,2006,andbyEdelson,2006)andthequalityofdesignresearch(chapterbyKelly,2006)whichneedfurtherreflectionandelaboration.

Finally,anumberofresearchreportsanddissertationshavebeenpublishedwhichareexemplaryforhowdesignresearchcanbeconducted(seechapter6forexamples).Butfordesignresearchtomaturefurthermoreresearchprojectsinavarietyofcontextsshouldnotonlybeconducted,butalsoreportedanddiscussedinresearchjournalsandatconferences.

OurhopeisthatthecommunityofeducationaltechnologistsinChinawillembarkonthisresearchendeavorandwillactivelycontributetothefurtherdevelopmentofeducationaldesignresearch.

Acknowledgement: in preparing this chapter, much use has been made of Van den Akker, Gravemeijer, McKenney and Nieveen (2006).


References

Bannan-Ritland,B.(2003).TheRoleofDesigninResearch:TheIntegrativeLearningDesignFrameworkEducational Researcher, 32(1),21-24.

Barab,S.&Squire,K.(2004).Design-basedresearch:puttingastakeintheground.Journal of the Learning Sciences, 13(1),1-14.

Brown,A.L.(1992).Designexperiments:Theoreticalandmethodologicalchallengesincreatingcomplexinterventionsinclassroomsettings.The Journal of The Learning Sciences, 2(2),141-178.

Cronbach,L.J.(1975).Beyondthetwodisciplinesofscientificpsychology.American Psychologist, 30(2),116-27.

Denscombe,M.(2007,3rded)).The good research guide for small-scale social research projects. Buckingham(UK):OpenUniversityPress.

Design-BasedResearchCollective(2003).Design-basedresearch:Anemergingparadigmforeducationalinquiry.Educational Researcher, 32(1),5-8.

diSessa,A.A.&Cobb,P.(2004).Ontologicalinnovationandtheroleoftheoryindesignexperiments.Educational Researcher, 32(1),77-103.

Edelson,D.C.(2006).Balancinginnovationandrisk:assessingdesignresearchproposals.In:VandenAkker,J.,Gravemeijer,K,McKenney,S.&Nieveen,N.(Eds).(2006).Educational design research.London:Routledge,100-106.

Gravemeijer,K.&Cobb,P.(2006).Designresearchfromalearningdesignperspective.In:VandenAkker,J.,Gravemeijer,K,McKenney,S.&Nieveen,N.(Eds).(2006).Educational design research. London:Routledge,17-51.

Gustafson,K.L.&Branch,R.M.(2002).Survey of instructional development models(4thed).Syracuse(NY,USA):ERICClearinghouseonInformation&Technology.

Kelly,A.E.(2006).Qualitycriteriafordesignresearch:evidenceandcommitments.In:VandenAkker,J.,Gravemeijer,K,McKenney,S.&Nieveen,N.(Eds).(2006).Educational design research.London:Routledge,107-118.

Knippels,M.C.P.J.(2002).Coping with the abstract and complex nature of genetics in biology education.Utrecht(TheNetherlands):UniversityofUtrechtCD-BtaPress.

Krathwohl,D.R.(1998,2nded).Methods of Educational and Social Science Research: An integrated approach.NewYork:Longman

Lijnse,P.L.(1995).DevelopmentalResearchasawaytoanempiricallybasedDidacticalStructureofScience.Science Education, 29(2),189-199.

Mafumiko,FideliceSimbagungileMbaruku(2006).Micro-scale experimentation as a catalyst for improving the chemistry curriculum in Tanzania.Doctoralthesis.Enschede(TheNetherlands)UniversityofTwente.Availablefrom:http://purl.org/utwente/55448

McKenney,S.(2001).Computer-based support for science education materials developers in Africa: exploring potentials.Doctoraldissertation.Enschede:UniversityofTwente.


McKenney,S.,Nieveen,N.&VandenAkker,J.(2006).Designresearchfromacurriculumperspective.In:VandenAkker,J.,Gravemeijer,K,McKenney,S.&Nieveen,N.(Eds).(2006).Educational design research. London:Routledge,62-90.

Nieveen,N.(1999).Prototypingtoreachproductquality.InJ.vandenAkker,R.M.Branch,K.Gustafson,N.Nieveen,&T.Plomp(Eds),Design approaches and tools in education and training.Boston:KluwerAcademic,125-136.

Nieveen,N.,McKenney,S.&VandenAkker,J.(2006).Educationaldesignresearch:thevalueofvariety.In:VandenAkker,J.,Gravemeijer,K,McKenney,S.&Nieveen,N.(Eds).(2006).Educational design research.London:Routledge,151-158.

Phillips,D.C.(2006).Assessingthequalityofdesignresearchproposals:somephilosophicalperspectives.In:VandenAkker,J.,Gravemeijer,K,McKenney,S.&Nieveen,N.(Eds).(2006).Educational design research.London:Routledge,93-99.

Reeves,T.(2000).Enhancingtheworthofinstructionaltechnologyresearchthroughdesignexperimentsandotherdevelopmentalstrategies.AERA.AERA.RetrievedOct.20,2006fromhttp://it.coe.uga.edu/~treeves/AERA2000Reeves.pdf

Reeves,T.C.(2006).Designresearchfromatechnologyperspective..In:VandenAkker,J.,Gravemeijer,K,McKenney,S.&Nieveen,N.(Eds).(2006).Educational design research.London:Routledge,52-66.

Shavelson,R.&Towne,L.(2002).Scientific research in education.WashingtonDC:NationalAcademicPress(website:http://books.nap.edu/books)

Tecle,A.T.(2006).The potential of a professional development scenario for supporting biology teachers in Eritrea.Doctoraldissertation.Enschede(TheNetherlands):UniversityofTwente.

Tessmer,M.(1993).Planning and conducting formative evaluations.London:KoganPage.Thijs,A.(1999).Supporting science curriculum reform in Botswana: The potential of peer

coaching.Doctoraldissertation.Enschede:UniversityofTwente.VandenAkker,J.(1999).PrinciplesandMethodsofDevelopmentResearch.InJ.vanden

Akker,R.M.Branch,K.Gustafson,N.Nieveen,&T.Plomp(Eds),Design approaches and tools in education and training.Boston:KluwerAcademic,1-14.

VandenAkker,J.(2005).Hoekansamenwerkingleidentotmeersuccesenwijsheidinonderwijsontwikkeling?[Howcancollaborationleadtomoresuccessandwisdomineducationdevelopment?].Pedagogische Studin, 82(4),343-7.

VandenAkker,J.,Gravemeijer,K,McKenney,S.&Nieveen,N.(Eds).(2006).Educational design research.London:Routledge.ISBN10:0-415-39635-2(pbk)

Wademan,M.(2005).Utilizing Development Research to Guide People Capability Maturity Model Adoption Considerations.Unpublisheddoctoraldissertation,SyracuseUniversity(NewYork,USA).

Yin,R.K.(2003,3eded).Case study research: design and methods.NewburyPark(CA,USA):SageAppliedSocialResearchMethodsSeries,volume5.


2. CurriculumDesignResearch Jan van den AkkerIntroduction

Thetitleofthischapter (Curriculum Design Research)intentionallycombinestwofields:curriculumdesignanddesignresearch.Itsymbolizestheaimofthistexttodiscussthefunctionandformsofdesignresearchfromacurricularperspective.Inparticular,itfocusesonhowdesignresearchcanincreasethequalityofcurriculumdesignanddevelopment.Also,itillustrateshowtherelevanceofeducationalresearch-awidelydebatedissue-canbenefitfromaconnectiontocurriculumpoliciesandpractices.Giventhisaimithelpstohaveanumberofbasicconceptsandanalyticalperspectivesavailablethatcanstructurecurriculardeliberationsandreducethecomplexityofcurriculumtasks.Thusmyinitialfocusinthischapter(buildingonvandenAkker,2003)isonsummarizingasetofconceptsandperspectivesthathelptoincreasethetransparencyandbalanceofcurriculumanalysis,developmentanddiscourse.Then,thefocuswillshifttowards(curriculum)designresearch(buildingonvandenAkker,1999,2006,andonvandenAkker,Gravemeijer,McKenneyandNieveen,2006).First,Iwillsketchthepotentialandcharacteristicsofdesignresearchinaddressingcomplexcurriculumchallenges.Second,Iwilladdressanumberofmethodologicalissues.Finally,Iwillpayattentiontoaclassicprobleminalleducationalresearch:generalizationoffindings.

Curriculum, whats in a name?

Whenthereisamyriadofdefinitionsofaconceptintheliterature(aswithcurriculum),itisoftendifficulttokeepaclearfocusonitsessence.Inthosecasesitoftenhelpstosearchfortheetymologicaloriginoftheconcept.TheLatinwordcurriculum(relatedtotheverbcurrerei.e.running)referstoacourseortracktobefollowed.Inthecontextofeducation,wherelearningisthecentralactivity,themostobviousinterpretationofthewordcurriculumisthentoviewitasacourse,trajectory,orplan for learning(cf.Taba,1962).Thisveryshortdefinition(reflectedinrelatedtermsinmanylanguages)limitsitselftothecoreofallotherdefinitions,permittingallsortsofelaborationsforspecificeducationallevels,contexts,andrepresentations.Obviously,contextualspecificationisalwaysneededincurriculumconversationstoclarifytheperspective.Giventhissimpledefinition,adifferentiationbetweenvariouslevelsofthecurriculumhasproventobeveryusefulwhentalkingaboutcurricularactivities(policy-making;designanddevelopment;evaluationandimplementation).Thenextdistinctionappearstobehelpful: International/comparative(orsupralevel) System/society/nation/state(ormacro)level(e.g.nationalsyllabiorcoreobjectives)


School/institution(ormeso)level(e.g.school-specificcurriculum) Classroom(ormicro)level(e.g.textbooks,instructionalmaterials) Individual/personal(ornano)level.

Thesupralevelusuallyreferstointernationaldebatesoragreementsonaimsandqualityofeducation,sometimesfuelledbyoutcomesofinternationallycomparativestudies(cf.PISAorTIMSS1).Curriculumdevelopmentatthesupralevelisusuallyofagenericnature,whilesite-specificapproachesaremoreapplicableforthelevelsclosertoschoolandclassroompractice.Moreover,theprocessofcurriculumdevelopmentcanbeseenasnarrow(developingaspecificcurricularproduct)orbroad(alongterm,ongoingprocessofcurriculumimprovement,oftenincludingmanyrelatedaspectsofeducationalchange,e.gteachereducation,schooldevelopment,testingandexaminations).Inordertounderstandproblemsofcurriculumdecision-makingandenactment,abroaderdescriptionofcurriculumdevelopmentisoftenmostappropriate:usuallyalongandcyclicprocesswithmanystakeholdersandparticipants;inwhichmotivesandneedsforchangingthecurriculumareformulated;ideasarespecifiedinprogramsandmaterials;andeffortsaremadetorealizetheintendedchangesinpractice.

Moreover,curriculacanberepresentedinvariousforms.Clarificationofthoseformsisespeciallyusefulwhentryingtounderstandtheproblematiceffortstochangethecurriculum.Acommonbroaddistinctionisbetweenthethreelevelsoftheintended,implemented,andattainedcurriculum.Amorerefinedtypology(vandenAkker,2003)isoutlinedinbox1.

INTENDED Ideal Vision (rationale or basic philosophy underlying a curriculum)Formal/Written Intentions as specified in curriculum documents and/or

materialsIMPLEMENTED Perceived Curriculum as interpreted by its users (especially teachers)

Operational Actual process of teaching and learning (also: curriculum-in-action)

ATTAINED Experiential Learning experiences as perceived by learnersLearned Resulting learning outcomes of learners

Box 1: Typology of curriculum representations

Traditionally,theintendeddomainreferspredominantlytotheinfluenceofcurriculumpolicymakersandcurriculumdevelopers(invariousroles),theimplementedcurriculum

1) PISAistheOECDProgrammeforInternationalStudentAssessment,asurveyeverythreeyearsofthe15-year-olds.TIMSSistheTrendsInMathematicsandSciencesStudy,conductedevery4yearsbytheInternationalAssociationfortheEvaluationofEducationalAchievement(IEA)inprimaryandsecondaryeducation.


relatesespeciallytotheworldofschoolsandteachers,andtheattainedcurriculumhastodowiththestudents.

Besidesthisdifferentiationinrepresentations,curriculumproblemscanbeapproachedfromvariousanalyticalangles.Forexample,Goodlad(1994)distinguishesthefollowingthreedifferentperspectives: substantive,focusingontheclassicalcurriculumquestionaboutwhatknowledgeisof

mostworthforinclusioninteachingandlearning; technical-professional,referringtohowtoaddresstasksofcurriculumdevelopment; socio-political,referringtocurriculumdecision-makingprocesses,wherevaluesand

interestsofdifferentindividualandagenciesareatstake.Somemightarguethatthislististoolimitedasitrefersespeciallytocurriculumissuesfortraditionalplanningforlearninginschools,anddoesnotincludethemorecriticalperspectivesthatareamplypresentincurriculumtheoryliterature(e.g.Pinar,Reynolds,Slattery&Taubman,1995).However,fromaprimaryinterestincurriculumimprovement,thethreeperspectivesseemusefulandappropriate.

The vulnerable curriculum spider web

Oneofthemajorchallengesforcurriculumimprovementiscreatingbalanceandconsistencybetweenthevariouscomponentsofacurriculum(i.e.planforlearning).Whatarethosecomponents?TherelativelysimplecurriculumdefinitionbyWalker(2003)includesthreemajorplanningelements:content,purposeandorganizationoflearning.However,curriculumdesignandimplementationproblemshavetaughtusthatitiswisetopayexplicitattentiontoamoreelaboratedlistofcomponents.Elaboratingonvarioustypologies,wehavecometoadheretoaframework(seeBox2)oftencomponentsthataddresstenspecificquestionsabouttheplanningofstudentlearning.

Rationale or Vision Why are they learning?Aims & Objectves Toward which goals are they learning?Content What are they learning?Learning activities How are they learning?Teacher role How is the teacher facilitating learning?Materials & Resources With what are they learning?Grouping With whom are they learning?Location Where are they learning?Time When are they learning?Assessment How to measure how far learning has progressed?

Box 2: Curriculum components


Therationale(referringtooverallprinciplesorcentralmissionoftheplan)servesasmajororientationpoint,andthenineothercomponentsareideallylinkedtothatrationaleandpreferablyalsoconsistentwitheachother.Foreachofthecomponentsmanysub-questionsarepossible.Notonlyonsubstantiveissues(seethenextsection),but,forexample,alsoonorganizationalaspectsas: Grouping: - Howarestudentsallocatedtovariouslearningtrajectories? - Arestudentslearningindividually,insmallgroups,orwhole-class? Location: - Arestudentslearninginclass,inthelibrary,athome,orelsewhere? - Whatarethesocial/physicalcharacteristicsofthelearningenvironment? Time: - Howmuchtimeisavailableforvarioussubjectmatterdomains? - Howmuchtimecanbespentonspecificlearningtasks?Therelevanceofthesecomponentsvariesacrossthepreviouslymentionedcurriculumlevels(supra,macro,meso,micro,nano)andrepresentations.Afewexamplesmayillustratethis. Curriculumdocumentsatthemacro-levelwillusuallyfocusonthefirstthree

components(rationale,aims&objectives,content;ofteninratherbroadterms),sometimesaccompaniedbyanoutlineoftimeallocationsforvarioussubjectmatterdomains.

Whenonetakestheoperationalcurriculuminschoolsandclassroomsinmind,alltencomponentshavetobecoherentlyaddressedtoexpectsuccessfulimplementationandcontinuation.

Thecomponentsoflearningactivities,teacherrole,andmaterials&resourcesareatthecoreofthemicro-curriculumintheclassroom.

Thecomponentofassessmentdeservesseparateattentionatalllevelsandrepresentationssincecarefulalignmentbetweenassessmentandtherestofthecurriculumappearstobecriticalforsuccessfulcurriculumchange.

Ourpreferentialvisualizationofthetencomponentsistoarrangethemasaspiderweb(Figure1),notonlyillustratingitsmanyinterconnections,butalsounderliningitsvulnerability.Thus,althoughtheemphasisofcurriculumdesignonspecificcomponentsmayvaryovertime,eventuallysomekindofalignmenthastooccurtomaintaincoherence.AstrikingexampleisthetrendtowardintegrationofICTinthecurriculum,withusuallyinitialattentiontochangesinmaterialsandresources.Manyimplementationstudieshaveexemplifiedtheneedforamorecomprehensiveapproachandsystematicattentiontotheothercomponentsbeforeonecanexpectrobustchanges.


Thespiderwebalsoillustratesafamiliarexpression:everychainisasstrongasitsweakestlink.Thatseemsanotherveryappropriatemetaphorforacurriculum,pointingtothecomplexityofeffortstoimprovethecurriculuminabalanced,consistentandsustainablemanner.

Figure 1: Curricular spider web

Perspectives on substantive choices

Aclassicapproachtotheeternalcurriculumquestionofwhattoincludeinthecurriculum(orevenmoredifficultaswellasurgent:whattoexcludefromit)istosearchforabalancebetweenthreemajorsourcesororientationsforselectionandprioritysetting: Knowledge:whatistheacademicandculturalheritagethatseemsessentialfor

learningandfuturedevelopment? Society:whichproblemsandissuesseemrelevantforinclusionfromtheperspectiveof

societaltrendsandneeds? Learner:whichelementsseemofvitalimportanceforlearningfromthepersonaland

educationalneedsandinterestsofthelearnersthemselves?

Assessm

ent

Aims & Ob

jectives

ContentLearning activitiesTeache

r role

Materials & Resources

Location

Time

Rationale

Grouping


Answerstothesequestionsusuallyconstitutetherationaleofacurriculum.Inevitably,choiceshavetobemade,usuallyinvolvingcompromisesbetweenthevariousorientations(andtheirrespectiveproponentsandpressuregroups).Oftentimes,effortsfailtoarriveatgenerallyacceptable,clearandpracticalsolutions.Theresultofaddingupallkindsofwishesisthatcurriculatendtogetoverloadedandfragmented.Implementationofsuchincoherentcurriculaeventuallytendstoleadtostudentfrustrations,failure,anddropout.Howtocreateabettercurriculumbalance?Easyanswersarenotavailable,butafewalternativesseemtohavesomepromise.First,inviewofthemultitudeof(academic)knowledgeclaims,itsometimeshelpstoreducethebignumberofseparatesubjectdomainstoamorelimitednumberofbroaderlearningareas,combinedwithsharperprioritiesinaimsforlearning(focusingonbasicconceptsandskills).Second,referringtotheavalancheofsocietalclaims,moreinteractionbetweenlearninginsideandoutsidetheschoolmayreducetheburden.However,themosteffectiveresponseisprobablytobemoreselectiveinreactingtoallsortsofsocietalproblems.AsCuban(1992)phraseditclearly:schoolsshouldnotfeelobligedtoscratchthebackofsocietyeverytimesocietyhasanitch.Andthird,aboutthelearnersperspective:worldwide,manyinterestingeffortsareongoingtomakelearningmorechallengingandintrinsicallymotivatingbymovingfromtraditional,teacher-andtextbook-dominatedinstructiontowardsmoremeaningful,activity-basedandautonomouslearningapproaches.

Development strategies

Tosketchcurriculumdevelopmentasaproblematicdomainisactuallyanunderstatement.Fromasocio-politicalstance,itseemsoftenmoreappropriatetodescribeitasawarzone,fullofconflictsandbattlefieldsbetweenstakeholderswithdifferentvaluesandinterests.Problemsmanifestthemselvesinthe(sometimesspectacularandpersistent)gapsbetweentheintendedcurriculum(asexpressedinpolicyrhetoric),theimplementedcurriculum(reallifeinschoolandclassroompractices),andtheattainedcurriculum(asmanifestedinlearnerexperiencesandoutcomes).Atypicalconsequenceofthosetensionsisthatvariousfrustratedgroupsofparticipantsblameeachotherforthefailureofreformorimprovementactivities.Althoughsuchblaminggamesoftenseemratherunproductive,therearesomeseriouscriticalremarkstobemadeonmanycurriculumdevelopmentapproachesworldwide.Firstofall,manycurriculumreformeffortscanbecharacterizedbyoverlybiginnovationambitions(especiallyofpoliticians)withinunrealisticallyshorttimelinesandwithverylimitedinvestmentinpeople,especiallyteachers.Second,oftentimesthereisalackofcoherencebetweentheintendedcurriculumchangeswithothersystemcomponents(especiallyteachereducationandassessment/examinationprograms).And


lastbutnotleast,timelyandauthenticinvolvementofallrelevantstakeholdersisoftenneglected.Fromastrategicpointofview,theliteraturehasofferedusmany(technical-professional)modelsandstrategiesforcurriculumdevelopment.ThreeprominentapproachesareTylersrational-linearapproach,Walkersdeliberativeapproach,andEisnersartisticapproach.Asitdoesnotfitwithinthepurposeofthischaptertoexplainthosemodelsinparticular,thereaderisreferredtoeducativetextsasfromMarshandWillis(2003).Obviously,thecontextandnatureofthecurriculumdevelopmenttaskathandwilldeterminetoalargeextentwhatkindofstrategyisindicated.Itisnoteworthythatwearebeginningtoseemoreblendedapproachesthatintegratevarioustrendsandcharacteristicsofrecentdesignanddevelopmentapproachesinthefieldofeducationandtraining(foranoverviewandaseriesofexamples:seevandenAkker,Branch,Gustafson,Nieveen&Plomp,1999).Somekeycharacteristics: Pragmatism:Recognitionthatthereisnotasingleperspective,overarchingrationaleor

higherauthoritythatcanresolvealldilemmasforcurriculumchoicestobemade.Thepracticalcontextanditsusersareintheforefrontofcurriculumdesignandenactment.

Prototyping:Evolutionaryprototypingofcurricularproductsandtheirsubsequentrepresentationsinpracticeisviewedasmoreproductivethanquasi-rationalandlineardevelopmentapproaches.Gradual,iterativeapproximationofcurriculardreamsintorealitiesmaypreventparalysisandfrustrations.Formativeevaluationoftentative,subsequentcurriculumversionsisessentialtosuchcurriculumimprovementapproaches.

Communication:Acommunicative-relationalstyleisdesirableinordertoarriveattheinevitablecompromisesbetweenstakeholderswithvariousrolesandinterestsandtocreateexternalconsistencybetweenallpartiesinvolved.

Professional development:Inordertoimprovechancesonsuccessfulimplementation,thereisatrendtowardsmoreintegrationofcurriculumchangeandprofessionallearninganddevelopmentofallindividualsandorganizationsinvolved.

Designordevelopment(al)researchisaresearchapproachthatincorporatessomeofthesecharacteristics,anditbecomesevenmorepromisingbyaddingtheelementofknowledgegrowthtoit(vandenAkker,1999).Suchresearchcanstrengthentheknowledgebaseintheformofdesignprinciplesthatofferheuristicadvicetocurriculumdevelopmentteams,when(morethanincommondevelopmentpractices)deliberateattentionispaidtotheoreticalembeddingofdesignissuesandempiricalevidenceisofferedaboutthepracticalityandeffectivenessofthecurricularinterventionsinrealusersettings.However,thereareseveralpersistentdilemmasforcurriculumdevelopmentthatcannoteasilyberesolved,letalonethroughgenericstrategies.Forexample:howtocombineaspirationsforlarge-scalecurriculumchangeandsystemaccountabilitywiththeneedfor


localvariationsandownership?Thetensionbetweentheseconflictingwishescanbesomewhatreducedwhenoneavoidsthealltoocommononesizefitsallapproach.Moreadaptiveandflexiblestrategieswillavoiddetailedelaborationandover-specificationofcentralcurriculumframeworks.In-stead,theyoffersubstantialoptionsandflexibilitytoschools,teachers,andlearners.Althoughstrugglesaboutprioritiesinaimsandcontentwillremaininevitable,theprincipleoflessismoreshouldbepursued.However,whatisincorporatedinacorecurriculumshouldbeclearlyreflectedinexaminationandassessmentapproaches.Theenactmentperspective(teachersandlearnerstogethercreatetheirowncurriculumrealities)isincreasinglyreplacingthefidelityperspectiveonimplementation(teachersfaithfullyfollowcurricularprescriptionsfromexternalsources).Thistrendputsevenmoreemphasisonteachersaskeypeopleincurriculumchange.Bothindividualaswellasteamlearningisessential(Fullan,2001).Teachersneedtogetoutoftheircustomaryisolation.Collaborativedesignandpilotingofcurricularalternativescanbeveryproductive,especiallywhenexperiencesareexchangedandreflecteduponinastructuredcurriculumdiscourse.Interactionwithexternalfacilitatorscancontributetocarefulexplorationsofthezoneofproximaldevelopmentofteachersandtheirschools.Cross-fertilizationbetweencurriculum,teacher,andschooldevelopmentisaconditiosinequanonforeffectiveandsustainablecurriculumimprovement.Theincreasinglypopularmissionstatementsofschoolstobecomeattractiveandinspiringenvironmentsforstudentsandteacherscanonlyberealizedwhensuchintegratedscenariosarepractised.

The potential of curriculum design research

Variousmotivesforinitiatingandconductingcurriculumdesignresearchcanbementioned.Abasicmotivestemsfromtheexperiencethatmanyresearchapproaches(e.g.experiments,surveys,correlationalanalyses),withtheirfocusondescriptiveknowledge,hardlyprovideprescriptionswithusefulsolutionsforavarietyofdesignanddevelopmentproblemsineducation.Probablythegreatestchallengeforprofessionaldesignersishowtocopewiththemanifolduncertaintiesintheircomplextasksinverydynamiccontexts.Iftheydoseeksupportfromresearchtoreducethoseuncertainties,severalfrustrationsoftenarise:answersaretoonarrowtobemeaningful,toosuperficialtobeinstrumental,tooartificialtoberelevant,and,ontopofthat,theyusuallycometoolatetobeofanyuse.Curriculumdesignersdoappreciatemoreadequateinformationtocreateasolidgroundfortheirchoicesandmoretimelyfeedbacktoimprovetheirproducts.Moreover,theprofessionalcommunityofdevelopersasawholewouldbehelpedbyagrowingbodyofknowledgeoftheoreticallyunderpinnedandempiricallytesteddesignprinciplesandmethods.


Anotherreasonforcurriculumdesignresearchstemsfromthehighlyambitiousandcomplexnatureofmanycurriculumreformpoliciesineducationworldwide.Thesereformendeavorsusuallyaffectmanysystemcomponents,areoftenmulti-layered,includingbothlarge-scalepoliciesandsmall-scalerealization,andareverycomprehensiveintermsoffactorsincludedandpeopleinvolved.Thoseradicalrevolutions,ifpromisingatall,cannotberealizedonthedrawingtable.Thescopeofdiverseneedsisoftenverywide,theproblemstobeaddressedareusuallyill-specified,theeffectivenessofproposedinterventionsismostlyunknownbeforehand,andtheeventualsuccessishighlydependentonimplementationprocessesinabroadvarietyofcontexts.Therefore,suchcurriculumreformeffortswouldprofitfrommoreevolutionary(interactive,cyclic,spiral)approaches,withintegratedresearchactivitiestofeedtheprocess(bothforwardandbackward).Suchanapproachwouldprovidemoreopportunitiesforsuccessiveapproximationoftheidealsandformorestrategiclearningingeneral.Inconclusion:curriculumdesignresearchseemsawiseandproductiveapproachforcurriculumdevelopment.

Features of curriculum design research

Curriculumdesignresearchisofteninitiatedforcomplex,innovativetasksforwhichonlyveryfewvalidatedprinciplesareavailabletostructureandsupportthedesignanddevelopmentactivities.Sinceinthosesituationstheimageandimpactoftheinterventiontobedevelopedisoftenstillunclear,theresearchfocusesonrealizinglimitedbutpromisingexamplesofthoseinterventions.Theaimisnottoelaborateandimplementcompleteinterventions,buttocometo(successive)prototypesthatincreasinglymeettheinnovativeaspirationsandrequirements.Theprocessisoftencyclicorspiral:analysis,design,evaluationandrevisionactivitiesareiterateduntilasatisfyingbalancebetweenidealsandrealizationhasbeenachieved.

Towhatextentdothesedesignresearchactivitiesdifferfromwhatistypicalfordesignanddevelopmentapproachesinprofessionalpractices?Whataretheimplicationsoftheaccountabilityofresearcherstothescientificforum?Attheriskofexaggeratingthedifferences,letusoutlinesomeofthem,basedonwhatisknownaboutroutinizedstandard-patternsincurriculumdevelopmentpractices.Ofcourse,alotofactivitiesaremoreorlesscommonforbothapproaches,sothefocuswillbeonthoseadditionalelementsthataremoreprominentindesignresearchthanincommondesignanddevelopmentpractices.

(1)PreliminaryinvestigationAmoreintensiveandsystematicpreliminaryinvestigationofcurriculumtasks,problems,andcontextismade,includingsearchingformoreaccurateandexplicitconnectionsofthat


analysiswithstate-of-the-artknowledgefromliterature.Sometypicalactivitiesinclude:literaturereview;consultationofexperts;analysisofavailablepromisingexamplesforrelatedpurposes;casestudiesofcurrentpracticestospecifyandbetterunderstandneedsandproblemsinintendedusercontexts.

(2) TheoreticalembeddingMoresystematiceffortsaremadetoapplystate-of-the-artknowledgeinarticulatingthetheoreticalrationaleforcurriculumdesignchoices.Moreover,explicitfeedbacktoassertionsinthedesignrationaleaboutessentialcharacteristicsoftheintervention(substantivedesignprinciples)ismadeafterempiricaltestingofitsquality.Thistheoreticalarticulationcanincreasethetransparencyandplausibilityoftherationale.Becauseoftheirspecificfocus,thesetheoreticalnotionsareusuallyreferredtoasmini-orlocaltheories,althoughsometimesconnectionscanalsobemadetomiddle-rangetheorieswithasomewhatbroaderscope.

(3) EmpiricaltestingClearempiricalevidenceisdeliveredaboutthepracticalityandeffectivenessofthecurriculumfortheintendedtargetgroupinrealusersettings.Inviewofthewidevariationofpossibleinterventionsandcontexts,abroadrangeof(direct/indirect;intermediate/ultimate)indicatorsforsuccessshouldbeconsidered.

(4) Documentation,analysisandreflectiononprocessandoutcomesMuchattentionispaidtosystematicdocumentation,analysisandreflectionontheentiredesign,development,evaluationandimplementationprocessandonitsoutcomesinordertocontributetotheexpansionandspecificationofthemethodologyofcurriculumdesignanddevelopment.

Morethanmostotherresearchapproaches,designresearchaimsatmakingbothpracticalandscientificcontributions.Inthesearchforinnovativesolutionsforcurriculumproblems,interactionwithpractitioners(invariousprofessionalroles:teachers,policymakers,developers,andthelike)isessential.Theultimateaimisnottotestwhethertheory,whenappliedtopractice,isagoodpredictorofevents.Theinterrelationbetweentheoryandpracticeismorecomplexanddynamic:is

introduction to education design research

Documents

research approach

various educational

tjeerd plomp2

policy circles

policy makers

national policy

contexts of policy

relevant curriculum