TowardanEvidenceFrameworkforEvalua4ngOnlineLearning

BarbaraMeansSRIInterna.onal

RussellRumbergerUCSantaBarbara

Theworkdescribedinthispresenta.onwassupportedbytheOfficeofEduca.onalTechnology,U.S.DepartmentofEduca.onthroughcontractED04‐CO‐0040withSRIInterna.onal.

EvidenceFrameworkReport•  EffortoftheOfficeofEduca.onalTechnologyoftheU.S.DepartmentofEduca.on

•  Explainswhythetransi.ontodigitallearningwarrantsrethinkingeduca.onalevidence

•  Describesnewapproachestoevidencegathering•  Explainshowthenewerapproachescanbeappliedto:−  Developingeffec.vedigitallearningresources−  Dynamicallyadap.nglearningtoindividualstudents−  Suppor.ngstudents’learningandwell‐beinginandoutofschool− Measuringhard‐to‐assesslearnercompetencies−  Findingdigitallearningresourcesthatwillmeetaprac..oner’sneeds

andpreferences

•  PresentsaDecisionModelprovidinggeneralguidancewithrespecttothestrengthofevidenceneededfordifferentkindsoflearningtechnologydecisions

•  Recommendsac.onsforpolicymakers,prac..oners,developersandresearchers

•  fo

LearningTechnologyEvidenceFrameworkTechnicalWorkingGroup

EvaBaker,UCLA JimPellegrino,UniversityofIllinoisatChicago

AllanCollins,NorthwesternUniversity

WilliamPenuel,UniversityofColorado

ChrisDede,HarvardUniversity ZoranPopovic,UniversityofWashington

AdamGamoran,UniversityofWisconsin‐Madison

SteveRiXer,CarnegieLearning

KenjiHakuta,StanfordUniversity RussRumberger.UniversityofCalifornia,SantaBarbara

KenKoedinger,CarnegieMellonUniversity

MikeSmith,CarnegieFounda.onfortheAdvancementofTeaching

DavidNeimi,KaplanInc. PhoenixWang,WilliamPennFounda.on

Implica4onsoftheShiBtoDigitalLearningfortheNatureofEvidence

•  ThedevelopmentofdigitallearningresourcesforK‐12studentsisexploding.

•  TheK‐12useofdigitallearningresourcesisalsorisingveryquickly.

•  Butthestudyofdigitallearningresourceshasnotkeptpace.

•  Wearejustbeginningtounderstandallthethingswecandowiththedatageneratedbydigitallearningsystems.

GoalsoftheEvidenceFramework

•  Ge^nginforma.onontheuseandoutcomesofnewlearningtechnologiesmuchmorequickly

•  Implemen.ngcon.nuousimprovementprocessesleveragingthedatafromlearningtechnologiessoeduca.oncangetbeXerover.me

•  Evidence‐supporteddecision‐makingthattakesintoaccounttherangeoffactorspeoplecareabout

SixEmergingSourcesofEvidence

•  Dataminingandlearninganaly.csappliedtodatafrommassivelyimplementedlearningsystems

•  RapidA/Btes.ngwithinonlinesystems

•  Design‐basedimplementa.onresearch

•  Meta‐datafromusersaccessingweb‐baseddigitalresources

•  Technology‐supportedevidence‐centereddesign•  Sharingdatasetsacrossprojects;linkingindividual‐leveldataacrossagencies

SomeKeyIdeasfromtheReport‐1

•  Effec.venessisnotatraitinherentinatechnologyorinterven.onperse.

•  Ideally,learningresourcesandtheirimplementa.onwouldbecon.nuallyrefinedinaprocesswithfeedbackloops.(Design‐basedimplementa6onresearch)

•  Internetdistribu.onofdigitallearningresourcesandthedatageneratedwhenusersaccessandusethoseresourcespermitquickandefficientgenera.onofevidenceatscale.(A/Btes6ng;datamining)

•  Buttherearelimitstowhatcanbelearnedbylookingsolelyatsystemdata—mostnotably,theextenttowhichlearnerperformancewiththesystempredictsperformanceinothertasksandse^ngs.

Howcanwemakesurethatatechnology‐basedinterven6oniseffec6veinfillingthegoalswesetforit?

SomeKeyIdeasfromtheReport‐2

•  Measuressuchasresponselatency,theamountofscaffoldingneeded,choiceofproblemorsolu.onpath,andresponsetofeedbackcanbeusedtobeXerunderstandanindividualstudent’slearning.(Evidence‐centereddesign)

•  Collec.ngtensofthousandsofdatapointsperlearnerperday,sophis.cateddigitallearningsystemscancon.nuouslyofferastudentapproachesthathaveworkedforother,similarlearners.

•  Thelargeamountsofmicro‐leveldatageneratedbylearningsystemscanleadtonewinsightsaboutthevariabilityandconstanciesoflearning.

•  New‐genera.onstudentdatasystemsarestar.ngtotrackawiderrangeofstudentinforma.on.Administra.vedataarebeinglinkedwithdatafromlearningmanagementsystems.(Linkeddatasystems)

•  Predic.vemodelscanbedevelopedfromthesedataandusedin“earlywarningsystems”thatprovide.melyfeedbacktobothstudentsandeducators.

Whatcanwedowithdatacollectedfromdigitallearningsystems?

SomeKeyIdeasfromtheReport‐3

•  Internet‐distribu.onenablesuserstomodify,create,share,andreviewlearningresourcesthemselves.

•  Usersconsidermul.pleaspectsofaproduct‐‐includingitsdesignprocess;endorsements;alignmenttostandards;fitfortheircontext;andeaseofuse;aswellasevidenceofeffec.veness.

•  Onlinerepositoriesandcommuni.esarehelpinginstructorsselectandcombinedigitallearningresources.Theyaremakingra.ngsandreviewsbyusersandexperts,userpanels,andtestbedswidelyavailable.(Meta‐datafromusers)

•  Dataminingcanbeusedtomakerecommenda.onsofdigitallearningresourcesthatsimilarusershavedownloadedandratedhighly.

Howcanwehelpeducatorsfindreliablesourcesofevidenceaboutdigitallearningresourcesanduseittomaketheirdecisions?

ConfidenceofImprovement

HighConfidenceHighRisk

HighConfidenceLowRisk

LowConfidenceHighRisk

LowConfidenceLowRisk

EvidenceDecision‐MakingModel

Im

plem

enta4on

Risk

Par4ngThoughts

•  Wehavethetechnicalcapabilitytocollectunprecedentedquan..esofmicro‐learningdatawithinonlinelearningsystems.

•  Thesedatacanhelpusassessstudentlearningandcoursebehaviorwhenthereiss.ll.meforcorrec.veac.on,

•  ANDthesedatacanbeusedinacon.nuousimprovementprocesstomakethelearningsystembeXerandbeXer.

•  BUTwes.llneedtodemonstratethatonlinelearningsystemswithhighimplementa.onrisk(becauseofscale,cost,stakesorimplementa.oncomplexity)producelearningoutcomesatleastasgoodasthoseofconven.onalinstruc.on,

•  ANDwecangetsmarterabouthowtodesignandimplementdigitallearningifwestudyitsystema.cally.

WatchforpublicreleaseathXp://evidenceframework.org/

Headquarters:SiliconValley

SRIInterna4onal333RavenswoodAvenueMenloPark,CA94025‐3493650.859.2000

Washington,D.C.

SRIInterna4onal1100WilsonBlvd.,Suite2800Arlington,VA22209‐3915703.524.2053

Princeton,NewJersey

SRIInterna4onalSarnoff201WashingtonRoadPrinceton,NJ08540609.734.2553

Addi6onalU.S.andinterna6onalloca6ons

www.sri.com

ThankYou