ReportingandDisseminationofLarge-ScaleLearning

Assessments:TheBiggerPictureSallyRobertsonandUrsulaSchwantner

ACER,CentreforGlobalEducationMonitoring(GEM)

Overview

• IntroductiontoACERandtheCentreforGlobalEducationMonitoring(GEM)

• Elementsofacomprehensiveassessmentsystem• Purposesoflarge-scaleassessmentsandreportinganddissemination

• Relationshipofreportinganddisseminationtoothercomponentsofanassessmentprogram

• Workshoptopicsanddiscussion

ACER

• AustralianCouncilforEducationalResearch

• Establishedin1930asanot-for-profitresearchorganisation

• ACER’smission:Creatingandpromotingresearch-basedknowledge,productsandservicestoimprovelearning

ACERresearch

https://www.acer.edu.au/research

• OECD-PISA• IEA-ICCS• IEA-TEDS• InternationalprojectsincollaborationwithDevelopmentPartners

ACERinternationalcontext

ACERCentreforGlobalEducationMonitoring

• TheGEMCentresupportsthemonitoringofeducationaloutcomesworldwidetoinformevidence-basededucationpolicyaimedatimprovingeducationalprogressforalllearners.

• CollaborationbetweenACERandDFAT

GEMkeyareas

• MTEGisacollaborativeapproachtodevelopandimplementanassessmentthatmeetsthespecificmonitoringneedsofacountry.

• MTEGisbasedonasetofdefineddesignprinciplesandqualitystandards,yieldinghigh-qualityassessmentdataofnationalrelevance.

https://www.acer.edu.au/gem/key-areas/system-strengthening/mteg

Monitoring Trends in Educational Growth partnership service

Assessmentpurpose

“Thefundamentalpurposeofassessmentineducationistoestablishandunderstandwherelearnersareinanaspectoftheirlearningatthetimeofassessment.”(Masters,2014,p.1)

Componentsofanassessmentsystem

• Classroombasedassessments

• Schoolbasedassessments

• Nationalbasedcensusassessments

• Nationalbasedsurveyassessments

• Internationalandregionalassessments

Classroombasedassessments

• Formonitoringtheprogressofindividualsandprovidinginformationindeterminingappropriatepedagogicalresponse

Schoolbasedassessments

• Formonitoringtheprogressofindividualsortoaidtheselectionoffutureeducationalpathways

• Couldbesingleschoolassessmentsordistrictbasedtoallowcomparisonsofprogress

Nationalbasedcensusassessments

• Formonitoringtheprogressofindividuals,foraccreditationpurposes,ortoaidtheselectionoffutureeducationalpathways

• Canbeusedforsub-populationmonitoringpurposes

Nationalbasedsurveyassessments

• Formonitoringthehealthoftheeducationsystem

• Formakingsub-nationalcomparisonsofeducationaloutcomessuchasbetweenprovinces

• Forbenchmarkingnationalperformanceagainstthatofothercountries

• Canbeusedforsub-populationmonitoringpurposes

Internationalandregionalassessments

Internationalassessments

Regionalassessments

Source:http://www.seaplm.org/seaplm/index.php/countries

Participationininternational,regionalandnationalassessments

• IEAconductedfirstinternationallycomparativestudyinmathematicswith12countriesin1964

• Therehasbeenagrowthinparticipationinalllarge-scaleassessments,particularlyamongstdevelopingcountries(KamensandMcNeely,2009)

• Between2007-2013,67%ofcountriesintheAsia-Pacificregionconductedanationalassessment(BenavotandKöseleci,2015)

Large-scaleassessments

• Thefocusoftheworkshopthisweekisonthereportinganddisseminationoflarge-scaleassessments.

• Large-scaleassessmentsare:– Representativeofaneducationsystem– Standardised– Notnationalexaminations– National,regionalandinternationalassessments

Tobin,Lietz,Nugroho,Vivekanandan,&Nyamkhuu(2015)

Examinationsvs.large-scaleassessmentsExamination Monitoring Assessment

Purpose Ranking students to select the most able students for the next stages of education and/or to certify students Measuring the degree to which curricular content has been learned

Quantify differences in learning between sub-populations Measure changes in learning over time Establish and describe the range of what students know

Type Census Sample or census

Content Based entirely on the intended curriculum of a single grade level

May be inside or outside of the curriculum May be at higher or lower levels of the intended curriculum for a particular grade

Approach Focus on whether the skills have been attained and the content has been learned

Focus on whether the learner can apply the skills and concepts that she or he has acquired and learned

Analysis Simple, percent-correct scores Complex statistical methods are requiredReporting Reports at the individual level Reports at the level of populations and sub-

populations

Evaluatingandmonitoringthequalityoftheeducationsystem

Ensuringequity

Exercisingaccountability

Purposesoflarge-scaleassessments

Tobin,Lietz,Nugroho,Vivekanandan,&Nyamkhuu(2015)

Policyimpactsoflarge-scaleassessments

Tobin,Lietz,Nugroho,Vivekanandan,&Nyamkhuu(2015)

IntegrationintopolicyprocessesMediaandpublicopinionQualityoftheassessmentprogram

Factorsthatinfluencetheutilisationineducationpolicy

Tobin,Lietz,Nugroho,Vivekanandan,&Nyamkhuu(2015)

Keyareasofarobustassessment

Source:ACER-GEM(https://www.acer.edu.au/gem/about/approach)

Policygoalsandissues

• Large-scaleassessmentsshouldbedesignedwiththeaimofobtainingdatatohelpaddresspolicyissuesofinterestandintendedpolicygoals.

Quality Equity Accountability

Projectteamandinfrastructure

• Essentialtothesuccessofanylarge-scaleassessmentare:– Adequatenumberofskilledstafftocoverthedifferentrolesneeded(e.g.,projectmanagement,itemdevelopment,sampling,analysis,reportingandcommunications)

– Adequateresourcesandacorephysicalinfrastructure

Technicalstandards

Thetechnicalqualityoftheassessmentcanactasabarrierorfacilitatortotheuseofassessmentresultsineducationdecision-making(Tobinetal.,2015).

Technicalstandards

• Guidelinestoensuretheconsistency,precisionandgeneralisabilityofthedata,forexample:– samplingstandards– datastandards– psychometricstandards

• Theoverallgoalistocreateadatasetofaqualitythatallowsvalidinferencestobemade.

Assessmentframework

• Explicitstatementanddiscussionaboutwhatanassessmentintendstomeasure

• Laysouttheprinciplesuponwhichanassessmentisbuilt

• Guidestestdevelopmenthttps://www.oecd.org/pisa/pisaproducts/PISA%202012%20framework%20e-book_final.pdf

Contextualframework• Basedontheoryorevidencefromresearchliterature

• Formulatesandaddressesimportantpolicyquestionsandissues

• Guidesthequestionnairedevelopment• Guidestheanalysisandreporting

OECD,2013,p.175

Highqualityassessmentitems…

• meettherequirementsoftheassessmentframework

• coverarangeofskills• arewelltargetedtostudents’abilities• arefair• willimprovetheexplanatorypoweroftheassessmentandincreaseusefulnessoftheassessmentfindings

Linguisticqualitycontrol

• Whereassessmentsareadministeredindifferentlanguages,thedifferentversionsshouldbeequivalent.

Testdesign

Lumley,Mendelovits,Stanyon,Turner&Walker(2015),p.1

Sampledesign

• Toenablevalidinferencebeyondthestudentsassessedtoapopulationofstudents,scientificsamplingtechniquesmustbeemployed.

Standardisedfieldoperations

• Standardisedtestadministrationproceduresarevitaltoensurethateverystudentisassessedunderthesameconditions.

Datamanagement

• Qualitycontrolprocessesshouldbeimplementedatallstagesofdatamanagementincluding:– dataentryandprocessing– creatingthedatabase– preparingthedataforreporting

• Itisessentialtoprovidefulldocumentationwiththedatabase.

ScalingmethodologyLumley,Mendelovits,Stanyon,Turner&Walker(2015)

Dataanalysis

• Analysingdatafromacomplexsurveydesignrequiresadvancedstatisticaltechniques.

• Dataanalystsneedtobehighlyskilledtoappropriatelyapplyandinterprettheanalyses.

• Adataanalysisplanshouldbedevelopedtoinformtheassessmentdesignandtoprovideinformationaboutthekindoffindingsthatwillbepossible.

Reportinganddissemination

• Reportinganddisseminationshouldbeguidedbyaneffectivestrategy,andincorporatearangeofapproachestocaterforadiversityofaudiences.

• Themainaimofallreportingshouldbetosupportanunderstandingoftheassessmentanditsfindings,inresponsetothepolicygoalsdefinedfortheassessmentprogram.

Reportinganddisseminationintheassessmentcycle

Source:ACER-GEM(https://www.acer.edu.au/gem/about/approach)

Whatishappeningoverthenextthreedays?

Wednesday•Disseminationapproachesfordifferentpurposesandaudiences•Developingadisseminationstrategy•DevelopingassessmentproductsThursday•Reportingachievementandcontextualdata•Interpretingassessmentresults•TranslatingassessmentfindingsintopolicyandpracticeFriday•Enhancingpublicawarenessofassessmentresults•Assessmentwebsitesasadisseminationtool

Discussion

• Whatisyourroleinassessmentreportinganddissemination?

• Whatdisseminationproductshaveyoupreviouslybeeninvolvedindeveloping/reviewing/editing?

• Whatchallengeshaveyoufacedwithreportinganddissemination?Howdidyouaddressthesechallenges?

• Whatisonequestionyou’rehopingtohaveansweredduringthisworkshop?

ReferencesBenavot,A.,&Köseleci,N.(2015).Seekingqualityineducation:Thegrowthofnationallearningassessments, 1990–2013.PapercommissionedfortheEFAGlobalMonitoringReport2015,EducationforAll2000–2015:Achievementsandchallenges.Paris:UNESCO.Kamens,D.H.,&McNeely,C.L.(2009).Globalizationandthegrowthofinternationaleducationaltestingand nationalassessment.ComparativeEducationReview,54(1),5–25.Lumley,T.,Mendelovits,J.,Stanyon,R.,Turner,R.,&Walker,M.(2015).Class6proficiencyinAfghanistan2013:Outcomesofalearningassessmentofmathematical,readingandwritingliteracy.Melbourne:AustralianCouncilforEducationalResearch.Masters,G.(2014).Assessment:Gettingtotheessence.Melbourne:AustralianCouncilforEducationalResearch.OECD.(2013).PISA2012AssessmentandAnalyticalFramework:Mathematics,Reading,Science,ProblemSolvingandFinancialLiteracy:OECDPublishing.OECD.(2010).PISA2009Results:WhatStudentsKnowandCanDo–StudentPerformanceinReading,MathematicsandScience(VolumeI):OECDPublishing.Tobin,M.,Lietz,P.,Nugroho,D.,Vivekanandan,R.,&Nyamkhuu,T.(September2015).Usinglarge-scaleassessmentsofstudents’learningtoinformeducationpolicy:InsightsfromtheAsia-Pacificregion.Melbourne:ACERandBangkok:UNESCO.ForinformationaboutACERsee:https://www.acer.edu.au/ForinformationaboutACER’sGEMCentresee:https://www.acer.edu.au/gem