INDIANATEACHERAPPRAISALSYSTEMSOFSUPPORT(INTASS)STATEBOARDOFEDUCATIONRESEARCHREPORTFORTEACHEREVALUATIONPLANIMPLEMENTATION
2012-13THROUGH2015-16
• REPORTISONEOFSEVERALONTEACHEREVALUATIONININDIANASUPPORTEDBYTHEINDIANASTATEBOARDOFEDUCATIONANDTHEJOYCEFOUNDATION
• RESEARCHDESIGNEMPLOYEDAMULTI-YEARAPPROACHWITHADATASETINCLUDINGMORETHAN2,352,964+ STUDENTRECORDS/199,588+EDUCATORRECORDS• DATASETINCLUDESINDIVIDUALEDUCATORANDSTUDENTATTRIBUTES,DISTRICT,SCHOOL,CLASSROOMDEMOGRAPHICS,TEACHEREVALUATIONPLANQUALITY,TEACHEREVALUATIONRATINGSANDSTUDENTOUTCOMES
• RESEARCHDESIGNINCLUDESMULTIPLEMODELSANDSTATISTICALANALYSESTOADDRESSRESEARCHQUESTIONSCONCERNINGTEACHERRATINGS,STUDENTOUTCOMESWITHPLAN QUALITY,CLASSROOM,SCHOOL,ANDDISTRICTS,ANDSTUDENTANDEDUCATORDEMOGRAPHICVARIABLES
CONTEXT:ACCOUNTABILITYENVIRONMENT2012-2016
•SENATEBILL1CHANGEDEXPECTATIONSFOREDUCATOREVALUATION•CHANGESINASSESSMENT,STANDARDSANDACCOUNTABILITY
•PERVASIVEISSUESWITHASSESSMENTADMINISTRATION
• INCONSISTENCYIN:üTEACHEREVALUATIONPOLICY&GUIDANCEINTERPRETATION
üDEVELOPMENTANDIMPLEMENTATIONOFTEACHEREVALUATIONPLANS
RESEARCHMODELCOMPONENTS
•DISTRICT,SCHOOL,EDUCATOR,ANDSTUDENTDEMOGRAPHICS,STUDENTDAYSATTENDANCE,YEARSOFPLANIMPLEMENTATION
•STUDENTANDEDUCATOROUTCOMEMEASURESRANGINGFROMSTUDENTSCALESCORES,STUDENTPROFICIENCYLEVELS,STUDENTGROWTHPERCENTILES ANDBOTHEDUCATORIGMANDSUMMATIVERATINGS
•STATE,DISTRICT,CLASSROOM,EDUCATOR,ANDSTUDENTLEVELANALYSES
LEVELSOFSIGNIFICANCETHEMULTIPLEMODELSANDANALYSESOFTENPROVIDEDFINDINGSTHATDIFFEREDINCONSISTENCYANDSIGNIFICANCE.THEFOLLOWINGSCALEWASUSEDTOINTERPRETTHE“COLLECTIVE”SIGNIFICANCE*OFSPECIFICVARIABLESWITHSTUDENTANDTEACHEROUTCOMES**:
üNONE:SIGNIFICANTINNONEOFTHESPECIFICATIONS&YEARS
üWEAK:SIGNIFICANTIN25%ORLESSOFTHESPECIFICATIONS&YEARS
üMEDIUM:SIGNIFICANTIN75%ORLESSOFTHESPECIFICATIONS&YEARS
üSTRONG:SIGNIFICANTINOVER75%OFTHESPECIFICATIONS&YEARS*WEAK&MIXEDRESULTSWERENOTFELTTOALLOWDISCERNIBLEPATTERNSOFSIGNIFICANCE
**1)STUDENTOUTCOMESREFERTOMETRICSDERIVEDFROMSTUDENTISTEPPERFORMANCEAND2)TEACHEROUTCOMESREFERTOIGMAND/ORSUMMATIVERATINGS.
STUDENTDEMOGRAPHICS,STUDENTOUTCOMESANDTEACHEREVALUATIONRATINGS
• THEREISANEGATIVERELATIONSHIPBETWEENSTUDENTFREEANDREDUCEDLUNCHSTATUS,RACIAL/ETHNICMINORITYGROUPMEMBERSHIP,ANDSTUDENTOR TEACHEROUTCOMES.• THEREISAPOSITIVERELATIONSHIPBETWEENSTUDENTATTENDANCEANDSTUDENTORTEACHEROUTCOMES.• HAVINGANIEPISNEGATIVELYASSOCIATEDWITHISTEPSCORESANDSUMMATIVERATINGS.• THEREISANEGATIVERELATIONSHIPBETWEENLIMITEDENGLISHPROFICIENTANDISTEPSCORESBUTAPOSITIVERELATIONSHIPWITHISTEPGROWTH.• THEREISNOTADISCERNIBLERELATIONSHIPBETWEENDISTRICTGROWTHWEIGHT,STUDENTDEMOGRAPHICSANDSTUDENTORTEACHEROUTCOMES.
TEACHERDEMOGRAPHICS,STUDENTOUTCOMESANDTEACHEREVALUATIONRATINGS
• THEREISNORELATIONSHIPBETWEENTEACHERRACE/ETHNICITYANDSTUDENTORTEACHEROUTCOMES.
• THEREAREPOSITIVERELATIONSHIPSBETWEENTEACHERSALARY,STUDENT PRIORISTEPSCORESANDATTENDANCEWITHSTUDENTANDTEACHEROUTCOMES.
• TEACHERYEARSOFEXPERIENCEHAVEDIFFERENTRELATIONSHIPSWITHSTUDENTANDTEACHEROUTCOMES.
• TEACHERGENDER,GRADEASSIGNMENTSHAVEDIFFERENTRELATIONSHIPWITHSTUDENTANDTEACHEROUTCOMES.
• THEREISNODISCERNIBLERELATIONSHIPWITHDISTRICTGROWTHWEIGHT,TEACHERDEMOGRAPHICSANDSTUDENTORTEACHEROUTCOMES.
DISTRICTDEMOGRAPHICS,STUDENTOUTCOMESANDTEACHEREVALUATION
•THEREISNODISCERNIBLERELATIONSHIPWITHDISTRICTGROWTHWEIGHT,DISTRICTDEMOGRAPHICS,ANDSTUDENTORTEACHEROUTCOMES
•THEREISASTRONGRELATIONSHIPBETWEENDISTRICTSTUDENTOUTCOMESINCLUDINGPRIORYEARISTEPSCORESANDTEACHEROUTCOMES
•THEREISARELATIONSHIPBETWEENDISTRICTFREEANDREDUCEDLUNCH,IGMRATINGSANDSUMMATIVERATINGS
PRINCIPALCHARACTERISTICS,STUDENTOUTCOMESANDTEACHEREVALUATIONRATINGS
• PRINCIPALSRATEDEFFECTIVERATETHEIRTEACHERSLOWERTHANPRINCIPALSRATEDHIGHLYEFFECTIVE
• PRINCIPALSRATEDNEEDSIMPROVEMENTRATETHEIRTEACHERSLOWERTHANPRINCIPALSRATEDHIGHLYEFFECTIVE
• PRINCIPALSRATEDINEFFECTIVERATETHEIRTEACHERSLOWERTHANPRINCIPALSRATEDHIGHLYEFFECTIVE
• THEREISNODISCERNIBLERELATIONSHIPWITHDISTRICTGROWTHWEIGHTANDSTUDENTORTEACHEROUTCOMES
• PRINCIPALGENDER,ETHNICITYOREXPERIENCEHAVENOSIGNIFICANTRELATIONSHIP WITHTEACHERRATINGS
CLASSROOMCOMPOSITIONANDSTUDENTANDTEACHEROUTCOMES
COMPARINGFOURTHANDFIFTHGRADECLASSROOMSWITHHIGHFREEANDREDUCEDLUNCHVERSUSLOWFREEANDREDUCEDLUNCHSHOWS:
üTHEREISASTRONGRELATIONSHIPWITHTHEPROPORTIONOFSTUDENTSONFREEANDREDUCEDLUNCHINACLASSROOMANDSTUDENTANDTEACHEROUTCOMES.
üTEACHERSINCLASSROOMSWITHHIGHERPERCENTAGESOFSTUDENTSONFREEANDREDUCEDLUNCHHAVELOWERIGMANDSUMMATIVERATINGS.
CLASSROOMCOMPOSITIONANDSTUDENTPERFORMANCE
• FOURTHANDFIFTHGRADECLASSROOMSFREEANDREDUCEDLUNCHDISTRIBUTIONDECILECOMPARISONS
üSEPARATINGTHEDISTRIBUTIONOFFOURTHANDFIFTHGRADECLASSROOMS BYDECILESSHOWSTHEREARE“BREAK”POINTSINCLASSROOMCOMPOSITIONWHEREIDENTIFIABLEDIFFERENCESINSTUDENTPERFORMANCEAREASSOCIATEDWITHPERCENTAGESOFSTUDENTSONFREEANDREDUCEDLUNCH.
• DECILECOMPARISONSOFCLASSROOMSBY%OFFREEANDREDUCEDLUNCH:
üDECILECOMPARISONSSHOWDISTINCTDIFFERENCESBETWEENCLASSROOMSBASEDONPERCENTOFSTUDENTSONFREEANDREDUCEDLUNCH.
üTHEREISAPOINTBEYONDWHICHINCREASINGFRLHASNOCORRESPONDINGDECREASEINSTUDENTOUTCOMES- TYPICALLYAROUNDTHE7TH DECILEOR65%.
üCLASSROOMSINTHESECONDDECILE(18%- 28.5%FRL)HAVESTATISTICALLYLOWERIGMSCORESTHANCLASSROOMSINTHEFIRSTDECILE(0%- 18%).
üCLASSROOMSWITH65%ORMOREOFSTUDENTSONFREEANDREDUCEDLUNCHSHOWNOSTATISTICALLYSIGNIFICANTDIFFERENCES.
üTHISSUGGESTSTHEREMAYBEIMMEDIATEDETECTABLEEFFECTSONIGMSCORES ARISINGFROMLOWSESSTUDENTSPRESENCEINACLASSROOM,BUTTHEEFFECTOFADDITIONALLOWSESSTUDENTSTAPERSOFFASTHEPROPORTIONGROWS.
IGMRATINGSANDSUMMATIVERATINGS
2013PercentofTeachers
2014PercentofTeachers
2015*Percentofteachers
SummativeratingofEffectivewithanIGMratingsofIneffective
13% 11% 12%
SummativeratingofHEwithanIGMratingsofIneffective
5% 4% 6%
SummativeratingofEffectivewithanIGMratingofNeedsImprovement
29% 33% 33%
SummativeratingofHEwithanIGMratingofNeedsImprovement
19% 16% 23%
PLANQUALITY,PLANCOMPONENTS,STUDENTOUTCOMESANDTEACHERRATINGS
•THESIXDISTRICTSRECOGNIZEDFORTHEIRPLANQUALITYALSOHADBETTERSTUDENTGROWTH
•THEREISAMEDIUMTOSTRONGRELATIONSHIPBETWEENPLANQUALITYANDTEACHERSUMMATIVERATINGSFORDISTRICTS.
•THEREISASTRONGRELATIONSHIPBETWEENPRIORYEARSTUDENTISTEPSCORESANDTEACHERRATINGSFORDISTRICTS.
TEACHERMOBILITYANDEVALUATIONRATINGS
• THEREISNORELATIONSHIPBETWEENTEACHERMOBILITYANDWHETHEROR NOTTEACHERSAREINANACCOUNTABILITYGRADEORSUBJECTAREA.
• TEACHERSTENDTOMOVEFROMDISTRICTSWITHHIGHPERCENTAGESOFSTUDENTSONFREEANDREDUCEDLUNCHTODISTRICTSWITHLOWERPERCENTAGESOFSTUDENTSONFREEANDREDUCEDLUNCH.
• THEREISNOTANASSOCIATIONWITHTEACHERMOBILITYFROMDISTRICTS WITHHIGHERPERCENTAGESOFSTUDENTSONFREEANDREDUCEDLUNCHTODISTRICTSWITHLOWERPERCENTAGESOFSTUDENTONFREEANDREDUCEDLUNCHANDCHANGESINIGMRATINGS.
• THEREISARELATIONSHIPBETWEENMOVEMENTFROMDISTRICTSWITHHIGHPERCENTAGESOFSTUDENTSONFREEANDREDUCEDLUNCHTODISTRICTSWITHLOWERPERCENTAGESOFSTUDENTSONFREEANDREDUCEDLUNCHANDSUMMATIVETEACHEREVALUATIONRATINGS.
• MOREEXPERIENCEDTEACHERSARELESSLIKELYTOMOVEDISTRICTS.
RECOMMENDATIONSMANAGEMENT&IMPLEMENTATION
• ENSUREDATAINTEGRITYINORDERFORVALIDRESEARCH-BASEDDECISIONMAKINGFORPOLICYANDGUIDANCE.
• ENSUREEXPLICITGUIDANCEONASSIGNINGSTUDENTSTOTEACHERSFORACCOUNTABILITYANDINCLUDETHISINTHEMONITORINGPROCESS.
• ENSUREVENDORMANAGEMENTANDCOMPETENCEWITHIMPACTFULPENALTIESENFORCEDFORLACKOFPERFORMANCE.
• ESTABLISHANEFFECTIVEMONITORINGSYSTEMTHATINCLUDESANANALYSISOFTEACHEREVALUATIONRATINGSANDSTUDENTOUTCOMESTOINFORMSTATESUPPORTFORTEACHERDEVELOPMENT.
RECOMMENDATIONSGROWTHMODEL
•REVIEWTHECURRENTGROWTHMODELTOENSURETHATSTUDENTCHARACTERISTICSAREEFFECTIVELYACCOUNTEDFORWHILEMAINTAININGHIGHEXPECTATIONSFORALLSTUDENTS.
•CONSIDEREXPLORINGATEACHEREVALUATIONGROWTHMODELTHATLOOKSATLIKECLASSROOMCOMPARISONSASTHEBASISFORTHEIGMRATING.
•CONSIDEREXPLORINGATEACHEREVALUATIONCLASSROOMGROWTHMODELTHATLOOKSATGROWTHBASEDUPONACLASSROOMMETRICOTHERTHANSTUDENTGROWTHPERCENTILES.
RECOMMENDATIONSTECHNICALSUPPORTFORPLANDEVELOPMENTAND
IMPLEMENTATION• PROVIDEONGOINGTRAININGANDSUPPORTTOENSUREINTER- RATERCONSISTENCYINTHETEACHEREVALUATIONPROCESS.
• ELEVATEPRINCIPALANDSUPERINTENDENTEVALUATIONQUALITYASSURANCETOTHESAMELEVELOFIMPORTANCEASTEACHEREVALUATION.
• ESTABLISHASETOFCRITERIABASEDUPONRESEARCHFINDINGSTOEVALUATEPLANDEVELOPMENTANDPLANIMPLEMENTATION.
• ESTABLISHINCENTIVESFORDISTRICTSTOENGAGEINATEACHEREVALUATIONPROCESSTHATFOCUSESONTEACHERDEVELOPMENTTHROUGHPROFESSIONALDEVELOPMENT.
• ESTABLISHAPROCESSTOENSURETHATTEACHERSAREINVOLVEDINTEACHEREVALUATIONASACOLLABORATIVEPROCESS.
• CONSIDERESTABLISHINGSTATEPROVIDEDINCENTIVESFORTEACHERSTOACCEPTTEACHINGASSIGNMENTSINDISTRICTSANDCLASSROOMSWITHHIGHPERCENTAGESOFSTUDENTSONFREEANDREDUCED LUNCH.
• PROVIDESTATESUPPORTFORPROFESSIONALDEVELOPMENTINCULTURALLY RESPONSIVEINSTRUCTION.
FUTURERESEARCHAGENDA• CONDUCTMORECOMPREHENSIVERESEARCHONTHERELATIONSHIPBETWEENHIGHQUALITYPLANIMPLEMENTATION,TEACHEREVALUATIONEXPERIENCESANDSTUDENTOUTCOMES.
• ENGAGEINQUALITATIVERESEARCHTODOCUMENTANDINCORPORATEEDUCATOREXPERIENCESINTOTHEANALYSISOFTEACHEREVALUATIONPROCESSESANDOUTCOMES.
• EVALUATETHEEFFECTIVENESSOFTECHNICALASSISTANCEANDSUPPORTONTHETEACHEREVALUATIONPROCESSININDIANA.
• CONDUCTRESEARCHONTHEINTER-RELATIONSHIPSOFCLASSROOM,SCHOOLANDDISTRICTCHARACTERISTICSONTEACHEREVALUATIONANDSTUDENTOUTCOMES.
• CONDUCTRESEARCHONSCHOOLANDDISTRICTRESOURCESANDSUPPORTSONTEACHEREVALUATIONANDSTUDENTOUTCOMES.
• CONDUCTRESEARCHONTHEDIFFERENTIALIMPACTOFTEACHEREVALUATIONONIGMVS.NON-IGMTEACHERS.
• PROVIDEFUNDSFORDISTRICTSTOENGAGEINASTATESUPPORTEDPILOT PROJECTTOREVISEANDIMPROVETHETEACHEREVALUATIONPROCESS.
AMULTI-DISCIPLINARYRESEARCHTEAM• HANNAHBOLTE,STATISTICALCONSULTANT/LECTURER,IUDEPARTMENTOFSTATISTICS
• MICHAELFRISBY,STATISTICALCONSULTANT,IUDEPARTMENTOFSTATISTICS
• DEMETREESL.HUTCHINS,MANAGEMENTANALYST,IUPUISCHOOLOFEDUCATION
• GARYPIKE,PROFESSOR,HIGHEREDUCATION&STUDENTAFFAIRS,IUPUI• SARAHPIES,RESEARCHASSOCIATE,INTASSPROJECT,INDIANAUNIVERSITY• HARDYMURPHY,IUPUIEDUCATIONALLEADERSHIPPROGRAM,EXECUTIVEDIRECTORINDIANAURBANSCHOOLSASSOCIATION,CODIRECTOROFINTASS
• SANDICOLE,DIRECTORCENTERONEDUCATIONANDLIFELONGLEARNINGIUB,CODIRECTORINTASS
• SPECIALACKNOWLEDGEMENT:THERESEARCHTEAMACKNOWLEDGESHAMMADRAHMAN,DATAMANAGEMENTSPECIALIST,INDIANADEPARTMENTOFEDUCATION,FORTHEIMPORTANTCONTRIBUTIONSMADETO THISRESEARCH.
QUESTIONS?