effective strategies for improving student learning: results from the low ses np … ·...
TRANSCRIPT
Effectivestrategiesforimproving
studentlearning:resultsfromthelowSESNP
evaluation
ShuyanHuoStephenLamb
Researchreport
UniversityofCanberraVictoriaUniversity
i
ExecutiveSummaryThis reportwas prepared by the Centre for International Research on Education Systems(CIRES) at Victoria University and is part of an evaluation of National Partnership reforminitiatives in low-SES schools. The evaluation is being undertaken by the University ofCanberrainpartnershipwithVictoriaUniversityonbehalfoftheStateofNewSouthWales.Inthisreport,theauthorsfocusonstudentperformanceintheLowSESschoolsinNSWthatparticipatedintheNationalPartnershipAgreements,theextenttowhichchangesinstudentperformance are related toNP initiatives or other factors and the initiatives thatmaybeidentifiedasmakingthestrongestcontributionstochangesinstudentoutcomes.
Previous findings associatedwith this study have already identified positive relationshipsbetweenLowSESNationalPartnershipparticipationand improvedstudentoutcomesoverthestudyperiod.
Theyinclude:
• significantimpactonstudentNAPLANresultsover3yearsbetween2011and2013inLowSESNPschools.
• scoresincreasingforeachadditionalyearofparticipationintheprogram.• reduction ingainscoregapsbetweenLowSESschoolsandothermoreadvantaged
settings.
In investigatinghow the reforms implemented inNP schools have led to improvement instudent learning outcomes and which reforms may be identified as contributing toimprovementthisreportdrawsonawidevarietyofdatasources,including:
• student levelNAPLANdata in2011and2013providedby theNSWDepartmentofEducationandTraining;
• NAPLANschoolprofiledatain2011sourcedfromACARA;• school leveldatafromthe2014LowSESNPPrincipalSurveyandthe2013lowSES
NPTeacherSurvey;and• CasestudiesofLowSESNPschools.
Thereportdevelopsthreekeysetsofanalyses: (1)estimatingaggregateschooleffectsonstudent achievement, (2) testing the relationship between aggregate effects and effortacross the broad NP reform strategy areas, and (3) identifying individual strategies orinitiativesthatwereassociatedwithstrongerstudentoutcomes.
UniversityofCanberraVictoriaUniversity
ii
AggregatelowSESNPeffects
Effects forNP schools on achievement scores in Year 5 and Year 9NAPLANReading andNumeracy were estimated using multi-level regression taking into account a number ofstudent and school level factors that influence student achievement, such as studentbackground,priorNAPLANachievement (two-yearsearlier),genderandtheconcentrationofstudentsbyfamilySES,concentrationofAboriginalstudents,schoolsizeandschooltype.The analysis measures how much each school ‘value-added’ to their student NAPLANachievementandhowwellaschoolperformscomparedtosimilarschools.
Schools participating in the low SES NP exhibit overall a positive effect on studentachievement,albeitwithvariationsforReadingandNumeracyandbetweendifferentyearsofNP implementation.TheNPeffect ismoreconsistentor strongeronNumeracy than inReading andmore consistent for the Year 9 cohort than for the Year 5 cohort. Schoolscommencingtheir lowSESNP in2011showonaveragethestrongestNPeffectamongallNPschools.
ComparedtostudentsinlowSESschoolswhodidnotparticipateintheNP,studentsinthelowSESNPschoolsperform,onaverage,3pointshigherinYear5Numeracy,3pointshigherinYear9Readingand6pointshigherinYear9Numeracy.
LowSESreformareaslinkedtobetterstudentoutcomes
ToexaminewhetherornotparticularaggregateeffectswereassociatedwitheffortplacedinparticularlowSESreformareas,atabroadlevel,datawereusedfromthe2013Teachersurvey, and the 2014 Principal survey to examine the aggregate NP effects. Teachersresponded in the 2013 Teacher Survey to a series of questions about changes in teacherpracticeandschoolmanagementasaresultofNPimplementation.Principalswereaskedinthe2014surveyfivegroupsofquestionsthatarebroadlyrelatedtothesixNPreformareas.Foreachgroupofquestions,principals responded to threeaspectsofNP implementationincluding:
• the intensity of NP implementation (i.e., the total number of NP initiativesimplemented),
• thelevelofeffectiveness(from“Notalleffective”to“Highlyeffective”),and• thetotalnumberofNPinitiativesthatwerecontinuingattheschool.
Asetofmeasureswereconstructedusingteacherresponsestoaseriesofquestionsinthesurvey, includingthe levelofpositivechanges inteacherpracticeandthe levelofpositivechanges in school management. These were correlated with the low SES NP effects onNumeracyandReading.Afurtheranalysisexaminestherelationshipbetweenthesizeoftheimpact of the low SES NP on student Reading and Numeracy achievement and principalviewsonwhatreformareascontributedmosttoimpact.
UniversityofCanberraVictoriaUniversity
iii
Theresultssuggestthatforprimaryschools(Year5)thereisapositiverelationshipbetweenNP school effect scores and principal-reported level of effectiveness for initiativesimplemented as part of Reform Area 4, providing innovative and tailored learningopportunities.Thehigherlevelofeffectivenessasperceivedbytheprincipal,themorelikelytheschoolshowsahigherNPeffect.TheresultsalsoshowforprimaryschoolsarelationshipbetweenthenumberofinitiativesimplementedfromReformArea4andNPschooleffects.It suggests that schools which placedmore effort in this reform areawasmore likely toshowahighereffectonstudentlearningoutcomes.
Individualinitiativeslinkedtobetterstudentoutcomes
NPschoolsmaybefacedwithsomecommonchallenges,suchas lowattendanceand lowachievement, however other conditions or circumstances they operate with may vary.Although government schools were required to address all six areas of the NP reform,schools could prioritise their efforts or resources on particular initiatives or activities toaddresstheiruniquechallengesorconditions.Informationprovidedbytheprincipals(inthe2014survey)enablesanalyses to identify the impact, ifany,of individualNP initiativesorstrategies.
To identifywhat initiativesorstrategiesthatmayhavedifferent influencesonthe levelofNPeffectamongNPschools,schoolsweregroupedbasedontherankingderivedfromthesizeoftheirNPeffectvalues.Threegroupswereidentified:(1)schoolswith lowNPeffectvalues, suggesting negligible or little effect on Reading and Numeracy achievement, (2)schoolswithsmallormodesteffects,and(3)schoolsatthetopendoftherankdisplayinglargereffects.
InlookingatwhatseparatedschoolsonthebasisofwhethertheirimprovementinNAPLANachievement had been ranked relatively higher or lower, the analysis revealed that highimpactschoolswerecharacterisedbythefollowingfeatures:
• They had placed greater emphasis than low impact schools on innovations inteachingpracticeandaddressingstudentneeds
• Theyhadprovidedstrongerwelfareandlearningsupportfordisadvantagedstudentsthrough providing teachers with training around individual learning needs, andstrengtheningtheuseofIndividualLearningPlans(ILPs)forstudents
• Strongeruseofevidencetoidentifystudentlearningneeds• Stronger emphasis on developing staff through providing opportunities for
professionallearninganddevelopment
ForschoolswheretheNAPLANachievementeffecthadbeen lowornegative,theanalysisrevealedthatschoolstendedtobecharacterisedbythefollowing:
UniversityofCanberraVictoriaUniversity
iv
• Some had a stronger emphasis on strategies to attract and retain teachers,particularlythroughleadershippositions,whichmayitselfreflectgreaterneedinthisareaandschoolsstrugglingtorecruitandretainqualityteachersandleaders
• Less focuson innovation in teaching, addressing studentneeds, and strengtheningschoolaccountabilityandmoreonstaffingandperformancemanagement
• Pursued strategies around teacher and leadership recruitment, retention anddevelopmentsuggestingthatstaffstabilitywasamajorissueforthem.
Thedifferences in focus and emphasis amongNP schoolsmaywell reflect the challengesthat they faceand towhich theyare responding, rather than theeffectsof thestrategiesthemselves.Forexample,schoolsthatshowlowNPimpactonstudentNAPLANresultshaveplacedgreateremphasisonattractinghighperformingteachersandimprovingthestaffingandperformancemanagement roleof theprincipal. Theymayhave issues in staffingandmanagement practices that need (and the schools recognise that need) to be addressed.High impact schools, on the other hand,may havemore stable staffing profiles, and it istheir staffing andmanagement stability thathasenabled these schools to target areasofimprovement in teachingpracticeand inaddressingstudentneedsthathasdeliveredrealtractionfromtheNPinitiatives.
UniversityofCanberraVictoriaUniversity
v
ContentsContents...............................................................................................................................................v
Listoftables........................................................................................................................................vi
Listoffigures......................................................................................................................................vii
Glossary............................................................................................................................................viii
1. Introduction....................................................................................................................................1
1.1 Previousfindings.....................................................................................................................................3
1.2 Thisreport...............................................................................................................................................4
2. Datasources....................................................................................................................................6
2.1 Limitationsofthestudy...........................................................................................................................9
3. MethodologyandDefinitions........................................................................................................11
3.1 Measuringstudentlearning..................................................................................................................12
3.2 Measuringschoolleveleffectsonstudentprogress.............................................................................13
4. AggregatelowSESNPeffectsonstudentlearning.........................................................................15
4.1 Influentialstudentbackgroundandschoolfactors...............................................................................15
4.1.1 Significantstudentlevelfactors...................................................................................................16
4.1.2 Significantschoollevelfactors.....................................................................................................18
4.1.3 NPschoolleveleffects..................................................................................................................18
4.2 Estimatingthesizeofschooleffects.....................................................................................................18
4.3 ComparingschooleffectsbetweenlowSESNPschoolsandlowSESnon-NPschools.........................21
5. Initiativeslinkedtobetterstudentoutcomes................................................................................23
5.1 KeyNPreforms......................................................................................................................................23
5.2 Overalllevelsofchangelinkedtoreformareasasperceivedbyprincipals..........................................245.3 NPeffectsassociatedwithchangesinteacherandschoolpractice.....................................................28
5.4 IndividualNPinitiativesorpracticesassociatedwithNPeffect...........................................................30
5.4.1 GroupinglowSESschoolsbasedonestimatedlevelofimpact....................................................30
5.4.2 GreatestimpactNPinitiatives......................................................................................................33
6. Casestudyschools.........................................................................................................................37
6.1ValleyPublicSchool......................................................................................................................................39
6.2SouthCoastHighSchool...............................................................................................................................416.3WesternPublicSchool..................................................................................................................................43
6.4WesternHighSchool....................................................................................................................................45
7. Conclusion.....................................................................................................................................50
References..........................................................................................................................................52
Appendix............................................................................................................................................55
UniversityofCanberraVictoriaUniversity
vi
ListoftablesTable2-1 Profiles ofmatched cohorts for student NAPLAN records 2011-2013: Year 3-5
andYear7-9,studentsinNSWgovernmentschoolsandstudentsinparticipatinglowSESNPCatholicschools 7
Table2-2 Profilesofschoolswithmatchedcohorts 8
Table4-1 Estimates of predictors of NAPLAN Reading and Numeracy achievement(standardisedscores)in2013:Year5andYear9students 17
Table5-1 TestsofSpearmanCorrelationbetweenschoolNPeffectsandtheeffectivenessofNPimplementationasperceivedbytheprincipal 27
Table5-2 TestofSpearmanCorrelationbetweenNPschooleffectandtheeffectivenessofNPimplementationasperceivedbyteachers 29
Table5-3 LowSESNPschoolsbyschooleffectcategory 31
Table5-4 OrdinallogisticregressionmodelresultsidentifyingsignificantpredictorsofsizeofNPschooleffectonNAPLANachievement 32
Table5-5 NPstrategiesassociatedwithNPeffect impactcategory(low,medium,high) inReadingandNumeracy 34
Table6-1 Profilesofcasestudyschools 38
TableA-1 Schoollevelcovariancestatistics(Intra-classCorrelationCoefficient) 55
UniversityofCanberraVictoriaUniversity
vii
ListoffiguresFigure3-1 Flowchartoutliningapproachtotriangulationofdata 11
Figure3-2 VariousmeasuresofstudentachievementsinNAPLAN(2011-2013cohorts) 13
Figure4-1 Distributions of estimated school level effects (school level residuals) by YearlevelsandDomains 20
Figure4-2 School effect on student NAPLAN scores, comparing low SES NP and low SESnon-NPschools 21
Figure5-1 DistributionofNPinitiativesimplementedinschoolsbyreformareas 26
UniversityofCanberraVictoriaUniversity
viii
Glossary
ACARA TheAustralian Curriculum, Assessment and Reporting Authority. Has oversight ofNAPLAN.
AECG AboriginalEducationConsultativeGroup
ARIA Accessibility/Remoteness Index of Australia—an index that underpins manylocationclassifications.
COAG Council of Australian Governments, an organisation consisting of the federalgovernment,thegovernmentsoftheeightstatesandterritoriesandtheAustralianLocalGovernmentAssociation.NPsareimplementedundertheauspicesofCOAG.
CIRES CentreforInternationalResearchonEducationSystems,VictoriaInstitute,VU.
DEC DepartmentofEducationandCommunities,NSW(fromApril2011).
DET DepartmentofEducationandTraining,NSW(untilApril2011).
FTE FullTimeEquivalent.HAT HighlyAccomplishedTeacher (governmentsector)orequivalent (non-government
sector). An initiative within the NP. A HAT models good teaching practice andmentorsotherteachersthroughsupervision,demonstrationandteamteaching.AHAT usually has half the teaching load of a regular classroom teacher and is amemberoftheschoolexecutive.
ICSEA
IndexofCommunitySocio-EducationalAdvantage,ascalethatrepresentslevelsofeducational advantage associated with the educational and occupationalbackgroundofparentsofstudents.Aschool'sICSEAvalueistheaverageleveloftheeducational advantage of its students. Developed by ACARA to assist with theinterpretationofNAPLANresults.
LowSESNP LowSocio-EconomicStatusSchoolCommunitiesNationalPartnership.MCEETYA TheMinisterial Council on Education, Employment, Training and Youth Affairs, (a
MinisterialCouncilofCOAG)whichwasreplacedfromJuly2009bytheMinisterialCouncilforEducation,EarlyChildhoodDevelopmentandYouthAffairs(MCEECDYA)andtheMinisterialCouncilforTertiaryEducationandEmployment(MCTEE).
NAPLAN National Assessment Program - Literacy and Numeracy. An annual nationalstandardisedliteracyandNumeracytestingprogramforstudentsYears3,5,7and9.
NP National Partnership, agreements between the Commonwealth and state andterritorygovernmentsmadeundertheauspicesofCOAGoutliningfunding.
ISP IndividualStudentPlan.PD/PL Professionaldevelopment/Professionallearning.UoC UniversityofCanberra
UniversityofCanberra VictoriaUniversity
1
1. Introduction
An ongoing interest among educational policymakers and practitioners is to identifyeffective school practices that lead to improved student learning outcomes. This isparticularlyimportantinhelpingschoolsandsystemsworkouthowbesttoinvestfundsandresources to gain improvements and maximise returns on investment. The NationalPartnerships initiatives implemented in New South Wales schools provided an idealopportunitytocontributetothisgoalbecauseofthestructuredandtargetedapproachtoreformsandtheavailabilityofdata,bothadministrativeandsurvey,withwhichtoestimateandmeasureimpactandchange.
TheNationalPartnershipAgreementswereaseriesofpolicy interventions initiatedbytheCouncilofAustralianGovernments(COAG)andimplementedacrossallstatesandterritoriesin Australia. There were three Smarter Schools National Partnerships - the Literacy andNumeracy, Improving Teacher Quality and Low Socio-Economic School CommunitiesNationalPartnerships. All threeNationalPartnershipsweredesignedto improvestudentlearning in schools. Although each National Partnership functioned independently, theywere conceived as a mutually complementary reform package. The Low SES SchoolsNational Partnership targeted schools serving disadvantaged communities and providedthemwithextra resources and targetedpolicy initiatives tohelp thembuild capacity andimprovestudentlearningoutcomes.
The Low SES NP featured six reform areas that encompassed a variety of initiativesimplementedbyschools:
ReformArea1: Incentivestoattracthigh-performingteachersandprincipals
ReformArea2: Adoption of best practice performance management and staffingarrangementsthatarticulateaclearroleforprincipals
ReformArea3: School operational arrangements that encourage innovation andflexibility
ReformArea4: Provideinnovativeandtailoredlearningopportunities
ReformArea5: Strengthenschoolaccountability
ReformArea6: External partnerships with parents, other schools, businesses andcommunitiesandtheprovisionofaccess
UniversityofCanberra VictoriaUniversity
2
Initiativesassociatedwithstaffing,managementandaccountabilityfellunder5outofthe6reform areas associatedwith the Low SES NP. Over the life of the Low SES NP, schoolsimplemented a variety of interconnecting and reinforcing strategies associated with thevariousreformareas.
ThisreportpresentsresultsofafurtherevaluationofthelowSESNPreforminitiatives.ItispartoftheevaluationthatinitiallywasdescribedastheEvaluationServicesAgreementforDETSSNP1024:EvaluationofSchoolexternalpartnerships,betweentheStateofNewSouthWales and the University of Canberra. That evaluation was revised to focus on studentperformance in low-SES schools that had participated in the National PartnershipAgreements and in particular to examine the extent to which changes in studentperformancearerelatedtoNPinitiativesorotherfactors.Therevisionisconsistentwiththeoriginal aims of the research project and builds on the findings of previous stages of theevaluation, which suggest a more thorough investigation of student performance in thecontextofNPinitiatives.
Partoftheaimofthiswork,consistentwiththeoriginalevaluationgoals,istoevaluatetheeffectivenessandlongtermimpactofschoolexternalpartnershipsontheschoolandpost-schooloutcomesofstudentsindisadvantagedschools,aswellasassesstheextenttowhichsupportingschoolexternalpartnershipsrepresentsvalueformoneyinachievingimprovedstudent outcomes. The two key research questions to be addressed in that evaluationwere:
(1)Towhatextenthaveschoolexternalpartnerships ledto improvedoutcomes inLowSESNPschoolsaftercontrollingforotheractivitiesintroducedundertheNP?
(2) What activities are associated with the greatest improvements in outcomes forstudents?
Theworkundertakenforthecurrentreportaimstoaddressthesequestionsaswellas toexaminetheimpactmorebroadlyofthedifferentreformsthatmadeupthelowSESNP.So,afurtherquestionofimportancetothisstudyis:
(3)Whichreformshavecontributedmosttoimprovedstudentoutcomes,andwhy?
These research questions are prompted by a policy interest in how student performancechangedbetween2008and2013and theextent towhichany changesare related toNPinitiatives.AkeyissueforpolicyistounderstandwhichNPinitiatives,ifany,hadapositiveimpactonstudentperformanceinlow-SESschoolcommunitiesandwhy.
UniversityofCanberra VictoriaUniversity
3
1.1 PreviousfindingsPrevious findingsassociatedwiththisstudyhave identifiedpositiverelationshipsbetweenLowSESNationalPartnershipparticipationandimprovedstudentoutcomesoverthestudyperiod.
Intheevaluationofthestaffing,managementandaccountability initiativesofthe lowSESNP, comprehensive analyseswere undertaken to evaluate the impact of the initiatives inimproving NSW student outcomes across a range ofmeasures including NAPLAN results,Year 12 attainment, HSC achievement, ATAR scores, and attendance (CIRES, 2015). Thereport identified positive effects of somemeasures at an aggregate or system level as aresultofLowSESNP.Inparticular,thepreliminaryanalysisusingNAPLANmatchedcohortdata suggested that low SES NP schools on average saw a significant impact on studentNAPLANresultsover3yearsbetween2011and2013.
The NAPLAN trend analysis employed an approximation of a multiple error componentmodel using OLS with student level fixed effects, to measure effects of Low SES NPparticipationonNAPLANscores. Estimationwasperformedonapproximatelyonemillionstudentscoresbetween2008and2013.ResultssuggestedthattheLowSESNPhashadasignificantpositiveeffectonstudentNAPLANachievement,withthelargestimprovementinwritingscores.Theanalysisalsoindicatedtheimportanceofconsideringthedurationthataprogram has been in place, with Reading, Spelling, Grammar and Numeracy scoresincreasingby1to1.5pointsperadditionalyearofparticipationintheprogram.
Anumberofextensionstotheanalysiswerealsoconsidered, includingcomparingschoolswithcommonICSEAscores,examiningheterogeneityinthetreatmenteffectandestimationonCatholicschools.Thecomparisonsampleestimatesaresimilarinsignandmagnitudetothe main results, suggesting modest positive effects on student NAPLAN scores ofparticipationinLowSESNP.
The Low SES NP overlaps with several different programs, and an analytical model wasdeveloped to consider the individual effect of each program and its duration. Programsincluded in the models, in addition to the broad Low SES NP, include the Literacy andNumeracyNP, the ImprovingTeacherQualityNP (including theNP for the teacherqualityenhanceddecisionmakingpilot,andtheNPforteacherqualityforschoolsparticipatingasa‘spoke’ofaCentreforExcellenceHub).ResultssuggestthatparticipationinthebroadLowSESNPprogramonlyhadthelargestpositiveeffectsonNAPLANscores.
Ananalysisofachievementgaincomparinggainscoresagainstthestateaverageandwithinschool variations shows there is a reduction in the gap between the primary schoolsparticipating intheLowSESNPandthestateaverage inReadinggain.Thereductionsaresmall,butconsistent.
UniversityofCanberra VictoriaUniversity
4
1.2 ThisreportPrevious resultspoint topositiveaggregateeffectsof the lowSESNP reforms.What theydon’tshowisiftheeffectisuniformorvariesacrossschools,andifsowhatmightexplainvariations in impact.While it is important to know if on average the Low SESNP helpedimprovetheeducationandlifeopportunitiesofstudentsfromlowSESbackgroundsthroughimprovementsinstudentoutcomes,itisalsoimportanttoknowiftheeffectsvaryandwhatcontributes to variation in impact. To examine this aspect, a range of analyses wereundertakenusingavailabledata toevaluate thevariations ineffectivenessof theLowSESNPinimprovingNSWstudentoutcomesusingNAPLANachievementdata.
Drawing on quantitative evidence by cross-matching administrative and survey datasources, this report provides analyses on how and to what the extent the NP effect onstudentNAPLANresultsvaryacrossschools.By linking individual studentNAPLANrecordsanddataonoveralllevelofNPrelatedpracticesandfeaturesinNPschools(asreportedbyprincipals and teachers), this report identifies the keyNP initiatives or interventions thatmaypromoteordrivestudentlearningimprovement.Italsoprovidesanalysesandinsightsonotherschool-levelfactorsthatmayassistinoptimisingprogrameffects.This report builds on the findingsof previous stagesof the lowSESNPevaluation,whichsuggest a more thorough investigation of student performance in the context of NPinitiativeswouldbeworthwhile.An important goal of the evaluation is to measure the impact of NP interventions byexamining student and school performance differences using a multi-level modelling or‘value-added’methodology1.Valueadded(VA)analysisprovidesarobustwayofestimatingpupilandschoolperformance.ItwillbeusedtoidentifywhichelementsoftheNPinitiativesmakeadifference,intermsofimprovedresultsinlow-SESNPschools.BylinkingindividualstudentNAPLANrecordsbetween2011and2013, this report isable to track the learningprogressof2cohortsofstudents(i.e.,Year3-5andYear7-9)inReadingandNumeracyoverthe3yearsfrom2011to2013.TheVAeffectofindividuallow-SESNPschoolsonNAPLANReadingandNumeracyarethenestimated.TheVAmeasuresadjustforstudentandschoolfactorsthatareknowntohaveanimpactonstudentlearningoutcomesincluding,atthestudentlevel,priorperformance,individuallow-SES student background, gender and ATSI status. The contextual characteristics of theschool—SES measures, type, location, total enrolments and enrolments by gender,AboriginalityandlanguagebackgroundsotherthanEnglish(LBOTE)—werealsoapplied.The
1Lu,LucyandRickard,Karen(2014)Value-addedmodelsforNSWgovernmentschools.TechnicalPaper.CentreforEducationStatisticsandEvaluation,NSWDepartmentofEducationandCommunities,OfficeofEducation.
UniversityofCanberra VictoriaUniversity
5
purpose of defining the VA measures is to ensure that the impact of a students’ priorachievementlevel,socio-economicstatusbackgroundandthatoftheirpeersareaccountedforwhenmeasuringstudentachievement.Thisenables identificationofwhich low-SESNPschools, if any, contributeadditionally to studentachievement relative to theaverageNPschoolovertheperiod.
Atamoredetailed level, theanalysisofVAmeasureshelpsdeterminewhich schoolsandwhichNP initiativeshavebeensuccessful in raisingstudentperformance forstudentsandhowlargetheimpacthasbeen.
This focus ispromptedbyapolicy interest inhowstudentperformancechangedbetween2008and2013and theextent towhichanychangeswere related toNP initiatives.AkeyissueforpolicyistounderstandwhichNPinitiatives,ifany,hadapositiveimpactonstudentperformanceinlow-SESschoolcommunitiesandwhy.
Itshouldbenotedthatprivatenon-Catholicschoolswerenotincludedintheanalyses.
UniversityofCanberra VictoriaUniversity
6
2. Datasources
Thisreportdrawsonawidevarietyofdatasources,including:
• studentlevelNAPLANdatain2011and2013;• NAPLANschoolprofiledatain2011sourcedfromACARA• schoolleveldatafromthe2014lowSESNPPrincipalSurvey,and• datafromthe2013lowSESNPTeacherSurvey.
Amajor goal of theNP initiativeswas to improve theeducationand lifeopportunitiesofstudents from low SES background through targeting teacher quality, better use of theassessment data, strengthened school leadership and strengthened partnerships withcommunity agencies and members including parents. One major question to be asked,therefore, iswhetherthereformsimplementedinNPschoolshave ledto improvement instudentlearningoutcomes.
NAPLAN is the only available source that provides standardised and calibrated tests ofstudent learning achievements administrated at regular intervals. It provides an annualassessmentforallstudentsinYears3,5,7and9invariousliteracydomainsandNumeracy.AlthoughNAPLANisnotadirectmeasureofcurriculumcontentcoveredinschoolprograms,theydoprovideanassessmentofessentialskillsthatwemightexpectchildrentopossesstoprogressthroughschoolandlife.
BylinkingindividualstudentNAPLANrecordsbetween2011and2013,weareabletotracktheNAPLANachievementof2cohortsofstudents:(1)Year3studentsin2011and(2)Year7studentsin2011.AchievementwasmeasuredinReadingandNumeracyin2013whenthetwocohortswere inYear5andYear9, respectively.Over52,000studentrecords in2011were successfully matched to the 2013 records for both cohorts. This accounts forrespectivelyaround66per centofYear3 studentsand70per centofYear7 students in2011.TheprofilesofthetwocohortsarepresentedinTable2.1.
TheresultsinTable2-1showthataround4percentofstudentsdonothaveachievementresultsduetoabsence,withdrawalorexemption.Exemptionsmaybegrantedbyschoolsforstudentswith a disability orwith a language background other than Englishwho arrivedfromoverseasandhavebeenattendingschoolforlessthanayearbeforethetest.Dataforstudentswithoutassessmentscoresaretreatedasmissingvaluesandareexcluded intheanalyses.
The results also show the loss of students due to the matching and linking of studentrecordsbetweenYears3to5andYears7to9.Studentsmayhavemissingdataatonepointrather than both because they changed schools orwere absent or exempt at one of the
UniversityofCanberra VictoriaUniversity
7
measurementpoints.ToensurethatthechangesinresultsorNPeffectsarelinkedtotherole of programs in individual schools, only students who stayed in the same schoolbetweenthetwoNAPLANassessmentperiodswereincludedintheanalyses.Differencesinlevelsofabsenceandexemptionacrossschools,anddifferencesinproportionsofstudentsstayinginthesameschoolacrossthetwotimepoints,maybeaninfluenceontheresults.However,onethingtonotefromTable2-1isthatatanaggregatelevelthematchedsampleis sufficiently similar to the total sample to give us confidence that the sample isrepresentativeofthepopulationofstudents.Theoverall ratesofmatchingfor indigenousstudents,forLBOTEstudents,formalesandfemales,andforthosenotassessedbecauseofexemptionorabsenceare roughly similar. There stillmaybedifferencesata school levelthatneedtobeconsidered.Futureworkwillbeundertakentoexaminewhatimpact,ifany,thishasonestimatedvalue-addedscores.
Table2-1 ProfilesofmatchedcohortsforstudentNAPLANrecords2011-2013:Year3-5
and Year 7-9, students in NSW government schools and students inparticipatinglowSESNPCatholicschools
NAPLANmatchedcohort NAPLANCohortin2011
Years3-5 Year7-9 Year3 Year7 Number
Sex Male 26,716 27,153 40,238 38,529Female 25,439 25,532 38,209 36,395
Indigenousstatus
Non-Indigenous 49,306 49,694 74,363 70,788Indigenous 2,849 2,991 4,084 4,136
LBOTEStatusNon-LBOTE 36,807 36,444 55,742 53,821LBOTE 15,135 15,595 22,265 20,379Unknown 213 646 440 724
NotassessedReading 1,996 2,047 2,884 2,782Numeracy 2,109 2,360 3,039 3,172
Total 52,155 52,685 78,447 74,924 Percentage
Sex Male 66.4 70.5 100 100Female 66.6 70.2 100 100
Indigenousstatus
Non-Indigenous 66.3 70.2 100 100Indigenous 69.8 72.3 100 100
LBOTEStatusNon-LBOTE 66.0 67.7 100 100LBOTE 68.0 76.5 100 100Unknown 48.4 89.2 100 100
NotassessedReading 69.2 73.6 100 100Numeracy 69.4 74.4 100 100
Total 66.5 70.3 100 100
UniversityofCanberra VictoriaUniversity
8
Table2-2 Profilesofschoolswithmatchedcohorts Primary Secondary Combined Total* Numberofschoolswithmatchedcohorts
SchoolSector
Catholic 72 58 9 139Government 1,524 370 67 1,961Total 1,596 428 79 2,112
SchoolLocation
Metropolitan 956 290 6 1,252Provincial 616 133 54 803Remote 20 4 12 36VeryRemote 4 1 4 9Total 1,596 428 79 2,112
NPschools:YearofNPparticipation
2009 70 22 25 1172010 105 33 8 1462011 121 23 1 1452012 50 16 13 79Total 346 94 47 487
Surveyparticipation
TeacherSurvey(2013) 217
PrincipalSurvey(2014) 188 Percentageofschoolswithmatchedcohorts
SchoolSector
Catholic 17% 54% 64% 26%Government 94% 93% 99% 94%Total 78% 85% 96% 80%
SchoolLocation
Metropolitan 78% 76% 4% 70%Provincial 74% 80% 46% 72%Remote 57% 100% 100% 71%VeryRemote 40% 50% 67% 50%Total 76% 77% 26% 71%
NPschools:YearofNPparticipation
2009 80% 96% 100% 87%2010 79% 89% 53% 79%2011 81% 82% 25% 80%2012 76% 70% 81% 75%Total 79% 85% 78% 80%
Surveyparticipation
TeacherSurvey(2013) 75%PrincipalSurvey(2014) 70%
*Totalsincludeschoolswherethetypeofschoolisunknown.NOTES:Specialschoolsareexcluded.
Other important sources of data have been developed over the course of the evaluationthrough a series of school-based surveys. Surveys of school principals and teachers haveassisted in obtaining detailed information about the implementation and effectiveness oflow SES NP initiatives or interventions. In this report, data on the low SES NP
UniversityofCanberra VictoriaUniversity
9
implementationandeffectivenesscollectedthroughthe2014PrincipalSurveyandthe2013Teacher Survey have also been linked to their student NAPLAN records using schoolidentifiers.
Data on the low SES NP implementation and effectiveness collected through the 2014Principal Survey and the 2013 Teacher Survey have been linked to the student NAPLANrecordsusingschoolidentifiers.
ProfilesofschoolswithmatchingcohortsareprovidedinTable2-2.Thenumbersofschoolsmatched to the NAPLAN cohort data are provided in the first panel of the table. Eachpercentagereportedinthesecondpanelofthetablereportswhatthenumberofmatchedschoolsisasapercentageofallschoolsforthatcategory. Forexample,1524governmentprimaryschoolswerematchedtotheNAPLANcohortdataandthisrepresented94percentofallpossiblegovernmentprimaryschools.
ThemajorityoflowSESNPschools—487outof607or80percent—havebeenmatchedtotheNAPLANcohortdataforthevariousanalysesinthisreport.
About75percentofschoolsthatparticipatedinthe2013TeacherSurveyand70percentofschoolsthatparticipatedinthe2014PrincipalSurveywerematchedtostudentNAPLANcohort data. While not covering all schools, the matching provides sufficient samples ofstudentsandschoolstoprovideconfidenceintheanalyticalresultspresentedinthereport.
2.1 Limitationsofthestudy
Datamatchingacrossmultiplesources,andusingdatafromsamplesurveys (teachersandprincipals),appliessomelimitationsthatneedtobeconsidered inusingtheresults inthiswork.
NAPLAN datawere available for government schools and for a small number of Catholicschools(thoseparticipatinginthelowSESNP).Duetothelackofacommonschoollinkagekey,someschoolswereexcludedfromthematcheddataset.ThismostlyaffectedCatholicschools.
Improvement inNAPLANresultsmaynotbe thebestor themost idealwayofmeasuringNP-related effects. Changes may occur in student behaviour, engagement, skills andoutlooks and in ways which could have lasting and beneficial effects in a lifelongperspective.Studentlearningandskillsmaymarkedlyimproveinvariousclassroomsubjectareasbutnotnecessarily inNAPLANtestdomains,oronothermeasures.Byvirtueof theprogram, theNP initiativesmayhave contributed to lowSES schoolsbecoming farbetterplacesinnurturingstudentsandaddressingtheirneeds,andpromotingabetterandmoreengaged and active community as a result of participation without it leading directly to
UniversityofCanberra VictoriaUniversity
10
improved studentoutcomes. Thedataused for this studydonot allowus toexplore thisissue.
Data fromthe2013TeacherSurveyandthe2014PrincipalSurveyprovidevery importantsources of information and insights about the implementation and effectiveness of NPstrategies and initiatives in schools. Although a sufficient number of schools haveparticipated inboth surveys, theoverall sample size forboth surveys are relatively small.The small sample sizemay impacton the statisticalpowerof someanalysespresented inthe report. To overcome this problem to some degree, results using data from the twosurveys have been cross validated and compared using different methods. Only the keythemesorpatternsthatareconsistentlyshownacrossallanalyseshavebeenreported.
Administrative data with a wider coverage of schools andmore detailed information ontheir NP implementation have now become available, and will be examined in the nextphaseofwork,butwewerenotabletoincorporatethedataintothisreport.
UniversityofCanberra VictoriaUniversity
11
3. MethodologyandDefinitions
Todeveloprich,robustandcomprehensiveevidenceinevaluatingthelowSESNPeffectonNAPLAN outcomes across NP schools, this report draws on data and information from awidevarietyof sources.Thedatavaryconsiderablyas to referenceperiods,definitionsofmeasures, and types or scale of data. Because of this, for the analyses in this report atriangulation approach has been adopted in recognition of the various limitations of thedataandtotestthevalidityoffindingsfromdifferentsources.This issetoutinschematicform in Figure 3.1. Using this approach, we are able to draw on evidence from bothqualitativeandquantitativedataandexaminetheconsistencyoffindingsbasedondifferentdatasources.
Figure3-1 Flowchartoutliningapproachtotriangulationofdata
Es^mateschoolleveleffecton
studentNAPLANachievement
Mulo-levelmodelling
2011-2013NAPLANmatchedcohortdataat
bothstudentandschoollevels
Testrela^onshipbetweenNPeffectandkey
schoola`ributes
Spearmancorrelaoonanalysis
NAPLANschooleffectand2013TeacherSurvey&2014
PrincipalSurvey
Iden^fykeyNPStategiesrelated
tohigh/lowimpactschools
OrdinalLogisocregression
NAPLANschooleffectand2014
PrincipalSurvey
CaseStudiesofNPschools
CIRESReports(2012&2013)
Method
Stages
Data
UniversityofCanberra VictoriaUniversity
12
Various statistical methods have been applied in analyses at different stages inconsiderationofthetypeorlimitationofeachdataset,including:
• multi-levelmodellingtoderivemeasuresofschooleffectsonNAPLANachievement,• Spearman's rank-order correlation test to identify the relationship between the
estimated level of NP effect for a school and the levels of effectiveness of NPinitiatives perceived by teachers or the principal. The Spearman's rank-ordercorrelationcoefficientsarecommonlyusedtomeasurethestrengthofassociationbetweentwoordinalvariables.
• Ordinal logistic regression models to identify the key NP initiatives or strategies
associatedwithlowandhighimpactschools.Sections 3.1 and 3.2 below describe how the key measures required for the multi-levelmodelshavebeendefinedandconstructedinthereport.
3.1 MeasuringstudentlearningIndividual studentperformance inNAPLAN is assessedonanational assessment scale foreachtest.Thesinglescaleallowsstudents,teachersandparentstomonitorprogressacrossthe years and compare results to those in previous years as students advance throughschool.Thecommonscaleforeachdomainisalsodividedinto10achievementbands(Band1–Band10)withsixbandsreported foreachyear level.Anationalminimumbenchmarkstandardissetbasedonbandscoreforeachyearlevel, i.e.,Band2forYear3,Band4forYear5,Band5forYear7andBand6forYear9.
While the scale for each of the five domains appears to be similar the results cannot bedirectly comparedacrossdomains. For this report,only resultsof studentperformance intwo key domains, Reading and Numeracy, have been used for the various analyses. Thisreport does not cover the full set of literacy and numeracy skills of students whoparticipatedinNAPLAN.
Figure3-2showsthatstudentsinNPschoolsperformconsistentlylowerthanthoseinnon-NP schools in both Reading and Numeracy, across different year-levels and across broadstagesof schooling.Students inNPschoolsperformonaverageatabout40pointsbelowstudentsinnon-NPschools.StudentsinNPschoolsarealsotwiceaslikelytoperformatorbelowthenationalminimumstandard(NMS)inbothReadingandNumeracy,andacrossallyearlevels.
UniversityofCanberra VictoriaUniversity
13
Figure3-2 VariousmeasuresofstudentachievementsinNAPLAN(2011-2013cohorts)
StudentNAPLANachievement(MeanScore)byschooltype
StudentNAPLANachievementbyNationalMinimumBenchmarkStandard
3.2 Measuringschoolleveleffectsonstudentprogress
Childrenlearnanddevelopthroughtheirinteractionswithdifferentenvironments,includingtheir families, peers, schools and a wider community and society. Student learningoutcomes in schools often reflect a compounded effect from both individual and schoolfactors.
426
509545
586
408
497
552
597
378
471506
553
372
458
512
557
300
350
400
450
500
550
600
650
AtYR3 AtYR5 AtYR7 AtYR9 AtYR3 AtYR5 AtYR7 AtYR9
Reading Numeracy
Non-NPSchool NPSchool
13% 10% 19% 21%28% 23% 37% 39% 13% 19% 19% 26% 28% 36% 37% 42%
AtYR3
AtYR5
AtYR7
AtYR9
AtYR3
AtYR5
AtYR7
AtYR9
AtYR3
AtYR5
AtYR7
AtYR9
AtYR3
AtYR5
AtYR7
AtYR9
Non-NPschool NPschool Non-NPschool NPschool
Reading Numeracy
AtorbelowNMS AboveNMS
UniversityofCanberra VictoriaUniversity
14
School level effects (or between school variability)measure the extent to which studentoutcomes differ across schools, after taking account of student individual factors. Forexample, students from a particular background attending one school might displayacademic outcomes that are stronger or weaker compared to students from the samebackgrounds attending other schools. Within school effects (or within school variability)measure the extent to which student outcomes differ within the same school based onstudentindividualcircumstances/attributes,suchasfamilybackgroundandmotivation.
Mixedeffectsormulti-levelmodellingmethodsprovidea robust approach for addressingchallenges associatedwithhierarchical data, suchasNAPLANdata for individual studentsnestedwithinschools.Usingamulti-levelmodellingapproach,weareabletopartitionandestimatethelevelsofeffectinstudentoutcomeswithinaschoolandbetweenschools.
To estimate to what extent NP implementation in a school has contributed to theimprovement of student learning outcomes, school effects were calculated for studentNAPLAN scores in Reading andNumeracy separately. The effectswere calculated using amultilevellinearmodel(LM)forcontinuousoutcomevariables.
In termsofequations, a simple linearmixedmodel (withone student levelpredictor andoneschoollevelpredictor)canbeexpressedas:
𝑦!" = 𝐵!! + 𝐵!!𝑥!" + 𝜀!"
𝐵!! = 𝛼!! + 𝛼!"𝑧! + 𝜇!
𝜀!"~ 0,𝜎!
𝜇!~ 0,𝜎!!!
Where Y is a continuous variable for student i in school j; β0j is the overallmean acrossschools,Xijrepresentsastudentlevelpredictorincludedinthemodel;Zjrepresentsaschoollevelpredictorincludedinthemodel;theschoollevelresidualµjisdefinedastheeffectofschooljontheirstudentoutcomes;andƐijistherandomerrorassociatedwithstudentiinschoolj.
UniversityofCanberra VictoriaUniversity
15
4. AggregatelowSESNPeffectsonstudentlearning
The lowSESNPeffect foreachschoolcanbeestimatedbymodellingachievementscoresaftertakingaccountofboththepriorachievementofstudentsandthefactorsknownfromresearch and analysis to make the biggest differences to the variations in outcomesbetween schools. Using multilevel modelling to control for student and school levelinfluences on achievement, and including an indicator for low SES NP, it is possible tomeasurethe independentcontributionof lowSESNPschoolstostudentachievementandtomeasurehowwellaschoolperformscomparedtosimilarschools.
This section examines the levels of impact of the low SES NP initiatives on NAPLANachievementthroughthefollowingsteps:
1. identifyingthestudentandschoolfactorsthatinfluenceNAPLANresults,2. modelling school effects on NAPLAN achievement, controlling for the influential
factors,3. examiningthesizeanddistributionoftheschooleffects,and4. identify the NP effect by comparing the size of the school effects of the low SES
schoolsthatparticipatedinNPcomparedtothosenotparticipatinginNP.
4.1 Influentialstudentbackgroundandschoolfactors
To identifykey influencesonNAPLANachievement, two typesofmulti-levelmodelswereconducted. The first, a multilevel linear model for continuous variables, was based onReadingandNumeracyachievementscoresinYear5andYear9andincludedthefollowingstudentbackgroundcontrols:
• studentpriorperformance(measuredbyNAPLANscoresinYear3orYear7)
• student SES (basedona scale scorederivedusinganequallyweighted indexofparentaleducationandoccupation)
• ATSIstatus
• gender,and
• whetherthestudentwasfromalanguagebackgroundotherthanEnglish(LBOTE)ornot.
Thefollowingschool-levelpredictorswereusedfortheanalysis:
• schoolLocation(ARIAareas)
• schoolsector
UniversityofCanberra VictoriaUniversity
16
• schoolSES(medianofstudentfamilySESindexwithintheschool)
• proportionofLBOTEstudents
• proportionofAboriginalstudents
• proportionofboys
• sizeofschool(enrolments)
• selectiveentryschoolornot(secondaryschoolsonly)
The model also included a flag for low SES NP schools by participation year to assesswhether there were independent effects for participation or not, and whether thiscontributedtoexplainingvariationinNAPLANReadingandNumeracyachievement.
Regressiondiagnosticswereundertakentoexaminethevalidityofthemodel.ThisincludedplottingofresidualsandrunningaKolmogorov-Smirnovtestfornormality.Foreachofthemodels in this section, the results showed that conditional residuals followed a normaldistribution and the results from the Kolmogorov-Smirnov tests for normality were notsignificant,suggestingnormalityofresiduals.
Table4.1showstheresultsofthelinearmixedmodels.ItreportstheimpactofarangeofstudentandschoolvariablesonstudentNAPLANscoresinReadingandNumeracyin2013.Resultsfromthetwomodelsaregenerallyconsistent.
4.1.1 Significantstudentlevelfactors
Previousstudentachievement,measuredinYear3in2011forYear5studentsandinYear7in 2011 for Year 9 students, appears to have the strongest predictive power in NAPLANoutcomesin2013forbothYear5andYear9cohorts.However,beingfromalowSESfamily,and fromanATSI backgroundexert independenteffectsonachievement inbothReadingandNumeracyforbothYear5andYear9cohorts.BeingfromalanguagebackgroundotherthanEnglishisassociatedwithgainsinNumeracyatbothyearlevels,andinReadingatYear9, independently of other factors. Being male is associated with a reduced achievementscoreinReadinginYear9andahigherscore,allelseequal,inNumeracyinbothYear5andYear9.
LBOTE appears to increase in effect in Reading fromprimary school to secondary school.Thismaysuggestthatalthoughbeingfromanon-Englishlanguagebackgroundisassociatedwith little advantage in Reading in the earlier years of schooling, LBOTE students inaggregatetendtoexcelinsecondaryschool.ThisisnotthecaseinnumeracywherethereisalreadyaclearadvantageinYear5.
There may be limitations or deficiencies in using LBOTE as a measure of educationaldisadvantage. LBOTE students cover potentially very diverse groups of students. Analysis
UniversityofCanberra VictoriaUniversity
17
undertakenbytheNewSouthWalesDepartmentofEducationandCommunities(NSWDEC,2011)showthat inNewSouthWales,aftercontrollingfortheeffectofstudents’parentalbackground and school-level effects, LBOTE students with limited English languageproficiencyexperiencetwicethelevelofdisadvantageasthosewhoarerefugeesorwhoareIndigenous.However,byfarthemostdisadvantagedgrouparerefugeestudentswhohavelimitedEnglish languageproficiencyandhavebeen inanAustralian school formore thanoneyear.
Table4-1 Estimates of predictors of NAPLAN Reading and Numeracy achievement(standardisedscores)in2013:Year5andYear9students
Year5Score Year9Score Reading Numeracy Reading Numeracy Intercept -0.333 -0.492 (***) -0.046 0.55 (***) StudentFactor
Priorperformance(Score) 0.711 (***) 0.688 (***) 0.784 (***) 0.81 (***)FamilySES 0.005 (***) 0.004 (***) 0.003 (***) 0.00 (***)
Aboriginalstatus -0.053 (***) -0.068 (***) -0.090 (***) -0.05 (***)LBOTEstatus 0.007 0.134 (***) 0.036 (***) 0.09 (***)
Gender(Male) -0.002 0.076 (***) -0.056 (***) 0.03 (***) Schoollevelfactor
SchoolSES 0.009 (***) 0.008 (***) 0.006 (***) 0.00 (***)Aboriginalconcentration(%) -0.006 (**) -0.011 (***) -0.015 (***) -0.02 (***)
Schoolsize(enrolments) -0.020 (***) -0.001 -0.002 -0.02 (**)CatholicvsGov. -0.016 -0.096 (***) 0.012 0.01
Non-selectivevsFullyselective -0.221 -0.75
PartiallyselectivevsFullyselective -0.160 -0.70
2009NPschoolvsNon-NPschool -0.043 (**) 0.026 0.017 0.08 (***)2010NPschoolvsNon-NPschool 0.009 -0.005 0.029 (*) 0.04 (**)2011NPschoolvsNon-NPschool 0.017 0.055 (***) 0.081 (***) 0.08 (***)2012NPschoolvsNon-NPschool -0.038 -0.035 -0.025 -0.02
***Indicatesresultsarestatisticallysignificantat1%level;**at5%leveland*at10%level;Allcontinuousvariableshavebeenstandardisedforcomparisonofcoefficients.
Theresults inTable4-1alsosuggestthatthesexofstudentsappearstohaveanoppositeeffect for Reading compared to Numeracy, with male students progressing better inNumeracyandfemalestudentsdoingbetterinReading.
AnotherinterestingcontrastisthattheeffectforpriorachievementwhichforNumeracyinYear9isstrongerthanforReading,whereastheoppositeistrueinprimaryschoolatYear5wheretheeffectofpriorachievementisstrongerforReadingthanNumeracy.
UniversityofCanberra VictoriaUniversity
18
4.1.2 Significantschoollevelfactors
After controlling for student background characteristics, school-level attributes contributeindependently toNAPLANachievement scores (Table 4-1).A small, but significant part ofthe variation in student progress is attributed to school environmental factors, such asschool level SES, and concentration of ATSI students. For achievement scores in bothReadingandNumeracy,schoolswithahigherSESstudentintakeandalowerconcentrationofATSIstudentsgainmorepositiveresults,allelseequal.
School size (measured by school full time enrolments in 2011) appears to have a smallnegativeeffectonstudentprogressinReadingforYear5andNumeracyforYear9.
4.1.3 NPschoolleveleffects
SchoolsparticipatinginthelowSESNPexhibitoverallapositiveeffectonstudentprogress,albeit with variations in Reading and Numeracy and between different years of NPimplementation.
SchoolscommencingthelowSESNPin2011showonaveragethestrongestNPeffectwhileschools participating from 2012 show the least effect. The patterns associatedwith timespentintheprogrammayreflectseveralfactors,includinga‘lag’effectand‘halo’effectforthe2009NPschools.TheremaynotbesufficienttimefortheNPinitiativetotakeeffectonthe2011-2013cohortsinschoolsthatjoinedlowSESNPin2012.
TheNPeffectappearstobemoreconsistentorstrongeronNumeracythaninReadingandmore consistent for the Year 9 cohort than for the Year 5 cohort. This warrants furtherinvestigations into the difference (if any) in school practices that contribute to thesedisparities,e.g.,whether it isdue to teachers/schoolsdevotingmore timeor resources inNumeracy related activities in secondary schools compared to primary schools. Someresearchersbelievethatmathachievementismoredependenton“withinschoolactivities”andrespondsmorequicklytocurriculumchange.PreviousstudiesbothwithinAustraliaandoverseas have found that Reading intervention programs have more difficulty producingdemonstrablegainsthatmathprograms(Loveless2013;Helal2014).
4.2 EstimatingthesizeofschooleffectsIn order to produce school effect measures that are comparable across NP and non-NPschools,school level residuals foreachof the2,112schools forwhichdatawereavailablewere derived using the multilevel models. The models were re-estimated including onlyvariablesforbothstudentandschoollevelfactorsthatwereshowntoexhibitastatisticallysignificanteffectonstudentperformance.NPidentifyingvariableswereexcludedfromthe
UniversityofCanberra VictoriaUniversity
19
modelestimationinordertoidentifyandestimatetheeffectofNPimplementationamongNPschools(seeTable4-2foralistofincludedvariables).Table4-2 Studentandschoollevelfactorsusedforadjustingschoolleveleffect
Linearmixedmodel
Schooleffectmeasure
StudentNAPLANscoreinReadingandNumeracy
Studentlevelvariablesandvariabletype
Priorperformance(Continuous)FamilySES(Continuous)
Aboriginalstatus(Binary)LBOTEstatus(Binary)
Gender(Male)(Binary)
Schoollevelvariablesandvariabletype
SchoolSES(Continuous)ATSIconcentration(Continuous)
Schoolsize(Continuous)
Selectiveschool(Binary,Year7-9cohortonly)
Regression results fromtheNPeffectsmodel showthat therearesignificantvariationsatthe school level in studentNAPLAN achievements regardless of year levels and domains,althoughtheyaresignificantlyreducedafteradjustingforstudentandschoollevelfactors.Theestimated school level variances and Intra-classCorrelationCoefficient (ICC) statisticsareprovidedinTableA1intheAppendix.
Theestimatedschooleffectforeachschooltakesintoaccountanumberofkeystudentandschoollevelfactorsthatinfluencestudentachievement,suchasstudentpriorachievement,the concentration of students by family SES, concentration of Aboriginal students, schoolsize and school type. Itmeasures howmuch each school ‘value-added’ to their studentNAPLANachievementandhowwellaschoolperformscomparedtosimilarschools.
Theschoollevelresidualsderivedoriginallyfromthemodelsarenormallydistributed(withameanofzeroandconstantvariance/standarddeviation).Theyareessentiallyarelativeorrankingmeasureonschooleffect.Notallschoolsexperiencedordisplayedthesamelevelofeffects.Themajorityofschoolsshowanaverageleveleffect,withsomeschoolsshowinglargerthanaverageeffectwhilesomeschoolsshowasmallerthanaverageeffect.
Figure4-1showsthedistributionsof school level residualsderived fromthemixedmodelmeasuringtheeffectsonstudentNAPLANaveragescorewithineachschool.
UniversityofCanberra VictoriaUniversity
20
Figure4-1 Distributionsofestimatedschool leveleffects(school levelresiduals)byYearlevelsandDomains
SchooleffectonstudentNAPLANscores
Year5Reading
Year5Numeracy
Year9Reading
Year9Numeracy
UniversityofCanberra VictoriaUniversity
21
4.3 ComparingschooleffectsbetweenlowSESNPschoolsandlowSESnon-NPschools
The regression analyses in Section 4.1 show that schools participating in the low SES NPexhibitoverall apositiveeffecton studentprogress, albeitwithvariations inReadingandNumeracyandbetweendifferentyearsofNP implementation. SchoolscommencingtheirlowSESNPin2011showonaveragethestrongestNPeffectamongallNPschools.
In this section,bycomparing theestimatedschooleffectsamong lowSESNPschoolsandlowSESnon-NPschools,weareabletofurtheridentifyandquantifythesizeofNPeffectson student NAPLAN achievement. The low SES schools are defined as schools that arerankedatthelower50thpercentilebasedontheirschoolSESindex(measuredasmedianofstudent family SES index within the school). To make the school residual results moremeaningful, the standardised school residuals were transformed back to the scale ofNAPLANscore(Figure4-2).
Figure4-2 SchooleffectonstudentNAPLANscores, comparing lowSESNPand lowSESnon-NPschools
low SES NP 2011 low SES Non-NP schools
Year5Reading
Year5Numeracy
Year9Reading
Year9Numeracy
UniversityofCanberra VictoriaUniversity
22
Figure4-2 compares thedistributionsof schooleffectson studentNAPLANscoresamonglowSESschoolsbyNPparticipationstatus.ResultsshowapositivegaporadvantageforlowSESNPschoolscomparedtothe lowSESnon-NPschoolsaftercontrollingfortherangeofother influential student and school influences. The advantage is more consistent orstrongeronNumeracythanReadingandmoreconsistentfortheYear9cohortthanforYear5. This is consistent with the regression results presented previously in section 4.1. Thenarrow spread of school effect sizes in Reading for both Year 5 and Year 9 cohortsmayreflect that Reading intervention programs have more difficulty producing demonstrablegains thanmathprograms, or they aremoredifficult tomeasure inNAPLAN instruments(seesection4.1formoredetaileddiscussion).
ResultsforeffectsonstudentNAPLANscores(Figure4-2)showthatstudentsinthelowSESNPschoolsperformonaverage3pointshigherinYear5Numeracy,3pointshigherinYear9Readingand6pointshigherinYear9Numeracy,comparedtothesamecohortinlowSESschoolsthatdidnotparticipateinNP.
NotallNP schoolsexperiencedordisplayed the same levelofeffectson student learningoutcomes after introducing the low SES NP reforms. Some NP schools show little or noeffectatall,despiteteachersandprincipals intheirsurveyresponsesreportingchangesinpracticesandsomeevenexpressingaview that the reformshadhadamarked impactattheirschool(CIRES,2015).OtherschoolsdisplaymarkedeffectswithstrongimprovementinNAPLANachievement.ApartfromtheschoollevelfactorsincludedinthelowSESNPeffectestimation models, there are other school level factors that may have an influence onstudent NAPLAN achievement, such as strategic and operational differences in NPimplementation and varying conditions or circumstances in which schools are operated.Section5willfocusonvariationsofschooleffectsamongallNPschoolsandthevariationsintheirNPinitiativesandpracticesthatmayhaveinfluencedtheirschooloutcomes.
UniversityofCanberra VictoriaUniversity
23
5. Initiativeslinkedtobetterstudentoutcomes
This chapter presents the results of analyses attempting to identify the practices andfeaturesofNPschoolsthatcontributedtothedifferentialimpactofNPreformsonstudentoutcomes.Whatarethereforminitiativesthathavecontributedtosomeschoolsrecordingstrong impact and improvement, and what characterises schools that show no or littleimpact?
The chapter begins looking at the broadNP school reformareas andwhether a focus byschoolsonparticularreformarealedtogreatergains.Toexaminethis,datafromprincipalsontheimpactofreformsareusedinconjunctionwithNAPLANresultstoassesstheimpactofbroadreformareas.Wecommencewithananalysisbasedontheinformationreportedbytheprincipalsonwhatinitiativesandschoolpracticeswereimplementedattheirschoolsandwhichreformstheyfelthadthegreatestimpact.
Following this, attention turns to examine the information provided by teachers on thepositivechangesthatoccurredintheirschoolsafterNPimplementationlinkedtochangesinteacherpracticesandchangesinschoolmanagement.
Thechapterconcludesbylookinginafarmoredetailedwayatindividualinitiativeswithinreform areas to identify individual initiatives and practices that accounted for strongerschooleffects.Analyseshavebeenundertakentoidentifywhetherandhowtheseinitiativesorschoolpracticesmayhavecontributedtotheimprovementinstudentlearningoutcomes.
The school effectsmeasure used in this section is based on the results of themultilevellinearmodeloutlinedinsection4.
5.1 KeyNPreforms
To improve student learning outcomes, the Low SES NP sought to build capacity inparticipatingschools.TheLowSESNPfeaturedsixreformareasthatencompassedavarietyofinitiativesimplementedbyschools:
ReformArea1: Incentivestoattracthigh-performingteachersandprincipals
ReformArea2: Adoption of best practice performance management and staffingarrangementsthatarticulateaclearroleforprincipals
ReformArea3: School operational arrangements that encourage innovation andflexibility
UniversityofCanberra VictoriaUniversity
24
ReformArea4: Provideinnovativeandtailoredlearningopportunities
ReformArea5: Strengthenschoolaccountability
ReformArea6: External partnerships with parents, other schools, businesses andcommunitiesandtheprovisionofaccess.
Allschoolswererequiredtoimplementactivitiesfromoneormoreofthesixreformareasover the duration of their four-year participation in the Partnership.Government schoolswere required to address all areas in each year of their participation. Schools could alsochoose to introduce strategies to address activities from the two other Smarter SchoolsNational Partnerships—Literacy and Numeracy and Improving Teacher Quality—wherethese also addressed a specific reform area. Schools were able to choose strategiesassociatedwithotherreformareasaspartoftheirschoolconsultationprocess.
5.2 Overall levels of change linked to reform areas as perceived byprincipals
InSeptember2014,principalsfrom267NPschoolsrespondedtoasurveyaddressingtheirexperiencesofandreflectionsontheimpactoftheLowSESNPreforms.ThissurveysoughtperceptionsfromtheprincipalofthelowSESNPprogramoverall,togetherwithinformationontheirownschool’sprocessesand initiatives.ComparedtoanearlierPrincipalSurvey in2012, principals in this survey were able to draw on a more sustained experience andunderstanding of the reform areas, and to project a picture of the sustainability of theirschoolinitiatives.
Consistentwiththeearliersurvey,principalswereverypositiveabouttheeffectivenessofmostof theNP initiatives, including indeveloping their existing staff capacity, developingoperationalarrangementsthatencouragedinnovationandflexibilityinstaffing,andbuildingteacherandstudentaccesstomoreinnovativeandtailoredlearningopportunities.
At the same time, principalswere least likely to express ‘strong’ support for propositionsthat they had been able to either attract or retain quality staff. In their open-endedresponsesanumberofprincipalsclarifiedsomeofthecomplexitiesassociatedwiththeLowSESNPprogramtoassistinunderstandingthispicture.Theymadereference,forexample,to the enhanced mobility of quality teachers whose development had been supportedthrough Low SES NP initiatives – such teachers were in demand on leadership teamselsewhereandforatleastsomeschoolswerethereforedifficulttoholdonto.
In relation to student learning outcomes, principals seemed to bemore reserved in theirsenseofimpact.Almostallprincipals(97percent)feltthattheprogramhadallowedtheir
UniversityofCanberra VictoriaUniversity
25
studentstoaccess‘moreinnovativeandtailoredlearningopportunities’andacomparableproportion agreed that as a result of their participation in the program ‘student learningoutcomes improved’. Only a third, however, ‘strongly agreed’ that the program hadachievedimpactonstudentoutcomes(CIRES2015).
The interest in thecurrentanalysis is toexaminetherelationshipbetweenthesizeof theimpactofthelowSESNPasmeasuredbyschooleffects(residuals)relatedtoReadingandNumeracy achievement and principal views on what reform areas contributed most toimpact.Principalswereaskedinthe2014surveyfivegroupsofquestionsthatarebroadlyrelated to the six NP reform areas. For each group of questions, principals responded tothreeaspectsofNPimplementationincluding:
• the intensity of NP implementation (i.e., the total number of NP initiativesimplemented),
• thelevelofeffectiveness(from“Notatalleffective”to“Highlyeffective”),and• thetotalnumberofNPinitiativesthatwerecontinuingattheschool.
ThreemeasureshavebeenconstructedforanalysesoftheseaspectsofNPimplementationinaschool.The firstandthirdmeasuresaredirectaggregationmeasuresbycounting thetotalnumberofvalid responseswithineachgroupofquestions.For thesecondmeasure,the categorical responses have been converted to ordinal scores (i.e., 0 for Not At AllEffective, 1 for Somewhat Effective, 2 for Effective and 3 for Highly Effective) and thensummed toestimate foreachschool theoverall levelofNPeffectivenessasperceivedbytheprincipal.
As all three measures derived from the survey are ordinal in scale, the Spearman rankcorrelation method was used to measure the association between school NP effects (asmeasuredbyVAeffectscores)andtheeffectivenessofNPimplementationasreportedbyprincipalsinthesurvey.
Figure5-1shows thevaryingnumberof initiatives schoolshave implementedwithineachreformareasacrossallNPschools.
Table 5.1 presents the results of the correlation analysis between the three measures(numberofinitiativesineachreformarea,principalperceptionsoflevelofeffectivenessofreforms,andwhethertheinitiativeshavecontinuedpostlowSESNP)andNPschooleffectVAscores.
Theresultssuggestthatforprimaryschools(Year5)thereisapositiverelationshipbetweenNP school effect scores and principal-reported level of effectiveness for initiativesimplemented as part of Reform Area 4, providing innovative and tailored learningopportunities.Thehigherlevelofeffectivenessasperceivedbytheprincipal,themorelikelytheschoolshowsahigherNPeffect.Theresultsalsoshowforprimaryschoolsarelationship
UniversityofCanberra VictoriaUniversity
26
betweenthenumberofinitiativesimplementedfromReformArea4andNPschooleffects.Themore initiatives implemented in a school, the higher theNP impact in the school. Itsuggeststhatschoolsthatplacedmoreeffortinthisreformareaweremorelikelytoshowastrongereffectonstudentlearningoutcomes.
Figure5-1 DistributionofNPinitiativesimplementedinschoolsbyreformareas
RF1:High-performingteachersand
principals
RF2-3:PerformanceandStaffingmanagement
RF4:Studentlearningopportunities
FR5:Schoolaccountability
RF6:Externalpartnershipswithparents,otherschools,businessesandcommunities
UniversityofCanberra VictoriaUniversity
27
Table5-1 Tests of Spearman Correlation between school NP effects and theeffectivenessofNPimplementationasperceivedbytheprincipal
Reading Numeracy
Coefficients Prob>|r=0| Coefficients Prob>|r=0| Primary(Year5)NumberofNPinitiativesimplemented
Attracthighperformanceteachers(RF01) -0.05 0.59
0.03 0.76Staffing&performancemanagement(RF02-03) -0.03 0.71
0.06 0.51
StudentLearningopportunity(RF04) 0.17 0.04 ***
0.18 0.04 ***Continuousimprovement(RF05) 0.10 0.25
0.15 0.07
Engagementwithparents/carers(RF06) 0.12 0.17 0.12 0.16 LevelofeffectivenessofNPinitiatives
Attracthighperformanceteachers(RF01) -0.05 0.59
0.03 0.76Staffing&performancemanagement(RF02-03 -0.03 0.71
0.06 0.51
StudentLearningopportunity(RF04) 0.17 0.04 *** 0.18 0.04 ***Continuousimprovement(RF05) 0.10 0.25
0.15 0.07
Engagementwithparents/carers(RF06) 0.12 0.17
0.12 0.16NumberofNPinitiativesretainedatschoolafterNP
Attracthighperformanceteachers(RF01) 0.17 0.05 ***
0.05 0.53Staffing&performancemanagement(RF02-03 0.04 0.71
0.00 0.97
StudentLearningopportunity(RF04) 0.09 0.33
0.09 0.36Continuousimprovement(RF05) 0.21 0.03 ***
0.09 0.36
Engagementwithparents/carers(RF06) 0.12 0.23
0.14 0.15 * Secondary(Year9)NumberofNPinitiativesimplemented
Attracthighperformanceteachers(RF01) -0.04 0.79
-0.22 0.10 **Staffing&performancemanagement(RF02-03) -0.17 0.19
-0.08 0.55
StudentLearningopportunity(RF04) -0.09 0.49
-0.03 0.85Continuousimprovement(RF05) -0.01 0.92 0.10 0.44
Engagementwithparents/carers(RF06) 0.03 0.81 -0.17 0.20 LevelofeffectivenessofNPinitiatives
Attracthighperformanceteachers(RF01) 0.08 0.56 -0.19 0.16
Staffing&performancemanagement(RF02-03 0.04 0.75
-0.08 0.54StudentLearningopportunity(RF04) 0.10 0.45 -0.02 0.87
Continuousimprovement(RF05) 0.17 0.21
0.11 0.40Engagementwithparents/carers(RF06) 0.20 0.14 * -0.06 0.67
NumberofNPinitiativesretainedatschoolafterNP
Attracthighperformanceteachers(RF01) 0.09 0.58
-0.20 0.20
Staffing&performancemanagement(RF02-03 0.05 0.73
-0.10 0.49StudentLearningopportunity(RF04) 0.04 0.78
0.05 0.73
Continuousimprovement(RF05) 0.07 0.63 0.07 0.65
Engagementwithparents/carers(RF06) -0.16 0.27 -0.28 0.05 ******Indicatesthatresultsarestatisticallysignificantat5%level;**indicatesstatisticalsignificanceat10%leveland*at15%level.
The results suggest that primary schools that placed more effort on initiatives linked toproviding innovative learningopportunitiesweremore likely to showastrongereffectonstudentlearningoutcomes.
UniversityofCanberra VictoriaUniversity
28
The relationships appear to be stronger ormore consistent for primary schools than forsecondary schools. This may be due to the small number of secondary schools (i.e., 46secondary vs 128 primary schools) included in the analysis, which reduces the statisticalpowerinthistest.
5.3 NPeffectsassociatedwithchangesinteacherandschoolpractice
Teachersinschoolsareanimportantsourceofinformationabouttheimplementationandeffectiveness of National Partnership strategies. Their views and reflections on the NPimplementationandeffectivenessintheirownschoolswerecollectedinthe2013TeacherSurvey.Atotalof2,408teachersfrom290NPschoolsrespondedtothesurvey.
ThelowSESNPevaluation(CIRES,2015)reportedthattheoverallviewsofteachersabouttheLowSESNPinitiativeswerestronglyfavourable,withthemajorityofteachersindicatingthatthelowSESNPinitiativeshadencouragedthekindsofchangesintended.
Inthisreport,asetofmeasureshasbeenconstructedusingteacherresponsestoaseriesofquestionsinthesurvey,includingthelevelofperceivedpositivechangeinteacherpracticesand the perceived level of positive change in school management as a result of NPimplementation.Bothmeasuresarecalculatedasthepercentageofteacherswhoprovidedavalidresponsefortherelevantquestions.
The Spearman rank order correlation method has been applied in testing the extent towhichthereisanassociationbetweenNPschooleffects(theNPeffectonstudentNAPLANscores)andpositivechangesinaschoolresultingfromNPimplementationasperceivedbyteachers. The Spearman correlation coefficients are calculated based on the ranking ofvariablesinthetest.Apositivecoefficientindicatesapositiverelationship,whileanegativeindicatesareversedrelationship.Forexample,thepositivecoefficientsforquestionsrelatedtochangesinteacherpracticeindicatesthatthemoreaspectsofchangeinteacherpracticeinaschool,themorelikelytheschoolshowsahighlevelofNPeffectonNAPLAN.
Table5.2presentstheresultsforNPeffectsonstudentNAPLANscores.Thereisapositiverelationship between the level of NP effect and the level of positive changes related toparents and community engagement in school. This suggests that the higher the level ofeffectivenessinparentsandcommunityengagementinaschool,themorelikelytheschoolshowsahighlevelofNPeffectoutcomeonNAPLANachievement.
UniversityofCanberra VictoriaUniversity
29
Table5-2TestofSpearmanCorrelationbetweenNPschooleffectandtheeffectivenessofNPimplementationasperceivedbyteachers
Reading Numeracy Coefficient Prob>|ρ| Coefficient Prob>|ρ|
Primary(Year5)
Changesinteacherpractice Meetingstudentindividuallearningneeds 0.08 0.31 0.13 0.11 *
Communicatingwithparents/carers 0.16 0.05 *** 0.17 0.04 ***Managingclassroombehaviours 0.08 0.34 0.11 0.18 Involvingparents/carersinstudentlearning 0.16 0.04 *** 0.04 0.64 Beingsupportedinclassroom 0.10 0.20 0.10 0.21 Abletoexplainschoolgoals 0.15 0.06 ** 0.16 0.04 ***
Teachingingeneral 0.12 0.12 0.06 0.44 Levelofpositiveviewsonschoolmanagement Theschoolrunsmoresmoothly 0.13 0.11 * 0.05 0.56 Teachersusebetterstrategiestosupportlearning 0.08 0.28 0.02 0.82 Improvementinthewayteachersrelatetostudents 0.06 0.44 0.04 0.58 Thisschoolusesmoreeffectivemethodstodetermine
howwellteachersareperforming 0.14 0.10 ** 0.12 0.15 *
Thereisamorestrategicapproachtoschoolplanning 0.09 0.27 0.11 0.15 *Schoolcommunicatesbetterwithparentsandcarers 0.10 0.20 0.17 0.03 ***
Schoolismoreeffectiveinengagingparentsandcarersfromdiversesocialandculturalgroups 0.12 0.15 * 0.19 0.02 ***
Schoolismoreengagedwithitswidercommunity 0.16 0.05 *** 0.15 0.06 **Schoolhasbecomeabetterplaceinwhichtoteach 0.15 0.06 ** 0.10 0.23 Schoolhasbecomeabetterplaceforstudentstolearn 0.13 0.10 ** 0.06 0.43
Secondary(Year9)Changesinteacherpractice Meetingstudentindividuallearningneeds 0.12 0.38 0.10 0.44 Communicatingwithparents/carers 0.07 0.61 0.24 0.08 **
Managingclassroombehaviours -0.07 0.61 0.13 0.37 Involvingparents/carersinstudentlearning 0.08 0.56 0.19 0.17 Beingsupportedinclassroom 0.13 0.35 0.15 0.29 Abletoexplainschoolgoals -0.05 0.71 0.17 0.23 Teachingingeneral 0.17 0.21 0.19 0.16 Levelofpositiveviewsonschoolmanagement Theschoolrunsmoresmoothly 0.02 0.89 0.04 0.78 Teachersusebetterstrategiestosupportlearning -0.03 0.80 0.12 0.36 Improvementinthewayteachersrelatetostudents 0.04 0.75 0.15 0.27 Thisschoolusesmoreeffectivemethodstodeterminehowwellteachersareperforming 0.01 0.97 0.04 0.77
Thereisamorestrategicapproachtoschoolplanning 0.09 0.52 0.19 0.16 Schoolcommunicatesbetterwithparentsandcarers 0.22 0.10 ** 0.14 0.31 Schoolismoreeffectiveinengagingparentsandcarersfromdiversesocialandculturalgroups 0.24 0.09 ** 0.26 0.06 **
Schoolismoreengagedwithitswidercommunity 0.22 0.10 ** 0.25 0.06 **Schoolhasbecomeabetterplaceinwhichtoteach 0.12 0.37 0.13 0.34 Schoolhasbecomeabetterplaceforstudentstolearn 0.05 0.71 0.13 0.33
***Indicatesresultsarestatisticallysignificantat5%level;**at10%leveland*at15%level.
UniversityofCanberra VictoriaUniversity
30
5.4 IndividualNPinitiativesorpracticesassociatedwithNPeffect
AllNPschoolsmaybefacedwithsomecommonchallenges,suchaslowattendanceandlowachievement, however some conditions or circumstances they operate under may vary.Although government schools were required to address all six areas of the NP reform,schools could prioritise their efforts or resources on particular initiatives or activities toaddresstheiruniquechallengesorconditions.
In this section, based on information provided by the principals (in the 2014 survey),wefocusontheimpactof individualNPinitiativesorstrategiesandattempttoidentifywhichparticularinitiativesorstrategiesmayindividuallyhavecontributedtotheimprovementofstudentNAPLANoutcomes.
5.4.1 GroupinglowSESschoolsbasedonestimatedlevelofimpactUsing the results of the multilevel linear model estimating Reading and Numeracyachievement to derive school effects (school level residuals), all schools (NP andnon-NP)weregroupedintothreecategories:
1. schools displaying higher NP effects on student NAPLAN scores, defined asschoolsrankedtowardsthetopofthedistribution(top25percent)
2. schoolsdisplayingpositive,butmoremodesteffectsonNAPLANresults,defined
asschoolsrankedinthemiddle50percent,and3. schoolsdisplayingnoeffectsorweakeffects,definedasschoolsrankedtowards
thebottomofthedistribution(bottom25percent).
Schoolswere rankedbasedon the values of their school residuals separately onReadingandNumeracyandseparatelyfortheirYear5andYear9cohorts.SchoolswithbothYear5andYear9resultsweregroupedaccordingtothehigherrankingamongthetworesults.
Table 5.3 shows the distributions of low SES NP schools according to their school effectcategory—low,middleorhigh—andbyschooltype.
ForReadingandNumeracy,about30percentoflowSESschoolsweretowardsthetopofthe performance distribution displaying higher levels of school effects on studentachievement,afurther50percentwereinthemiddlegroup,whileabout20percentwerein the bottom group exhibiting low effects on student achievement. In primary schools,nearlyeightoutoftenlowSESNPschoolsdisplayedatleastmodesteffectsonReadingandNumeracy performance with over 30 per cent in the top group of schools. Secondaryschoolsontheotherhandsawaround85percentinthetoptwogroups,butamuchhigher
UniversityofCanberra VictoriaUniversity
31
proportion—nearly40ofthisgroup—witheffectscoresplacingtheminthetopgroup.Onlyaverysmallproportionofcombinedschoolswereinthebottomgroup.
Table5-3 LowSESNPschoolsbyschooleffectcategory
Schooltype Schoolgroup Reading Numeracy
Primary Low 76 22% 80 23%
Middle 167 48% 163 47%
High 103 30% 103 30%
Total 346 100% 346 100%
Secondary Low 15 16% 13 14%
Middle 41 44% 47 50%
High 37 40% 34 36%
Total 93 100% 94 100%
Combined Low 2 4% 0 0%
Middle 37 79% 32 68%
High 8 17% 15 32%
Total 47 100% 47 100%
AllNPschools Low 93 19% 93 19%
Middle 245 50% 242 50%
High 148 30% 152 31%
Total 486 100% 487 100%
ItisrelevanttonotethatgroupinglowSESNPschoolsbasedontherankingoftheirschooleffectvalues is insomerespectsarbitrary,andtheremaynotbeanymaterialdifferencesamong some schools with residual values close to the cut off points separating groups.However,ourprimary interest istoexaminetherelationshipsbetweenthe levelofschooleffectsandtheimplementationofindividualinitiativesorstrategies,andtofurtheridentifywhatinitiativesorstrategiesthatmayseparateschoolsatthetopandatthebottomoftheschoolrankingbasedoneffectvalues.
Ordinallogisticregressionwasusedtotestthestatisticalsignificanceofschoolattributesinpredictingaschool’slikelyNPeffectgroup:low,mediumorhigh.TheresultsarepresentedinTable5.4.Theestimatesforeachpredictorcanbeinterpretedasmeaningthatforaoneunitincreaseinthepredictor(suchasattendance,forexample)weexpectanincreaseintheorderedlogoddsprovidedinthesizeoftheestimate(0.041)ofbeinginahigherNPeffectcategory,givenalloftheothervariablesinthemodelareheldconstant.
Theresultsshowthat:
UniversityofCanberra VictoriaUniversity
32
• Variationsinstaffexperience,ageandmale-femalestaffratiodonotappeartobepredictiveofaschool’sNPeffectoutcome.However,staffmobility(measuredasproportionofstaffturnoveroverin2010)appearstobeasignificantpredictorforNPeffectinNumeracy,takingintoaccountotherschoolfactors.Schoolswithlowerturnovertendedtobeinthehigherimpactcategory,allelseequal.
• TheconcentrationofLBOTEstudents inschoolappearstobeassociatedwitha
school’s likely effect groupoutcome forNumeracy, albeitwith aweak level ofsignificance.SchoolsinhighercategoriesofNPeffectforNumeracytendtohavehigherpercentageofLBOTEstudents,allelseequal.
• Theattendancerate ina lowSESNPschoolalsoappears tobeassociatedwith
theschool’slikelyeffectgroup.SchoolsinhighercategoriesofNPeffecttendtohavehigherattendancerates,allelseequal.ForNumeracy,attendancehadthestrongestindependenteffectofNPeffectcategorylocation.
Table5-4 Ordinal logistic regression model results identifying significant predictors ofsizeofNPschooleffectonNAPLANachievement
NPEffectonReading NPEffectonNumeracy
Estimate p-value Estimate p-value
SchoolSES-EOIindex 0.003 0.01 -0.001 0.58
PercentofLBOTEstudents -0.005 0.19
0.006* 0.15
SchoolLocation 0.144 0.54 0.077 0.74
Attendancerate 0.047* 0.15 0.064*** 0.03
Staffprofile
StaffMobility(turnoverrate) -0.001 0.79
-0.007 0.10**
AgeprofileIndex 0.008 0.61 -0.009 0.57Maletofemalestaffratio 0.002 0.71
0.007 0.21
Lengthofcurrentposition 0.012 0.38
0.011 0.43
***Indicatesresultsarestatisticallysignificantat5%level;**at10%leveland*at15%level.
UniversityofCanberra VictoriaUniversity
33
5.4.2 GreatestimpactNPinitiativesTo identify the sorts of strategies that high impact schools have implemented, and thatappeartomakeadifference,anordinallogisticregressionmodelwasappliedestimatingtherelationship between implementation of NP initiatives and predicted school NP outcomecategory(low,medium,highimpact).Theanalysiswasundertakenintwostages.First,allofthestrategiesschoolshadimplementedaspartoflowSESNP(asreportedbytheprincipal)were included as explanatory variables (in binary values, 1= Yes and 0=No). From thisanalysis,becauseof the largenumberof initiativesand the limitednumberof schools forwhichprincipal responseswereavailable, initiativeswere identifiedwhichhadprobabilityvalueslessthan0.4oneitherReadingorNumeracy.Second,ananalysiswasconductedontheshorterlistofinitiativeswhichweremorelikelytobeassociatedwithimpact.
Table 5-5presents the results of theordinal regressionprocedurewhere the groupingofschoolswasbasedonthelevelofNPeffect(low,medium,high).Selectedinitiativeswereincluded,thosesuggestingahigherinitialestimationofimpact.
Theresultsshowtheeffect foreach initiativeestimatedasan increase in theordered logoddsofbeinginahigherNPeffectcategory,withalloftheothervariablesinthemodelheldconstant. TheWald values and probabilities are also reported to identify if the effect isstatisticallysignificant.ResultsarepresentedforReadingandthenNumeracy.
For Reading, schools that have implemented the following initiativesweremore likely toshowhigherimpact:
• Providedopportunitiesforprofessionallearninganddevelopment(RF1)• ProvidedIndividualLearningPlans(ILPs)forstudentsneedingassistance(RF4)• Providedprofessionallearningforteachersonmeetingindividuallearningneeds(RF4)• Usedassessmentandotherdatatoidentifystudentneeds(RF4)• Engagementwithemployers(RF6)• Partnershipswithprimaryschools(RF6)
ForNumeracy,schoolsthathaveimplementedthefollowinginitiativesweremorelikelytoshowhigherimpact:
• Providedopportunitiesforprofessionallearninganddevelopment(RF1)• ProvidedIndividualLearningPlans(ILPs)forstudentsneedingassistance(RF4)• Providedprofessionallearningforteachersonmeetingindividuallearningneeds(RF4)• Usedassessmentandotherdatatoidentifystudentneeds(RF4)• Madeplanningand/or reportingprocesseswithin theschoolmorepubliclyavailable
(RF5)• Engagementwithemployers(RF6).
UniversityofCanberra VictoriaUniversity
34
Table5-5 NPstrategiesassociatedwithNPeffectimpactcategory(low,medium,high)in
ReadingandNumeracy Reading Numeracy
Estimate Wald Prob Estimate Wald Prob
ReformArea1:Attracthighperformanceteachers
Establishedleadershipandstrategicpositions -0.829 3.684 0.055 ** 0.072 0.027 0.869
Providedmentoringsupporttoteachers -0.120 0.088 0.766
-1.319 9.099 0.003 ***Providedattractiveterms/conditionsoutsidestandard
entitlements -0.306 0.223 0.637
-1.643 5.977 0.014 ***Providedopportunitiesforprofessionallearningand
development 0.371 0.773 0.379
0.471 1.152 0.283
ReformAreas2-3:Staffingandperformancemanagement Employedadditionalparaprofessionalsinlearningsupportroles 0.314 0.564 0.453
0.285 0.439 0.507
Employedadditionalparaprofessionalsinothersupportroles -0.521 1.622 0.203
0.520 1.519 0.218
Adoptedteam-basedapproachestoteachingandplanning 0.338 0.795 0.373
-0.571 2.155 0.142 *Introducedincreasedflexibilityintimetablingand/orschool’s
hours -1.078 8.557 0.003 *** -0.076 0.042 0.838 Introducedgreatercooperationwithotherschoolstoshare
resources 0.174 0.205 0.651
0.381 0.949 0.330Providedqualityprofessionallearningforschool-basedteams
ofstaff 0.267 0.474 0.491
-0.279 0.488 0.485 Implementedrelevantandappropriateprofessionallearning
forstaff -0.500 0.870 0.351
-0.184 0.112 0.738 Providedprofessionaldevelopmentonarangeofbehaviour
managementtheoriesandapproaches -0.696 2.669 0.102 * -0.666 2.410 0.121 *Engagedstaffthroughprofessionaldialogueonbehaviour
management 0.303 0.489 0.485
0.511 1.367 0.242
ReformArea4:StudentLearningopportunity
Usedassessmentandotherdatatoidentifystudentneeds 0.708 2.761 0.097 ** 0.699 2.121 0.145 *Implementeddifferentiatedteachingmethodstobettermeet
theneedsofallstudents -0.414 1.022 0.312
-0.403 0.904 0.342 Providedprofessionallearningforteachersonmeeting
individuallearningneeds 0.551 2.194 0.139 * 0.569 2.222 0.136 *
Providedstudentswithaccesstolearningsupportservices -0.469 1.348 0.246
-1.073 6.343 0.012 ***
ProvidedILPsforstudentsneedingassistance 0.675 2.418 0.120 * 1.633 12.750 0.000 ***
ReformArea5:Strengtheningschoolaccountability
Madeplanningand/orreportingprocesseswithintheschool
morepubliclyavailable -0.006 0.000 0.986
0.691 3.677 0.055 **
Expandedtherangeofschoolactivitiesthatareevaluated -0.050 0.017 0.897
0.432 1.181 0.277ReformArea6:Externalpartnerships
Parentsandcarersgenerallyintheschoolcommunity 0.103 0.062 0.803
-0.372 0.783 0.376
Employers 1.334 6.200 0.013 *** 0.777 2.111 0.146 *
OneormoreTAFEsandtrainingproviders 0.586 1.403 0.236
-0.692 1.812 0.178
Oneormoresecondaryschools -0.583 2.182 0.140 * -0.204 0.256 0.613
Oneormoreprimaryschools 0.678 3.245 0.072 ** -0.118 0.095 0.758 Note***P<.05**p<.10*p<.15
UniversityofCanberra VictoriaUniversity
35
ForReading, schools that implemented the following initiativesweremore likely to showlowerimpact:
• Establishedleadershipandstrategicpositions(RF1)• Introducedincreasedflexibilityintimetablingand/orschool’shours(RF2/3)• Providedprofessionaldevelopmentona rangeofbehaviourmanagement theories
andapproaches(RF2/3)• Partnershipswithoneormoresecondaryschools(RF6)
ForNumeracy,schoolsthatimplementedthefollowinginitiativesweremorelikelytoshowlowerimpact:
• Providedmentoringsupporttoteachers(RF1)• Providedattractiveterms/conditionsoutsidestandardentitlements(RF1)• Adoptedteam-basedapproachestoteachingandplanning(RF2/3)• Providedprofessionaldevelopmentona rangeofbehaviourmanagement theories
andapproaches(RF2/3)• Providedstudentswithaccesstolearningsupportservices(RF4)
In lookingat the initiativesnegativelyassociatedwithNP impact, there is forNumeracyasignificantdifferencelinkedtostrategiesattemptingtoattractordevelophighperformingteachers.Possiblyreflectinggreaterstabilityinstaffing,highimpactschoolswerelesslikelyto provide mentoring support or establish attractive terms and conditions to recruitteachers. Higher impact schools did, however, have a significantly higher likelihood ofimplementingstrategiestoprovideopportunitiesforprofessionallearninganddevelopmentofstaff.
While in terms of Reading scores, higher impact schools were more likely to haveimplementedstrategiesaroundpartnershipswithprimaryschools, theywere less likelytoimplement strategies associated with staffing and performancemanagement. Again, thismayreflectthechallengesfacedbylowimpactschoolsaroundstaffingandmanagement.
Thedifferences in focus and emphasis amongNP schoolsmaywell reflect the challengesthat they faceand towhich theyare responding, rather than theeffectsof thestrategiesthemselves.Forexample,schoolsthatshowlowNPimpactonstudentNAPLANresultshaveplacedgreateremphasisonattractinghighperformingteachersandimprovingthestaffingandperformancemanagement roleof theprincipal. Theymayhave issues in staffingandmanagement practices that need (and the schools recognise that need) to be addressed.High impact schools, on the other hand,may havemore stable staffing profiles, and it istheir staffing andmanagement stability thathasenabled these schools to target areasof
UniversityofCanberra VictoriaUniversity
36
improvement in teachingpracticeand inaddressingstudentneedsthathasdeliveredrealtractionfromtheNPinitiatives.
Finally, many other contextual factors that are not captured by the data may have aninfluence on the effectiveness of NP implementation in NP schools. For example, thereseems to be difference in specific approaches and operational practices in NPimplementation across NP schools. In the following section we present case studies ofschools that provide further contextual insights and understanding of results from thequantitativeanalyses.
UniversityofCanberra VictoriaUniversity
37
6. CasestudyschoolsPreviously, as part of the agreed work program for the evaluation of NP initiatives, anumberofcasestudiesofschoolswereundertakentoexaminemorein-depththeadoptionand operation of low SES reform initiatives. Thesewere presented in a series of reports(CIRES,2012;CIRES,2013;UoC,2012,UoC,2013)focusingontwodifferentNPreformareasrespectively:evaluationofSchoolStaffing,ManagementandAccountabilityandevaluationof school External Partnerships. The case studies provide a snapshot of what specificstrategies/actionshavebeenimplementedinschoolsandhowthesestrategies/actionshaveworked in different school settings. They are useful to re-examine in the context of thisreport because some are high impact schools in terms ofNP effects onNAPLAN studentlearningoutcomes,somearemediumimpact,andotherslowimpact.Subsequentanalyseswill draw further on case studydata to illustrate the complexity of schools’ planning andimplementationconsiderationsinarrivingatstrategiesthatwillbebothlocally-appropriateandeffective. What canwe learn fromthese schools that canhelpourunderstandingofdifferentialeffects?
Inthisreport,wehavelinkedsomeofthecasestudyschoolswiththeirNPeffectdataand,therefore, areable toprovide further contextual insightsonwhat initiativeshaveworkedwelland inwhatcontext theyareeffectiveornoteffective in improvingstudentNAPLANperformance.
In this section, we present four schools as examples of differences in context and effectoutcomes.Theseschoolsdiffer in termsof their school settings, thechallenges they face,andthefocusoftheNPstrategiesandinitiativesthattheyimplemented.
Table6.1providesageneralprofileof the schools.As inearlier reports thenamesof theschools have been changed to remain consistent with the original guarantees ofconfidentialitygiventoschools.
TheprofilesinTable6.1revealthatbothValleyPublicSchoolandSouthCoastHighSchoolshowahigh levelNPeffectonNAPLANoutcomesandahigh levelofeffectiveness in theimpactoftheirNPstrategiesandinitiatives,withonebeingaprimaryschoolandtheotherasecondary school. No schoolwith low impact on both Reading andNumeracy have beenmatchedtothecasestudyschools.
WesternPublicSchoolshowsanaverageresultonNPeffectinbothReadingandNumeracy.TheschoolenteredtheLowSESNationalPartnershipsprogramatapointofinstabilityinitsleadership arrangements. Essentially the Low SESNP funding and processes provided theschoolwithopportunitiesandspacetofocusonestablishing,implementingandcontinuallyreviewingitsimmediateorganisationalchangemanagementmodel.Interviewswithschoolleadershipandteachersrevealedarangeofreformsandinitiativesappliedbytheschoolas
UniversityofCanberra VictoriaUniversity
38
partofitsparticipationinNationalPartnerships.Howeversurveyresponsesandotherdatasuggest that these processes and strategies have not been effective and have not beentranslatedintoimprovedNAPLANoutcomes.
Table6-1 Profilesofcasestudyschools
WesternHigh SouthCoast
HighValleyPublic WesternPublic
Schoolprofile
SchoolType Secondary Secondary Primary PrimarySchoolLocation Metropolitan Provincial Provincial MetropolitanSchoolSES(SESEOIIndex) Low(Q4) Low(Q4) Medium(Q3) Medium(Q2)
Fundingprograms
NPparticipation 2010 2010 2010 2010PrioritySchoolProgram2010 Yes No Yes Yes
Staffprofile(2010)
Malestaff(proportion) Medium(35%) High(65%) Medium(16%) Medium(7%)Ageindex Medium(Q3) High(Q4) High(Q4) Medium(Q2)
<30 13% 2% 4% 21%30-49 57% 23% 60% 57%50+ 30% 75% 36% 21%
StaffMobility High(32%) Medium(8%) Medium(10%) Low(0%)
Studentprofile
AboriginalConcentration High(13%) High(10%) High(25%) Medium(4%)LBOTEConcentration High(44%) Medium(3%) Low(1%) High(40%)Attendancerate2011 Low(82%) Low(89%) Low(90%) Medium(94%)Attendance2013 Low(83%) Medium(89%) Low(92%) Medium(94%)
NPEffectonNAPLAN
Numeracy Medium High High Medium
Reading Low High High Medium
WesternHighSchoolshowsalowlevelofNPeffectinReadingandanaverageNPeffectinNumeracy. Western High School provides a good example of the complexity of thechallenges confrontingmany schools in planning and acting on school improvement. TheSchool’splannedresponsedevelopedundertheNPinitiativefocusedonliftingtheschool’sperformance.Asuiteofstrategiesandinitiativeswereimplementedcoveringabroadrangeof NP reform areas, including building leadership and staff capacity, utilising quality datamanagement systems and learning methods for students and building community andparentpartnerships.However,evidencesuggeststhattheplannedstrategiesandactivitiesfailed to translate through to substantial improvements on student NAPLAN outcomeswithin the timeframe of the evaluation. As discussed below the case demonstrates thechallenges and the time required to shift school outcomes in a very challengingenvironment.
UniversityofCanberra VictoriaUniversity
39
6.1ValleyPublicSchool
NPeffectonNAPLANachievement:HighonbothReadingandNumeracy
SchoolProfileValley Public School is a provincial Government public school that caters for over 400kindergarten to Year 6 students. Located in the Hunter Central Coast, the school has ateachingteamofover20staffandboastsasmallbutactiveParentandCitizenAssociation(P&C)andactiveinterestfromlocalEldersandtheAboriginalEducationConsultativeGroup(AECG).In2013theschool’sICSEAvaluewas855,whichismarkedlybelowthestatemeanand similar to the school’s ICSEA value in 2010. In 2013 the school was made up of 26percent indigenous students, and four percent of students were from a languagebackgroundotherthanEnglish(LBOTE).ValleyPublicSchoolhadanattendancerateof92percentandtheschooloffersanumberofprogramsforgiftedstudentsaswellasstudentswithlearningdisabilities.
MainChallengesandareasforimprovementValleyPublicSchoolfacedanumberofchallengesduetoalevelofparentaldissatisfactioninhomeandschoolcommunication;accesstoteachers;inputintodecisionmaking;aswellasthetransitionprogramtosecondaryschool.Theschoolhadbecomeawareofanumberofconcerns raised by the Parents and Citizens Committee (P&C), including perceivedinconsistency in the provision and approach to homework, poor attitudes to teacherstowardsparents,andinadequatecommunicationaroundwhattheschooloffersandhowitmakesdecisions.
EffectiveActions/InitiativesValleyPublicSchoolpursuedanumberofstrategiestoaddressthechallengeshighlightedbyparentsandthecommunity.AspartoftheNPreformeffort,theschool:
1. Establisheda strategic focusonparentpartnerships through theemploymentof aCommunityEducationOfficer(CEO)
ThispositionwasfundedbytheLowSESNPProgram,andsupportedteachers,parentsandstudentstoestablishrelationshipswithintheschoolandcommunity.TheCEOhelpedcreateaculturalchangewithintheschoolbymediatingandaddressingmattersbetweenparentsand teachers. This was accomplished by the CEO’s implementation of regular parentsurveys, parent focus group meetings, attendance to P&C meetings and participation inactionresearchprojects.Staffreportedthatthesepositivelinkswithparentsbenefitedtheteacher-parent relationship,which led tobettercollaborationonstudent learning.Due tothesuccessoftheCEO,theP&Chasofferedtoraisefundstomaintainthepositionbeyonditscurrentfunding.
2. Tried to establish a better process of communicating with parents through activeengagementindialoguewithparents
UniversityofCanberra VictoriaUniversity
40
Anumberofinitiativessuccessfullyimprovedthedialoguebetweentheschoolandparents.ThepresenceofthePrincipalintheplaygroundduringmorningsandafterschoolcreatedaninformal space forparents to interactwith thePrincipaland school staff.Additionally theschool’s newsletterwasused as a communication tool topromotepositive conversationswith a new format that included student achievements, learning journeys and individualstories.Overall,parent’sappraisaloftheschoolshiftedtoapositiveone,evidentinsurveyresults.
3. IncreasedparentparticipationthroughmeasuresthatimprovedtheeffectivenessoftheP&Candencouragedparentstoprovidefeedbackandtomakesuggestions
TheschoolreportedthattheP&Coperatedmoreeffectivelyandnewpeoplejoinedastheculturalchangeencouragedmoreparentparticipation.AroundeightparentsattendedeachPandCmeetingin2012comparedwithsubstantiallymorescantorunreliableattendancesin earlier years. Additionally, a number of parents volunteered in the school canteen.Furthermore,promptedbytheschool’seffortstocreateabetterdialogue,parentsprovidedfeedbackandmadesuggestionsvia informalcontactwithstaff,attendancetofocusgroupmeetingsandteacherforumsorparticipationinschoolsurveys.Parentsurveysundertakenat the end of 2012, indicated that 67 per cent of parents reported that the provision ofregularcommunicationisadequatecomparedto50percentin2011.
4. Developedparentknowledgeaboutstudentlearningthroughtheprovisionofparentinformationsessionsandworkshops
Theschoolimplementedparentinformationsessionsandworkshops.Theseweredesignedtofacilitateparent’sknowledgeabouttheirchild’slearning.Fromtheseexperiencesparentshighlightedthedesire foradditionalopportunities toobserveteachingand learning intheclassroom.
5. Facilitated home-school collaboration through the improvement of Three-WayConferences and engagement of parents in student discipline, as well as theirtransitiontohighschool
The implementationof theThree-WayConferencebetween teacher, parents and studentwas successful. After altering it so that parents and teacher meet prior to meeting thestudent, the Three-Way Conference became a tool to create constructive dialogue,acknowledge achievements, find solutions, make commitments and plans for the future.Additionally,theschoolranmeetingsforparentstodiscussstrategiesforimprovingstudentbehaviours.Alongwith increasedparent attendance to assemblies that celebrate studentachievementsandawards,theschoolnotedadropinthesuspensionrate.Furthermore,theschoolranmeetingsforparentswithchildrenwhoaretransitioningtoYear7inanefforttodiscusstransitionstrategiesandhowboththeschoolandparentscouldbettersupporttheirchildren.
UniversityofCanberra VictoriaUniversity
41
ReflectiononresultsTheValleyPublicSchool’sefforttocreateasustainable,opendialoguewithparentscreateda positive cultural change that benefitted the students, staff and community at large.Parents respondedwell to the school’s incorporationof parentpartnerships to its overalloperation,evidentinthepositivefeedbackfromparentsurveys, increasedcommunicationwithstaffandattendancetoforum,meetingsandworkshops.Additionally,theworkoftheCEOpavedthewayforfurtherdevelopmentofrelationshipsbetweenparents,theP&C,thePrincipal,schoolleadersandteachers.TheschoolandP&CacknowledgedtheimportanceoftheCEO’sworkandexploredstrategiestocontinuetheworkaftertheLowSESNPProgram.The CEO has been the cornerstone to the positive cultural changes that have led toimprovedparentpartnershipswithinValleyPublicSchool.
6.2SouthCoastHighSchoolNPeffectonNAPLANachievement:HighonbothReadingandNumeracy
SchoolProfileSouthCoastHighSchool isaco-educationalgovernmentsecondaryschool inthe IllawarraandSouthEastRegionwith justunder690enrolments in2013(thekeyyearforanalysis).Studentnumbershaveremainedfairlystablesincetheearly2000s.
ThreedistincttypesorgroupsoflargelyEnglish-speakingfamilieswereidentifiedbyschoolleadersasmakinguptheschool.While themajorityof familiesare low-incomeandoftensingleparenthouseholds,thereisarelativelysmallgroupoffamiliesinwhichbothparentsare employed. In addition a significant Aboriginal community within the town and itssurroundingsmeansthattheproportionofAboriginalstudentswithintheschoolhassatataround12percentformanyyears.
KeyChallenges
Improved parental engagement was viewed as a key area of need, particularly for theindigenous population using the school. Critical to this was working out how to betteraccommodatetheneedsofAboriginalfamilies.Therewerespecificchallengesindoingthis,challenges associated with better communicating with these families, in particular withregard to the role of the home in students’ education. Linked to this, the school faced anumberofchallengesinincreasingthenumberofparentsinvolvedintheschoolgenerally.Initialstrategiesexplicitlyaddressedoutreachapproachesto“hardtoreach”parents.
Effectiveactions/initiatives
South Coast High School’s Low SES NP plan aimed to lift the attendance, retention andperformance of students through a whole-of-school and community engagement reform
UniversityofCanberra VictoriaUniversity
42
approach. The approach reflected the priorities of a regional comprehensive secondaryschoolcateringtotheneedsofadiverseschoolcommunity.Partnershipswithfamilieswerefundamental to its strategic focusandaparentengagementstrategywasconsideredasaleadingdrivertoimprovementsinschool-homerelationships.Thestrategyincluded:
• Enhanced communicationwithparents including regularnewsletters, regular emailcontacts with parents, changed use of school website (updated regularly andincreasinglytheinformationconduitforteachers,parents,studentsandcommunity).As a result of the extensive promotion of the parent and teacher meetings innewslettersandSMSmessaging,theschoolreportedthatin2013therewasamorethan 30 per cent increase in the number of parents attending parent and teachermeetingscomparedtopreviousyears.
• Toassistparents totakeonastrongerrole in theirchildren’s learning, informationsessions were provided to offer parents opportunities to learn about the schoolsystemandtheschool’sownstrategiesaimedatsupportingchildrentoimprovetheirperformance. Regular workshop sessions were scheduled for this purpose andactivelypromoted.Parentswereexplicitlyencouragedtovolunteerattheschoolandinvitedinfollowupcontactstoidentifyareaswheretheycouldbebestcalledontoassist–suchas inReadingsupport, languageteachingsupportandassistancewithexcursions.Thesesessions,designedtoassistparentstoengagemorefullywiththeirchildren’slearningatschoolandathomewereincreasinglywell-attended.Sessionsand workshops covered themes such as: family Reading; writing and Numeracy;positiveparenting;understanding reportsandstudentdata; studentpathwaysandcareeroptions;andstudentwellbeingandmentalhealth.
• EffectivelyengagingaprioritygroupbyworkingdirectlywithAboriginal families inrevitalisation of the Personalised Learning Plan (PLP) processes. The school’s newapproachtoPLPs,whichaimedtoincreasethelevelofstudentengagement,enlisteddirectsupportfromparentsandfamilies.AlocalcoachwasemployedtoredevelopthePLPstobuildtheirutilityasaplanforfamiliesandstudentsthat“enabledthemtolearn”.Thenewapproachshiftedemphasisfromateacher-directedapproachtoone that highlighted student and parent ownership of the process. Aboriginalparents were encouraged to engage with the school in a more positive way.Accordingtofeedbackformtheschool,formanyparentsitmarkedadeparturefromcoming into the school only as a result of misbehaviours or expulsions. The PLPmeetingwithparentswasdesignedtogiveparentsa“positivereasontocometotheschool”,andenabledthemtoprovideinputintotheirchild’slearningplans.
Reflectiononresults
An important early outcome from improved and targeted communication between theschoolandthehome,accordingtotheschool,wasasignificantimprovementinthestudent
UniversityofCanberra VictoriaUniversity
43
attendance rate. A head teacher (welfare) was employed to coordinate a whole- of-communitywelfare support service, andNP fundingalso supported theemploymentof aspecialist school administrative support (SAS) staff member to provide administrativeassistanceforNPinitiativesassociatedwithstudentengagementandattendance.
Withintwoyearsofinitialimplementationtheschoolwasabletoidentifysomeindicationsthat the parent engagement strategy had led to changes in parent expectations of theschoolandintheirconnectionwiththeirchildren’slearning.
School personnel reported that parent engagement had shifted from a limited group ofparentsofhighperformingstudentstoamorerepresentativebodywithamore informedandfocussedinterestonhowtheschoolviewedstudent learningandachievement:“theirinterest was evident through their questions about student learning and schoolexpectations”(principal).
The school reported that strengthened parent and community engagement enabled theschool to take amore rigorous approach toworkingwith families in an effort to supportproductivestudentlearningandextendedopportunitiesavailabletostudents.Forexample,engagementofparents intheSchoolBasedApprenticeshipandTraineeshipsprogramwascitedasanexampleofsomethingthathadincreasedthesuccessoftheprogram.Parentssupportedtheestablishment, implementationandreviewofprogram. Inparticular,manyparents were instrumental in assisting students to locate suitable employers andworkplaces.
6.3WesternPublicSchoolNPeffectonNAPLANachievement:MediuminbothReadingandNumeracy
SchoolProfile
Western Public School is a preschool to Year 6 school with around 210 students in theprimary school and 40 pre-school students. The school is located in theWestern SydneyRegion. Its 2013 enrolment of 223 and attendance rate of 94 per cent are both slightlyhigherthanthoseofearlieryears.Theschool’sAboriginalpopulationisgrowing(8percentin 2013) and 53 per cent of students are reported to be from families of a languagebackgroundotherthanEnglish.
KeyChallenges
WesternPublic entered the LowSESNPprogramat apointof instability in its leadershiparrangements.Between2007and2012theschoolhadseenfourprincipals;atthetimeofinitialinterviewin2012thethenprincipalhadbeenintheschoollessthanaterm.
UniversityofCanberra VictoriaUniversity
44
KeyStrategiesandinitiatives
Partly as a result of this extendedperiod of instability, and in linewith the school’s ownassessmentofimmediateneeds,itsNPplanandstrategiesfocussedonimprovementofthequality of its teaching, school planning and accountability processes. In consequence, theschoolestablishedanewexecutiveleadershipmodeltoimproveleadershipeffectivenessinthe school. The executive team was expanded to include stage and learning areacoordinators.Members of thenewexecutive team, including theprincipal, had access tomentors and also had opportunities to participate in team based leadership trainingprograms.
Afurtherkeygoaloftheschoolwastobuildteachercapacitybyimplementingawholeofschool approach to professional learning. Data associated with student achievementoutcomeswereusedtoinformthedesignofstaffprofessionallearningactivities.Itlinkeditsstaffprofessionallearningwithschoolplandevelopment,withafocusontheimprovementofstudentprograms.Forexample,teacherswererequiredtoensurethattheyusereflectivejournals or learning logs which incorporate relevant Low SES SSNP professional learningactivities.
Reflectionontheresults
Essentially, theLowSESNP fundingandprocessesprovidedtheschoolwithopportunitiesandspaceto focusonestablishing, implementingandcontinually reviewing its immediateorganisational changemanagementmodel. Thesewere subject to changeover the life oftheprogram.Forexample,teachersandleadersregardedtheirreviewprocessesasinitiallynot usefully linked to the broader context of the school, restricting the capacity of theprocess to contribute strategically to achieving school goals. A more explicit link to theschoolplanasrequiredwassubsequentlyimplemented.
These elements – consolidating effective leadership and implementation ofwhole schoolchangeinteachingpractice—servedasimmediateandpressingprioritiesforthisschoolinitsengagementwiththeLowSESNP.
At thepointof interview in2012therewasaperceptionfromschool leadershipandstaffthattheprofessionaldialogueintheschoolhadimproved.Staffmembersweredescribedasmorewillingtodiscussanddebateandinsodoingabletobetteraddressissues.
WesternPublicSchool,whereplannedactivityfailedtotranslatethroughtotheobjectiveofsubstantialimprovementsonstudenttestscores,demonstratestheneedforunderstandingofthetimerequiredtomaketheculturalshiftsthatunderpinenhancedschooloutcomes.
UniversityofCanberra VictoriaUniversity
45
6.4WesternHighSchoolNPeffectonNAPLANachievements:LowinReadingandMediuminNumeracy
SchoolProfile
WesternHighSchool isacoeducationalgovernmentsecondaryschool(Years7-10)withanenrolmentof575in2013,downfrom630in2012.ItservesthecurricularandsocialneedsofaverydisadvantagedandhighneedspopulationintheWesternRegionofSydney.Nearlyhalf its student population report language backgrounds other than English andmany ofthese are from Pacific Islander families. A further 15 per cent of the student populationreportAboriginalorTorresStrait Islanderbackgrounds.Parentemployment isprimarily inunskilledorlowskilledwork;unemploymentisalsohighandsignificantnumbersofparents,including many single parents, are not in the labour force. In the first year of NAPLANtesting, the school reported 94 per cent of its student population to be drawn from thelowestSESquartile.
AttendanceininitialyearsoftheLowSESNPwasreportedat81percentandtheschool’sinitiatives includedasignificantdirectionofNP funding toward improving theserates.MySchoolreportsreflectrisingratesofattendance,to85percentin2014:
Year 2010 2011 2012 2013 2014
StudentAttendance Rate(%)
81 83 82 83 85
Source:MySchool
Challenges
Theschool’sLowSESNPProgramrespondedtoaninitialsituationalanalysisundertakenin2009thatidentifiedanumberofdistinctconcernstobeaddressedthroughNPinitiatives.
• First,students’learningandengagementoutcomeswerelowerthanstateaveragesona rangeof keymeasures suchasNAPLAN scores, attendancedata and studentretention.
• Second,staffing issuespresentedfurtherconcerns—attritionamongteacherswashighwithacomparativelyhighturnoverofstaffandtheproportionofcomparativelyinexperiencedstaffwasalsoaconcern.Theschoolwantedtoholdenergeticyoungstaff and better support all staff in working more effectively with their peers;sustainedprofessionalsupportanddevelopmentwasneededtohelpteachersbettermeetstudents’complexneeds.
UniversityofCanberra VictoriaUniversity
46
• Third, the school identified community perceptions of the school as problematic;parents’engagementwith theschoolwasvery lowandparticularly lowforcertaingroups(suchasAboriginalstudents)withintheschoolcommunity.
KeyStrategies/initiatives
The planned response developed under the initiative therefore focused on lifting theschool’sperformanceonseveral linkedbutdistinctfronts.Nothingshortofaculturalshiftwithin theschool– inareasofbuilding leadershipandstaffcapacity,utilisingqualitydatamanagement systems and learning methods for students and building community andparentpartnerships–wasregardedasrequiredtoeffect thedesiredchanges instudents’literacyandnumeracyachievementlevels.
ActivitiesassociatedwiththeNPwereaccordinglyspreadacrossseveralkeyareasofactivitywithastrongfocusonleadershipandstaffimprovement.AnAssistantPrincipalpositionwascreated to oversee the complex program associated with the NP strategies. At theleadershipandstafflevel,forexample,severalkeyinitiativeswerereportedtobeintegraltotheschool’splansforimprovedoutcomes.In2012theseinitiativesweredescribedas:
• Focusonperformancemanagementandstaffmentoring,whichhadalready led to“positive change” in the school’s staffing profile. Long-time staff who had beenresistanttoreformshadlefttheschoolwhileothers,energisedbythenewactivities,notonlywantedtoremainbuthadtakenoneducationalleadershipchallengesandwere actively encouraging like-minded colleagues to join the school community.Youngstaffinparticularhadservedas“attractors”toothersimilarly-motivatedstaffwithin theirnetworks. Being regardedasan“employerof choice”wasanewandwelcomeexperienceforthisschoolandagreatmoralebooster.
• Sharingofpractice:thebuildingoflearningteamsandprofessionalrelationshipswasakeyfocusoftheNPinitiativesandfundamentaltoreformswithintheschoolandresourcesoftimeandprogramfundingwereallocatedtosupportingthisinitiative.
• Strategic use of technology: a classroom-based digital recording system allowedteacherstoshareandcommentontheirteachingpractice,buildingamorecollegialand team based approach to practice. The system was described as “constantly”utilisedandverypopularwithstaff.Otherschoolsdrewonthisschool’sexampleinsettingup similar systems.A factor in itsacceptancehadbeen involvementof thewhole staff in development of professional protocols and safeguards surroundingthe program – leadership emphasised from the outset that the purpose of theprogram was not oversight and control but the sharing and encouragement ofinnovativepractice.
• Teachersasresearchersundertookprojectsonresearchtopicsassociatedwiththeirpractice –these projects were regularly reviewed by peers and awards given for
UniversityofCanberra VictoriaUniversity
47
strong practice. The process played an important role in changing staff culturethroughopeningupclassrooms,collaboratingonactivitiesandsharinginformation.
• Ateacher-basedreviewcommitteewasestablishedtooverseenewstaffingappraisalarrangements and students and parents had regular opportunities to providefeedback on teachers (three times a year). This information and associatedreflections played a role in Teacher Annual Assessment reviews. These newprocesses were described as “evaluative and reflective processes” of greatimportanceincontributingtochangesinteachers’professionalapproaches.
As noted a significant focus ofWHS’ initiative also involved improvement of attendancerates through amix of engagement strategies and substantial teacher allocations for themonitoringandfollowingupofstudentabsences.Workstudiesclasses,forexample,provedsuccessful in turning around attendance and completion rates for at-risk Year 9 and 10students.Whileattendanceratesremainedlowerthanstateaveragesimprovementswereseenannuallysincetheintroductionofthesemeasures:
Year 2012 2013 2014
Student AttendanceRate
82 83 85
Source:MySchool
AnabidingconcernforWHSwasthatparentengagement intheschoolwasvery low.Theconcerntoenhanceparentandcommunitypartnershipsconstitutedafinalstrategicfocusfortheschool’sLowSESNPplan:Iftherewasonethingwecouldfixthatwouldbeit,theparentengagement(Communityengagementcoordinator).
• The school allocated Low SES NP funds to develop and implement a parent andcommunityengagement strategy. Twopart-timepositionswerecreated in2009:aParent Education coordinator; and a Community Education Officer. Parents wereencouraged to channel their requests for assistance on specific concerns orquestions to these staff in the first instance to promote a “triaging” that wouldensuretheirissueswereaddressedinwaysthatweretimelyandeffective.Anotherteacher was allocated a fractional load to improve content and delivery of thenewsletter as a means of communicating more effectively with the schoolcommunity.
• Theschoolexperimentedwith itsorganisationofparentand teachermeetingsandforums,always verypoorlyattended in thepast. Itwas successful in staging somecommunity evenings attended by many parents and community members andsoughttobuildonthesuccessofsuchpositiveevents.Moretargetedactivitiesforspecific community groups were also organised, such as administration of surveyinterviewsandfocusgroupmeetingsforAboriginalfamilies.Inconsequencearange
UniversityofCanberra VictoriaUniversity
48
ofopportunitieswereidentifiedforteacherstofacilitatehome-schoolpartnerships,especially inmaking direct contactwith parents to encourage their attendance atassemblies, award ceremonies, other celebratory community events, and studentextra-curricularactivities.Teachersreceivedprofessionaldevelopmenttobuildtheirconfidenceandskillsinworkingwithallparents,inparticularwithregardtostudentattendance.
Reflectiononresults
Western High School provides a good example of the complexity of the challengesconfrontingmany schools in planning and acting on a school improvement plan. In thiscase,theplannedandimplementedactivitiesandinterventionsfailedtotranslatethroughtotheschool’sobjectiveofsubstantialimprovementonstudenttestscores.Theexperienceof WHS highlights a point made by a number of research studies on improvementinterventions in school settings (Lamb and Rice 2009, CDRP 2015): that the interactionsbetween school culture (characterised in terms of staff and student stability, orderlyenvironmentsandconsistencyof leadership)andthequalityandfocusof interventionwillimpact on effect. The case demonstrates the need for understanding of the time or pre-conditionsrequiredtomakeimprovementinschooloutcomes.
Leaders in this school made reference to “lining up the ducks” for improvement andbelieved that their NP approaches –which included targeting 1. student attendance andengagement, 2. teacher and leadership capacity and staff renewal and 3. School-parentrelations–weresettinguptheconditionsforimprovementstobemade.
By 2012, teachers and parentswere able tomake some cautious assessments about thesuccess of this strategy: “there is an increase because itwas just so low before. Studentattendancehasdefinitelyincreasedalbeitremaininglowerthanstateaverages.Therehavebeenincreasesinawardshandedoutforacademics,sporting,citizenship,andattendance”.Therewasasense,too,thatstaffdevelopmentstrategieshadbroughtanewenergytotheschoolandsupportednewandmoreproductivewaysofworking.
ForWHS,however,thesuiteofstrategiessopainstakinglydevelopedandimplementedaspartoftheschool’sNPplanshowedverylittleoutcomeinstudentperformanceinNAPLAN.The intractability of poor test outcomes, especially in NAPLAN results, was a source offrustration,disappointmentanddisbeliefamongschoolstaff,giventhatschoolplansandarange of pedagogical and staff development initiatives had been introduced with theexpresspurposeofaddressingandturningaroundNAPLANperformances.
Essentially the school’s planning and activity, especially in its targeting of studentengagementandstaffdevelopment,hadcentredonculturalchange.Thestrategiesfocusedonthetransformationofstaffasprovidingtheclearestandmostdirectwaytosecurethe“runson theboard” required fora stronger,more stable school communitywithpositive
UniversityofCanberra VictoriaUniversity
49
community profiles and a strong interfacewithparent communities. Itwas felt that thisculturalshiftwouldinturnwouldliftperformanceoutcomesinlongrun.
The challenges faced by this school weremany. Interim indicators thatmay have betterreflectedincrementalgainsmadeovertheperiodofthestudycouldhavebeenlinkedmoredirectly to the factors associatedwith the preconditions for success rather thanwith thestudent outcomes themselves. These could have linked for example to attendance, staffdevelopment or parental engagement outcomes, all factors associatedwith a productiveanddynamic school culture. Indicators of change, especially in the context of communityexperiences and perceptions, could be the quality and incidence of contactsmadewithparents outside the contexts of conventional parent involvement – such as the parent-teacher evenings, school forums or other contexts that tend to define parental roles inschool. Strategies that enlist parent support in building attendance rates or developingindividuallearningandpathwayplansforyoungpeopleatriskinvolveparentsinwaysthatconnect closelywith students’ experiencesof school; the impactofprograms suchas theschool’sYear10workstudiesprograminengagingparentsinstudents’programsandthusimprovingattendanceandretentionratesisimportant.Similarly,theinitiativesofyearlevelcasemanagers incontacting familiesasamatterofcoursehasbuiltconnectionsbetweenschool and the home. Steps such as these provide a stronger platform for furtherengagement and their role in shifting parent-teacher relationships to stronger andmorepositive foundations may be found in parent and teacher perceptions of incrementalchange.
UniversityofCanberra VictoriaUniversity
50
7. Conclusion
ThisstudywascommissionedbytheCentreforEducationStatisticsandEvaluation(CESE)toexamine student performance in low-SES schools which participated in the NationalPartnership Agreements, and the extent to which changes in student performance arerelatedtoNPinitiativesorotherfactors. Ofparticular interestwasidentifyingwhetherornot schoolexternalpartnershipsandassociatedNP initiatives lead to improvedoutcomesfor schools and students. Toexamine thisquestion, a rangeof analyseswereundertakenusing available data to evaluate the effectiveness of the Low SES NP in improving NSWstudentoutcomesacrossarangeofmeasuresincludingNAPLANresults.
Thereportexplorestheissueofgoodpracticeusingdataderivedfromavarietyofsourcesincludingfromadministrativedata,NAPLANachievementdata,surveyresultsfromteacherandprincipalsurveys,andcasestudymaterialscollectedinearlieryearsfromsitevisitstoselected schools. Drawing on information from a variety of data sources helps to gain abetterunderstandingoftheeffectoftheinitiativesimplementedbyschools.Importantinthis are several earlier reports,many of which provided highly detailed accounts of howspecificschoolshaveworkedwiththeNPorhowschoolsandschoolpersonnelacrossNSWhaveexperiencedthe initiativesovertime.Thesearevaluablepiecesofwork intheirownright and their findings inform this report. But themain focusof this report lieswith thebroadquestionofoutcomeandimpact.
Previous work, documented in the low SES NP staffing, management and accountabilityevaluation report, has identified positive effects of the NP program at an aggregate orsystemlevel. Inparticular,theanalysisusingNAPLANmatchedcohortdatasuggestedthatlowSESNPschoolsonaveragehaveasignificantimpactonstudentNAPLANresultsover3yearsbetween2011and2013.Whatthecurrentreportaimedtodowastounpackfurtherthe results of the earlier work to identify variations in impact across schools in order tobetterunderstandwhereandinwhichcontextsparticularstrategieshaveworkedwell,andwheretheyseemtohavehadlittleeffect.
Among NP schools there is a positive relationship between the level of their NP effectoutcomeandtheoveralllevelofeffectiveimplementationintwoNPreformareas:
• ReformArea4—Provideinnovativeandtailoredlearningopportunities,and
• Reform Area 6 — Promote external partnerships with parents, other schools,businessesandcommunities.
Themoreinitiativesrelatedtothesetwoareasimplementedinaschool,thehigherimpactof NP in the school. The higher level of effectiveness as perceived by the principal and
UniversityofCanberra VictoriaUniversity
51
teachers,themorelikelytheschoolshowsahigherNPeffect.Itsuggeststhatschoolsthatplacedmoreeffort inthesetworeformareasweremorelikelytoshowahighereffectonstudentlearningoutcomes.
In looking at what separated high impact schools from low impact schools, the analysisrevealed that schools with the following characteristics weremore likely to show a highimpactontheirstudentNAPLANimprovement:
• They had placed greater emphasis than other schools on innovations in teachingpracticeandaddressingstudentneeds
• Theyhadprovidedstrongerwelfareandlearningsupportfordisadvantagedstudentsthrough providing teachers with training around individual learning needs, andstrengtheningtheuseofILPsforstudents
• Strongeruseofevidencetoidentifystudentlearningneeds• Stronger emphasis on developing staff through providing opportunities for
professionallearninganddevelopment
ForschoolswheretheNAPLANachievementeffecthadbeenlowornegligible,theanalysisrevealedthatschoolstendedtobecharacterisedbythefollowing:
• Some had a stronger emphasis on strategies to attract and retain teachers,particularlythroughleadershippositions,whichmayitselfreflectgreaterneedinthisareaandschoolsstrugglingtorecruitandretainqualityteachersandleaders
• Lessfocusonuseofevidenceforstrategicplanninganddecision-making• Less focuson innovation in teaching, addressing studentneeds, and strengthening
schoolaccountabilityandmoreonstaffingandperformancemanagement• Pursued strategies around teacher and leadership recruitment, retention and
developmentsuggestingthatstaffstabilitywasamajorissueforthem.
Inmeasuring variations inNAPLAN impact across schools, this report has shown howsomeschoolshaveimprovedstudentoutcomesasaresultoftheNP,whileothershavebeen less effective in doing so. The next phase ofwork for this projectwill include acloserstudyofanumberofcasestudyschoolsthathavedisplayedhighNPimpact,andanumberoflowimpactschoolstotrytoteaseoutwhathascontributedtodifferentialimpact. Thiswill be complemented through amore extensive analysis based on newandmorecomprehensivedataoftheinitiativesthathavebeenemployedinschoolsthathaveaddedsignificantlymorevaluetostudentachievementrelativetootherschools,andthe initiatives of schools that have held ground but not necessarily improved relative tootherschools,andtheinitiativesofschoolswithdecliningornegligibleperformance.
UniversityofCanberra VictoriaUniversity
52
References
AustralianCouncilforEducationalResearch(ACER)(2013).PISA2012:HowAustraliaMeasuresUp
AustralianCouncilforEducationalResearch(ACER)(2012).HighlightsfromTIMSS&PIRLS2011
fromAustralia’sperspective.
Centre for Research on Education Systems (CRES) (2012). Evaluation of the School Staffing,Management and Accountability Initiatives. Progress Report No. 1. Prepared for: the NewSouthWalesNationalPartnershipsEvaluationCommittee.
COAG Reform Council (2010). Education 2010: Comparing performance across Australia. Sydney:COAGReformCouncil.Retrievedfrom:
www.coagreformcouncil.gov.au/reports/docs/education/2010compare/education_2010_report.pdf
CouncilofAustralianGovernments(2008).NationalPartnershipAgreementonLowSocio-EconomicStatusSchoolCommunities.Retrievedfrom:
www.smarterschools.gov.au/nationalpartnerships/Documents/SSNatPartnerAgreem.pdf
CouncilofAustralianGovernments (2009). "NationalEducationAgreement: Fact Sheet."Retrievedfrom:
www.coag.gov.au/coag_meeting_outcomes/2008-11-29/docs/20081129_national_education_agreement_factsheet.pdf
Darling-Hammond, L. (2000). "TeacherQualityandStudentAchievement:AReviewofStatePolicyEvidence."EducationPolicyAnalysisArchives8(1).
Department of Education and Training (2011). Low Socio-economic Status School CommunitiesNationalPartnership.InformationPackageforSchools.Retrievedfrom:
www.lowsesschools.nsw.edu.au/wcb-content/uploads/psp/file/2_percent20LOW_SES_InfoPackage.pdf
Elmore,R.(2000).BuildingaNewStructureforSchoolLeadership.Washington:TheAlbertShankerInstitute.
Elmore,R.(2007)."EducationalImprovementinVictoria."Retrievedfrom: www.eduweb.vic.gov.au/edulibrary/public/staffdev/schlead/Richard_Elmore-wps-v1-
20070817.pdf
Field, S.,M. Kuczera& B. Pont (2007).NoMore Failures: Ten Steps to Equity in Education. Paris:OECD.Retrievedfrom:
www.education.vic.gov.au/edulibrary/public/publ/research/publ/policy_and_provision-research_rpt.pdf
Friedlaender,D.&L.Darling-Hammond(2007).HighSchoolsforEquity:PolicySupportsforStudentLearning inCommunitiesofColour.SchoolRedesignNetwork,StanfordUniversity.Retrievedfrom:www.srnleads.org/press/pdfs/hsfe_report.pdf
Fullan,M.(2001).LeadinginaCultureofChange.SanFrancisco:Jossey-Bass.
Gustafsson, J. (2003). “What do we know about the effects of school resources on educationalresults?”SwedishEconomicPolicyReview10.
Harris,A.,T.Allen,&J.Goodall(2008).“CapturingTransformation:Howschoolssecureandsustainimprovement”.Retrievedfrom:
www.almaharris.co.uk/files/capturing_transformation.pdf
UniversityofCanberra VictoriaUniversity
53
Helal, M. (2014). Evaluating the Impact of the Smarter Schools National Partnerships on StudentAchievement,MelbourneInstituteWorkingPaperSeries:2014.
Kellock,P.,G.Burke,etal. (2007).TheUseofEquityFundingto ImproveOutcomes,CentrefortheEconomicsof EducationandTraining,MonashUniversity.Reportprepared for theVictorianDepartmentofEducationandEarlyChildhoodDevelopment.
Lamb,S.(2007).“SchoolReformandInequalityinUrbanAustralia:ACaseofResidualizingthePoor”.SchoolReform&Inequality.R.TeeseandS.Lamb,Dordrecht:Springer.
Lamb, S.&B.McGaw (2007). TheperformanceofAustralian schools, internationalbenchmarking.Federalist Paper 2: The Future of Schooling in Australia. Retrieved from:www.caf.gov.au/Documents/TheFutureofSchoolinginAustralia.pdf
Lamb,S.&R.Teese(2005).EquityprogramsforgovernmentschoolsinNewSouthWales:aReview,CentreforPost-compulsoryEducationandLifelongLearning.
Loveless,T. (2013)The2013BrownCenterReportonAmericanEducation:HowWellAreAmericanStudentsLearning,Volume3,Number3.
McKinsey & Company (2007).How the world's best-performing school systems come out on top.Retrievedfrom:
mckinseyonsociety.com/downloads/reports/Education/Worlds_School_Systems_Final.pdf
Ministerial Council on Education Employment Training and Youth Affairs (2008). MelbourneDeclaration on Educational Goals for Young Australians. Ministerial Council on EducationEmploymentTrainingandYouthAffairs.
National Smarter Schools Partnerships (2011).New SouthWales Progress Report 2011. Retrievedfrom:www.nationalpartnerships.nsw.edu.au/resources/documents/2011-ProgressReport.pdf
NewSouthWalesDepartmentofEducationandTraining.IncentivesMenu.SmarterSchoolsNationalPartnerships on Improving Teacher Quality and Low Socioeconomic Status SchoolCommunities.Retrievedfrom:
www.nationalpartnerships.nsw.edu.au/resources/documents/ITQ-LSES-DETIncentivesMenu.pdf
NSW Department of Education & Training.National Partnership of Low SES School Communities:ResearchUnderpinningtheReforms.Retrievedfrom:
www.lowsesschools.nsw.edu.au/wcb-content/uploads/psp/file/Resources/Reforms_paper.pdf
NSWNationalPartnershipsEvaluationCommittee(2011).ProjectBrief:EvaluationofSchoolStaffing,ManagementandAccountabilityInitiatives,AttachmentA:AnOverviewofInitiatives.TenderdocumentprovidedtoCRES.
NewSouthWalesDepartmentofEducationandCommunities(NSWDEC)2011),NewSouth
WalesDepartmentofEducationandCommunitiesdiscussionpaper:Australianschoolfunding
arrangements,papersubmittedtotheReviewofFundingforSchooling.
OECD. (2005). Teachers Matter. Attracting, Developing and Retaining Effective Teachers.OrganisationforEconomicCo-OperationandDevelopment:Paris.
ProductivityCommission.(2012).SchoolsWorkforce.ResearchReport,Canberra.
Robinson,V.(2007).“Theimpactofleadershiponstudentoutcomes:Makingsenseoftheevidence”.Theimpactofleadershiponstudentoutcomes:MakingsenseoftheEvidence.
Smarter Schools National Partnership (2011). New South Wales Smarter Schools NationalPartnershipsImplementationPlan.PreparedbytheAustralianGovernment,NewSouthWales
UniversityofCanberra VictoriaUniversity
54
Government, Association of Independent Schools NSW and NSW Catholic EducationCommission.Retrievedfrom:
www.nationalpartnerships.nsw.edu.au/resources/documents/2011-NSW-Implementation-Plan.pdf
Teese, R. and S. Lamb (2009). Low achievement and social background: patterns processes andinterventions; a Discussion paper. Centre for Post-Compulsory Education and LifelongLearning,TheUniversityofMelbourne.
Welch,A.,S.Helme,etal.(2007).“RuralityandInequalityinEducationtheAustralianExperience”.InR.Teese,S.LambandM.Duru-Bellat, InternationalStudies inEducational Inequality,TheoryandPolicy,Springer:Netherlands:271-293.
Zbar,V.,R.Kimber,etal.(2008).Howourbestperformingschoolscomeoutontop:anexaminationofeighthighperformingschools,DataandEvaluationDivision;DepartmentofEducationandEarlyChildhoodDevelopment.
UniversityofCanberra VictoriaUniversity
55
AppendixTableA-1 Schoollevelcovariancestatistics(Intra-classCorrelationCoefficient)* Linearmixedmodel
Schoolvariance ICC
Notadjustedforstudentandschoollevelfactors
Year5Reading 0.18 18%
Year5Numeracy 0.19 19%
Year9Reading 0.25 25%
Year9Numeracy 0.38 38%
Adjustedforstudentandschoollevelfactors
Year5Reading 0.01 4%
Year5Numeracy 0.02 8%
Year9Reading 0.01 1%
Year9Numeracy 0.01 5%
*Allresultsonschoollevelvariancearestatisticallysignificantat1%level.