STRENGTHENING ACCOUNTABILITY IN PRIMARY EDUCATION IN MADAGASCAR:

RESULTS OF AN IMPACT EVALUATION

Jee-Peng Tan & Cornelia Jesse,

HDNED Chief Economist Office & Results for Development Institute Seminar “Demanding Good Governance - Inside & Out”

March 25, 2010

Primary Education in Madagascar: Much Progress, but still Many Challenges

Signs of progress: Primary completion rate doubled from 35% in1999 to 71% in 2008

Evidence of weak sector performance: ½ of each cohort of 1st graders does not finish the primary cycle; Repetition rate still high at 18% in 2005 (30% in 2000) Low quality: in 2004-5 PASEC, average test score of 50% in Maths

and Malagasy and 32% in French; deteriorated since 1997-98

Multiple systemic causes : Inconsistencies in teacher allocation across schools; Ineffective management of pedagogical processes at

school and classroom levels

A Problem of Internal AccountabilityTasks essential for student learning are neglected:

Inadequate supervision of pupil and teacher absenteeism

e.g. more than 80% of directors fail to report teacher absences to administrators at the sub-district and district levels

Neglect of basic pedagogical tasks e.g. 20% of teachers don’t prepare daily lesson plans Poor monitoring of and communication on

student learninge.g. results of student tests and quizes are poorly recorded and communicated to parents, if at all; school directors hardly follow-up on student performance: 3/4 don’t discuss learning outcomes with their teachers

What tasks are deemed essential?

Teacher: Takes daily roll call Prepares daily lesson plan Prepared bi-monthly lesson plans

Monitors student learning Has tested pupils during the past two months

Helps lagging students Discusses student learning issues with the director

School director: Keeps a register of enrollments

Signs off on daily roll call Analyzes student absences on a monthly or bi-monthly basis

Reviews pupils’ test results Takes stock of teacher absences

Informs sub-district or district officer about teacher absences

Follows up with teachers on lesson planning

Tighter Management to Improve AccountabiltyConceptual Intervention Framework & IE

Design: Workflow tools to clarify tasks and internal

accountabilities; Facilitation of meetings between school and community; Better information flows within school and between

school and community; Structured training for teachers and school heads

Leading to: improvement in actors’ behavior through

bottom up and top down accountabilitybetter managed school

increased school quality higher student learning

Key Questions for Policymakers

What is the impact of tighter management of processes on school functioning and student performance?

At what administrative level are management interventions the most effective (school, district or inspection level)?

Impact Evaluation Design (1)

Method: Randomized experimental design over 2 school years

Interventions: Specify actors’ responsibilities & their mutual

accountability the processes through: Management Tools and Guides for key tasks (e.g.

pedagogical, administrative) Training

Focus attention on results to clarify goals through: Report cards: School, district and inspection report

cards School meetings: Facilitated school meetings &

development of school improvement plans based on school report cards

School & District Report Cards for Better Information Flow

Report cards for school directors, sub-district and district levels officers:

Complement the tools and proceses Draw attention to schooling outcomes Include comparative data, allowing a school

to compare its outcomes with those of other schools

Serve as basis for dialogue and accountability

Impact Evaluation Design (2)

9

303 Schools AGEMAD

TREATMENT 3

303 SchoolsZAP AGEMADTREATMENT 2

15 CISCO AGEMAD

89 ZAP AGEMAD

80 ZAP CONTROL

303 SchoolsCONTROL

15 CISCOCONTROL

84 ZAPNON-AGEMAD

303 SchoolsCISCO AGEMADTREATMENT 1

Collecting Data

Actors’ Behavior (direct effects): Questionnaire from impromptu school visits in 1,200

schools, with information for 4,000 teachers Questionnaires for District and Community admin. level Collection and analysis of tools used in 40 schools (850

tools)

Schooling outcomes (indirect effects): Test scores from standardized tests in 3 subjects National year-end school census data: flow rates,

repetition, CEPE pass rate

Timeline: 2 school years, 2005-2007 Baseline survey/test and post-intervention survey/test

Results: Effects on Actors’ Behavior

 AGEMAD schools

Control schools

Teacher absence (%) 8.7 9.2

Teacher completes all key tasks (%) 63.0* 42.4

All teachers in school perform their key tasks (%)

42.9** 22.1

Well managed schools (%) 36.5** 15.2

*significant at 5% level; **significant at 1% level

12

Results: Effects on Schooling Outcomes

 AGEMADschools

Controlschools

Student attendance (%) 90.7* 86.6

 Repetition (%) 17.5* 22.6

 Drop out (%) 5.5 6.1

 Success rate at CEPE exam 73.0 61.9

Student test results (post-test)    

Math 51.2 49.4

Malagasy 50.9 48.5

French 30.0 29.4

All subjects 43.5 41.9

*significant at 5% level

Policy Implications Prioritize school-level actors

“Cascade” training model alone, as currently defined, doesn’t work Though results are encouraging, better management essentially entails

changing peoples’ behaviors, which takes time and effort Mainstream IE results into MoE activities

Need a champion from the start Need early involvement of a national team, with good technical support Necessary to sustain change in actors’ attitudes & behaviors

Use existing structures and mechanisms for scale up: Tools, guides and training modules integrated into teacher training Tool distribution, training and facilitated school meetings funded

through the local catalytic funds based on regional, district and school performance plans and needs

Develop leaders to drive change in management practices Discussion underway on collaboration in leadership training between

Madagascar MoE and partner organization in another country

Stay tuned…Publications forthcoming

Africa Human Development Working Paper Series«Améliorer la gestion de l'enseignement primaire à Madagascar - Résultats d'une expérimentation randomisée »

Journal Article undergoing peer review «Managing for results in primary education in Madagascar: Evaluating the impact of selected workflow interventions »

It takes a village…

Government commitment: Stable counterpart team (15 staff from MoE with coordinator)

Partner commitment: Financial and technical assistance from AFD (via two staff) WB team lead by Jee-Peng Tan and Cornelia Jesse, consisting

of Gérard Lassibille and Trang van Nguyen (with in-country field coordinators)

Local NGO Aide et Action to assist with training

Financing: WB, AFD, MoE, EFA-FTI (EPDF), Irish Aid, Ministry of Foreign Affairs of Norway

Timeline: 2004 – 2007 Total number of people involved: 50

The Perils of Data Collection…