operationalizing ie: case study example: textbooks and teacher training in sierra leone apeie...
TRANSCRIPT
Operationalizing IE:Case Study example: Textbooks and
Teacher Training in Sierra Leone
APEIE Workshop, May 10-14 2010
Operational Key Steps
2. IE Design
4. InterventionStarts
1. Policy
5. Survey
6. Analysis, Reports, Policy Advice3. Baseline
Survey
IE Design – methodological issues
Available data and required data Identify control group and treatment group
– Randomization– Transparency
Sampling and sample sizes– Power calculations– Budget and access constraints
Identify outcome indicators
Data Instruments
Utilize survey experts Who do you survey?
– Households, community, facilities, government…
Pilot instruments– Clear questions and coding– Includes all information needed for analysis and
covariates
Intervention Roll Out
Possible contamination of evaluation results!– Intervention benefits rolled out to control and
treatment schools evaluation compromised– Intervention or treatment schools only receive
additional benefit evaluation compromised
Monitor roll out– Train implementers – they need to know why!– Hire supervisors– Continue supervision throughout roll out
Background
In June 2007, the World Bank and a team from the Sierra Leone Ministry of Education and Sports (MOES) met to rigorously evaluate the impact of two pilot studies on student performances.
Two questions– What is the effect of textbooks on student performance?– What is the effect of teacher training on student
performances?
Random Selection– 5 districts in Sierra Leone
Kailahun, Kambia, Pujehun, Western Rural, Western Urban– Neediest schools in district (using EMIS data)
Student to teacher ratio, presence of latrine, school construction
360 schools total Random Assignment
– Comparison schools: receive regular allotment of books– Books schools: receive additional books– Books and training schools: receive additional books +
teacher training
How do we design the study to get rigorous results?
Timeline
Sept: WB and MOE agree on pilot
Oct: Met with local councilors to select districts
Nov: Baseline data collection and teacher training manual developed
Nov: Mid-term data collection
June: Baseline data collection
Dec: Roll out of textbooks and teacher training
June: Roll out of textbooks
2007 2008 2009
Dec: Mid-term data collection
NO teacher training up to this point
REALITY
TARGET
What happened?
Met with several challenges on the ground that caused significant delays in the project:
– Textbook distribution– Teacher Training– Data Collection
Challenge 1: Textbook Distribution
Target date: December 2007 Actual date: June 2008 Number of books:
– MOES accidentally distributed original books, so had to use Sababu books
– had fewer books than project initially called for Communication – information not filtering down
– Sababu not properly informed – Service Providers unwilling/unable to change distribution– Headteachers in comparison schools unaware of project
Challenge 1: Textbook Distribution continued
Textbook distribution: June 2008 Data collection: December 2008 Consequences:
– Not enough books distributed in some districts to make a statistical comparison
– Distribution happened very late and at a bad time (right before summer vacation)
Allowing for only a small window for students to utilize the extra textbooks
– Some Comparison schools received books
Challenge 2: Teacher Training
Target date: December 2007 Actual date: has not yet occurred Time and budget constraints The ‘+’ part of the ‘intervention +’ so took a
back seat to the textbook distribution
Challenge 3: Mid-term Data Collection
Target date: November 2008 Actual date: December 2009 Delayed funding disbursement Non-existent schools (poor quality of initial
data) Headteachers unaware of project – not
willing for enumerators to collect data
Solutions:
Back up list of schools on hand for enumerators Better communication with all levels of organization
involved Constant project sensitization Greater monitoring of distribution Inclusion of Process Surveys Allow for sufficient time for procurement and
disbursement
Next Steps
Analyze mid-term data – Was there a significant effect due to textbooks?– Has education in Sierra Leone improved in the past 2
years?– Where in the textbook distribution are there major
informational stop gaps?
Move forward on teacher training Write up findings and circulate report
Conclusion
Impact evaluations a useful tool for policy considerations
But need to understand every process involved in the impact evaluation
– Textbook distribution, not just textbooks Study can be ineffective if you don’t have full buy-in
and understanding at all levels of government, all service providers, and headteachers:
– SENSITIZATION!– COMMUNICATION!– MONITORING!