summary annual progress report - … · summary annual progress report october 1, ... tos teacher...

EdData II

Task Order 15: Data for Education Programming in Asia and the Middle East (DEP-AME)

Summary Annual Progress Report October 1, 2013 – September 30, 2014 REDACTED VERSION Education Data for Decision Making (EdData II) Technical and Managerial Assistance Contract Number AID-EHC-E-00-04-00004 Task Order Number AID-OAA-BC-11-00001, RTI Task 15 01 October 2013 – 30 September 2014 This report was produced for review by the United States Agency for International Development. It was prepared by RTI International.

Task Order 15: Data for Education Programming in Asia and the Middle East (DEP-AME)

Summary Annual Progress Report October 1, 2013 – September 30, 2014 EdData II Technical and Managerial Assistance, Task 15 EdData II Contract Number EHC-E-00-04-00004-00 Task Order Number AID-OAA-BC-11-00001 RTI Project No. 09354.015 Prepared for Mitch Kirby USAID/Asia Bureau, Senior Education Advisor 1300 Pennsylvania Ave, NW Washington, DC 20523

Prepared by Michelle Ward-Brent RTI International* 3040 E. Cornwallis Road Post Office Box 12194 Research Triangle Park, NC 27709-2194 RTI International is one of the world’s leading research institutes, dedicated to improving the human condition by turning knowledge into practice. Our staff of more than 2,800 provides research and technical expertise to governments and businesses in more than 40 countries in the areas of health and pharmaceuticals, education and training, surveys and statistics, advanced technology, international development, economic and social policy, energy and the environment, and laboratory and chemistry services. For more information, visit www.rti.org. RTI International is a trade name of Research Triangle Institute.

The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government.

http://www.rti.org/

EdData II Task Order 15 (DEP-AME)

Annual Report Year 3 (01 October 2013 – 30 September 2014) iii

Abbreviations

Allegro Allegro Global Procurement Solutions, Inc.

AME Asia/Middle East

CLA central level agencies

CMS content management system

DEP-ASIA/ME Data for Education Programming in Asia and Middle East

DEO district education officers

DepED Department of Education of the Philippines

DD Deputy Director

DM Deputy Minister

DP development partner

ECD early childhood development

Ed-Data Education-Data

EMES Education Management Efficiency Study

EGR early grade reading

EGRA early grade reading assessment

ETC education training center

FY Fiscal Year

GBAO Gorno-Badakshan Autonomous Province

GOB Government of Bangladesh

IPD Institute of Professional Development

LOI language of instruction

MENA Middle East/North Africa

MOE Ministry of Education

MOEHE Ministry of Education and Higher Education

MTB-MLE Mother Tongue Based-Multilingual Education

NA not applicable

PA Palestinian Authority

PT parent-teacher association

RC research centers

RTI Research Triangle International

SES socio-economic status

SMC school management committee

SOW Scope of Work

TBD to be determined

TOS Teacher Observation Study

USAID U.S. Agency for International Development


iv EdDataII TO 15: DEP-AME Annual Report Year 3 (01 October 2013 – 30 September 2014)

Table of Contents

Page

Abbreviations ....................................................................................................... iii

Executive Summary ..............................................................................................1

Programming and Project Support ........................................................................1

Activity Status ........................................................................................................2

Report Summary ...................................................................................................2

Operational Activities .............................................................................................6

USAID Communication and Meetings ...................................................................6

Reports and Deliverables ......................................................................................6

Overall Progress: Results, Requirements, Standards ............................................8

Progress toward Objectives (by Result) ..............................................................14

Result 1: AME Mission Strategy-Related Data Needs Met ............................14

Result 2: Availability of AME Education Data and Trends Expanded.............29

Result 3: Measurement Tools for Cross-Country Use ....................................32

Result 4: USAID Staff Knowledge Strengthened ............................................44

Annex A: Financial Summary ............................................................................ A-1


Annual Report Year 3 (01 October 2013 – 30 September 2014) 1

Executive Summary

Task Order 15 (TO 15), Data for Education Programming in Asia and Middle East (DEP-

ASIA/ME), is a centrally-funded Time and Materials Task Order under the Ed-Data II

Indefinite Quantity Contract (IQC) between the U.S. Agency for International

Development (USAID) and RTI International, with support from Allegro Global

Procurement Solutions, Inc. (Allegro).1 The contract was awarded with a Task Order

ceiling price of [redacted]. The project began on September 26, 2011. A September 2014

contract modification raised the level of obligated funding from [redacted] to [redacted].

The contractual end date is presently November 30, 2014. A no-cost extension request

has been submitted to enable DEP-AME to complete the activities begun and within the

scope of work. Labor categories use fixed daily rates.

Support activities under the Task Order are intended to provide technical assistance and

training services in direct support of USAID’s 2011 Education Strategy in Asia and the

Middle East, in 17 Missions implementing or planning basic education programs,

including Afghanistan and Pakistan. Specific focus is placed on ensuring that education

data needs are met in the region. It is designed to support Mission responses to USAID’s

2011 Education Strategy, and its three specific goals:

Goal 1: Improved reading skills for 100 million children in primary grades by

2015

Goal 2: Improved ability of tertiary and workforce development programs to

generate workforce skills relevant to a country’s development goals

Goal 3: Increased equitable access to education in crisis and conflict

environments for 15 million learners by 2015

Programming and Project Support

DEP-ASIA/ME is intended to generate regional and country-specific education data—

and analysis of those data—that can be used by Asia and Middle East (AME) bureaus,

missions, and partner countries to prioritize education needs and corresponding

investments. The project is also intended to strengthen local skill in design, evaluation,

and management of education programs, and quality data capture and analysis to support

them, across the region. Activities have been designed for four core Result Areas.

Result 1: AME Mission Strategy-Related Data Needs Met

Result 2: Availability of AME Education Data and Trends Expanded

Result 3: Measurement Tool with Applicability across Countries Developed

Result 4: Education Personnel Knowledge of Selected Education and Evaluation

Topics and Processes Strengthened

1 Contract number EHC-E-00-04-00004-00; Order Number AID-OAA-BC-11-0000.


2 Annual Report Year 3 (01 October 2013 – 30 September 2014)

Activity Status

This report covers activities implemented in four Result areas across a 12 month period

from October 1, 2013-September 30, 2014. Activity status updates are summarized in the

table below.

Results Activity Status

Result 1 Egypt: Grade 3 Early Grade Reading Assessment Baseline Completed

Result 1 Egypt: EGRA Baseline Comparison Electronic vs. Paper Completed

Result 1 Philippines: Analytic Support Services for Early Grade Reading Completed

Result 1 Nepal: EGRA, Teacher Observation Study, Management Effectiveness Studies Completed

Result 1 Bangladesh: Institutional Assessment of Early Grade Reading Underway

Result 1 West Bank: Early Grade Reading Assessment Underway

Result 1 Morocco: Reading Research Studies Completed

Result 1 Tajikistan (GBAO) Early Grade Reading Assessment Underway

Result 2 Topical Analysis: Scale-up and Sustainability Submitted

Result 2 Topical Analysis: Early Grade Reading Instruction in Arabic Completed

Result 2 Translation of Topical Analysis: Early Grade Reading Instruction in Arabic Completed

Result 3 Reading Improvement Modeling Tool (EGRA Barometer) Underway

Result 4 Mission Education Staff Training Modules on Impact Evaluation Completed

Result 4 Evaluation for Education Projects Workshop (Bangkok) Underway

Reports Submission of Yearly Report #11 Completed

Report Summary

Quarterly Activity Overview

Activities undertaken under each of the four Result Areas during Fiscal Year (FY) 2014

proceeded at a rapid pace this year with the largest volume of analytic analysis and

technical assistance support occurring under Result 1. This included direct support to six

USAID Mission programs. Several activities concluded this year under Result 1 and



Result 2. Significant increases in technical support also occurred under Result 3 and

Result 4.

First quarter activity included continuation of existing activities and the launch of several

ones, including support for the Bangladesh: Institutional Assessment of Early Grade

Reading. This field-based activity had been delayed several times due to security and

scheduling considerations. On October 21, the Mission requested that the assessment

team be fielded during November. On November 15, DEP-AME was informed that the

November timeline would need to be postponed until January for security reasons. In

response, DEP-AME conducted desk review work during the intervening months through

December in preparation for eventual field mobilization in January, 2014.

Second quarter activity continued analytic support for activities in the West Bank, the

Philippines, Nepal, Bangladesh, Morocco and Tajikistan. This included the field launch

of the Bangladesh: Institutional Assessment of Early Grade Reading; planning and

launch of the Morocco: Reading Research Studies; completion of survey data collection

for the Nepal: EMES, TOS, and EGRA; implementation of the Philippines: Analytic

Support Services for Early Grade Reading; planning for the Tajikistan : Early Grade

Reading Assessment in Tajikistan/GBAO; and implementation of the West Bank: Early

Grade Reading Assessment implementation in Palestinian schools.

Third quarter technical assistance and analytic support further advanced these activities

through field-based research, data analysis, and policy dialogue workshops. This included

the revision and completion of the Bangladesh: Institutional Assessment of Early Grade

Reading report draft based upon written feedback from USAID and the field Mission;

field research and data collection for the Morocco: Reading Research Studies;

completion of the report and conduct of a policy dialogue workshop in Katmandu with

USAID, the MOE, and stakeholders at which the Nepal TOS, EMES, and EGRA studies’

findings were shared; continued implementation under the Philippines Analytic Support

Services for Early Grade Reading in preparation for an August Policy Dialogue

Workshop; mobilization and field launch of the Tajikistan/GBAO Early Grade Reading

Assessment in GBAO with meetings with USAID, the MOE, the Institute for Professional

Development (IPD), M-Vector, implementers and donors, revision of the SOW and

budget based upon findings, and planning for an August-September instrument

adaptation, assessor training and data collection; and drafting of the West Bank: Early

Grade Reading Assessment report and a Policy Dialogue workshop in Ramallah with

USAID, MOE counterparts and stakeholders.

Fourth quarter activities included continued technical support to the USAID Missions in

West Bank, Bangladesh, Tajikistan (Gorno-Badakshan [GBAO] region), Morocco and

the Philippines for Early Grade Reading Assessments (EGRAs) and the completion of

some activities. DEP-AME translated into Arabic the Topical Analysis: Early Grade

Reading Instruction in Arabic, and continued robust activity on the EGRA Barometer

pilot, including a focus on data integration, tracking barometer use, and design of

country-level piloting. Design and refinement of the Evaluation for Education



Programming training course for USAID Education Officers continued, including the

identification of a replacement trainer due to scheduling conflicts with the original

candidate.

Result Area Support by Quarter

Quarter 1: October 1–December 31, 2013—DEP-AME activities during the

first quarter included assistance to the USAID Missions in Bangladesh, Morocco,

Nepal, the Philippines, Tajikistan (GBAO) and the West Bank, as well as analytic

research and training development.

Result 1 activity included the launch of the USAID/West Bank EGRA with

launch meetings in Ramallah, and the completion of the scope of work for the

Philippines: Analytic Support Services for Early Grade Reading. Data from the

latter, designed to build local capacity on mother tongue early grade reading

assessments, was presented at the November 2013 All Children Learning

Workshop in Rabat. In Nepal, adaptation of the Education Management

Efficiency Study (EMES) and Teacher Observation Study (TOS) instruments took

place, along with training of the EMES/TOS assessors for data collection.

Planning for the Bangladesh: Institutional Assessment of Early Grade Reading

also launched.

Result 2 activities supported research and drafting of a Topical Analysis: Early

Grade Reading Instruction in Arabic. This too was presented at the All Children

Learning Workshop in Rabat, Morocco. DEP-AME research supported

preparation of a Topical Analysis on Scale-up and Sustainability.

Result 3 activity included acceleration of the analytic design of the Beta version

leading to the demonstration of the Reading Improvement Modeling Tool (EGRA

Barometer). Planning for the launch of Phase 2 piloting of the EGRA Barometer

occurred, along with integration of data from selected EGRA programs.

Result 4, activity focused on the design, subcontracting, and planning for an

anticipated February 2014 training workshop entitled Evaluation for Education

Programming for USAID Education Officers. This was subsequently delayed

until November 2014.

Quarter 2: January 1–March 30, 2014—During the second quarter, multiple

activities were underway, including field-based Early Grade Reading

Assessments (EGRAs) and launches of institutional assessments and reading

research studies.

Result 1 activity continued support to the USAID Missions in West Bank,

Philippines, and Nepal, as well as the addition of new activities in Bangladesh,

Tajikistan (Gorno-Badakhshan Autonomous Province [GBAO] region) and

Morocco.

http://en.wikipedia.org/wiki/Gorno-Badakhshan_Autonomous_Province



Result 2 activity finalized the Topical Analysis: Early Grade Reading Instruction

in Arabic, complete with all Annexes, as well as a draft of the Topical Analysis:

Scale up and Sustainability.

Result 3 activity focused on the Beta version of the reading improvement

modeling tool launch, the EGRA Barometer, and the refinement of a Phase 2 plan

to pilot the tool in two countries, to integrate country EGRA data, and to

continually refine functionality.

Result 4 activities included dialogue with the subject matter expert on the

Evaluation for Education Programming training workshop planned for mid-late

2014). Refinements to the Analytic Agenda and workshop design occurred.

Quarter 3: April 1–June 30, 2014—Activities during the third quarter included

assistance, for analytic support, EGRA instrument adaptation, assessor training

and data collection; and Policy Dialogue workshops.

Result 1 activity support was provided to the USAID Missions in West Bank,

Bangladesh, Tajikistan (Gorno-Badakshan [GBAO] region), Morocco and the

Philippines for EGRAs, continued, and completion of some activities.

Result 2 activity completed the translation into Arabic of the Topical Analysis:

Early Grade Reading Instruction in Arabic.

Result 3 activity continued a strong emphasis on the EGRA Barometer pilot,

including a focus on data integration, tracking barometer use, and design of

country-level piloting.

Result 4 activity focused on the continued design and refinement of the

Evaluation for Education Programming training course for USAID Education

Officers and the identification of a new subject matter expert due to scheduling

conflicts with the original trainer.

Quarter 4: July 1-September 30, 2014: During the fourth quarter of Fiscal Year 2014

activities largely focused on country-level support and piloting of the EGRA Barometer.

Result 1 activity included support to the USAID Missions in West Bank,

Bangladesh, Tajikistan (Gorno-Badakshan [GBAO] region), Morocco and the

Philippines for Early Grade Reading Assessments (EGRAs), continued, and

completion of some activities.

Result 2 activity was limited this quarter.

Result 3 activity focused on the EGRA Barometer pilot, including a focus on data

integration, tracking barometer use, and design of country-level piloting.

Result 4 activity focused on the continued design and refinement of the

Evaluation for Education Programming training course for USAID Education

Officers and the identification of a new subject matter expert due to scheduling

conflicts with the original trainer.



Operational Activities

No significant operational activities or changes have occurred.

USAID Communication and Meetings

Communications and Meeting: Regular meetings, phone calls, and written

communication occurred between RTI and USAID’S Senior Education Advisor Mitch

Kirby and Education Development Officer Chris Capacci-Carneal, as well as with

subcontractor Allegro Consulting. Social Impact, and varied consultants. DEP-AME staff

also participated in meetings where the EGRA Barometer was presented and showcased,

and in the November 2014 All Children Learning Workshop in Rabat, Morocco. These

are referenced below.

Reports and Deliverables

Documentation and Reporting: The Fiscal Year 2014 Financial Summary is included at

the end of this report. It is inclusive of the obligated amount to date; total expenditure,

outstanding commitments, total expended to date, balance remaining of obligated funds,

and the baling remaining of the total estimated cost. Final reports, proposals or

deliverables submitted to USAID over the course of this year have been submitted

previously under separate cover. They are listed below for reference.

Quarter 1—Reports and Deliverables:

Financial Summary

Result 1: GBAO Trip Report

Result 1: DEP-AME Morocco Component 2

Result 1: EGR Assessment Bangladesh Report

Result 1: West Bank Grade 2 Results Presentation

Result 2: Topical Analysis of Arabic Reading Instruction 4 April 2014

Result 3: Early Grade Reading Barometer Usage Reports (April-June)

Result 4: Impact Evaluation Training Annotated Agenda


Financial Summary

Result 1: Philippines: Analytic Support Services for Early Grade Reading

Result 1: Nepal: EGRA, TOS, and EMES

Result 1: Bangladesh: Reading/Education Sector Support/Assessment

Result 1: West Bank: Early Grade Reading Assessment



Result 2: Draft 4 Topical Analysis: Scale-up and Sustainability

Result 2: Topical Analysis: Early Grade Reading Instruction in Arabic


Financial Summary

Result 1: GBAO Trip Report and Presentation in Tajik

Result 1: DEP-AME Morocco Component 2


Result 1: Nepal AME Trip Report and EMES Report

Result 1: West Bank Grade 2 Results Presentation (and Arabic)

Result 1: Philippines Trip Report

Result 2: Topical Analysis Early Grade Reading—Updated report

Result 3: Early Grade Reading Barometer Usage Reports (April-June)

Result 4: Castleman - USAID training Annotated Agenda


Financial Summary

Result 1: PhilED Data January Trip Report

Result 1: West Bank EGRA Memorandum of Understanding

Result 1: GBAO Proposal


Result 1: Nepal AME Trip Report (November, submitted January)

Result 3: Early Grade Reading Barometer Overview

Result 4: Impact Evaluation Training Annotated Agenda

The table of Overall Progress: Results, Requirements, Standards in the next section

serves as a complete summary of DEP- AME activities planned, underway or completed

since inception.



Philippines: Analytic Support Services for Early Grade Reading Training

Overall Progress: Results, Requirements, Standards

This table captures all DEP-AME activities and status to date at the close of this reporting

period.


Requirement 1.1: Determine Feasibility of Mission-Specific Requests (Data Support) Status

Nepal: Early Grade Reading Institutional Assessment Completed

Burma: Education Sector Desk Review Completed

MENA: Region Cost Comparisons of Early Grade Reading Completed

Egypt: Early Grade Reading Case Study Completed

Egypt: Grade 3 Early Grade Reading Assessment Baseline Completed

Philippines: Analytic Support Services for Early Grade Reading Completed

Bangladesh: Institutional Assessment of Early Grade Reading Underway

West Bank: Early Grade Reading Assessment Underway

Nepal: EGRA, Teacher Observation Study, Management Effectiveness Studies Completed

Morocco: Reading Research Studies Underway




Tajikistan (GBAO) Early Grade Reading Assessment Underway

Requirement 1.2: Develop Plans for Country-Specific Data Support Status











Requirement 1.3: Adapt or Develop Country-Specific Education Tools Status











Requirement 1.4: Produce Reports on Country-Specific Support Status






MENA: Region Cost Comparisons of Early Grade Reading Completed










Requirement 2.1: Develop AME Regional and Country Education Trends Analyses Status

Research Conducted and Data Compiled Completed

Graphic Presentations Designed Completed

Literature Review on Early Grade Reading and MTI Trends Completed

Regional report with Country Data Submitted on MENA Region Completed

Regional report with Country Data Submitted on EA Region Completed

Regional report with Country Data Submitted for CA Region Completed

Requirement 2.2: Conduct Identification and Analyses of U.S. and International Evidence-Based Strategy-Related Education Programs and Interventions Status

Potential Themes Identified Completed

Review and Analysis on First Theme (Reading) Completed

Literature Review of the Scale-up and Sustainability Completed




Requirement 2.3: Conduct AME Workshops/Seminars Status

Training Summer 2013 in Washington DC Cancelled

Training Impact Evaluation, November 2014 (Bangkok, Thailand) Underway

Result 3: Measurement Tool with Applicability Across Countries Developed

Requirement 3.1: Identify Areas Where Measurement Tools Are Needed Status

Concept Notes Drafted for USAID Review Completed

Concept Note Review by USAID Completed

Refine Concept Note Completed

Requirement 3.2: Develop the Selected Tool Status

Reading Achievement Data and Set Standards Completed

Generate Learning Outcome Projections Completed

Tool 1: Use data to: understand existing reading achievement levels, factors that influence or constrain performance; set reasonable standards for reading performance, consider the scope for and likely trajectory of improvement

Ongoing

Tool 2: Use data to: generate projections for how reading interventions could impact learning outcomes on an increasing scale; examine relationship between the variation in the quality and fidelity of implementation and variation in learning outcomes achieved; and model likely impacts of national programs at scale

Ongoing

Tool 3: Use data to: measure impact of reading programs by comparing treatment and control schools and investigate the variations within treatment schools; evaluate cost- effectiveness of an intervention; help improve implementation; re-examine initial assumptions on standards for reading performance; and inform policy decision-making and management of national scale programs

Ongoing

Requirement 3.3: Pilot the Tool Status

Review Variations in Quality and Fidelity Pending

Measure Treatment and Control Schools Pending

Evaluate Cost-Effectiveness and Improve Implementation Pending

Policy Dialogue Pending



Result 4: Education Personnel Knowledge of Selected Education and Evaluation Topics and Processes Strengthened

Requirement 4.1: Identify Education Strategy Goal 1 Topics for Classroom Training, Online Training, Online Information/Resource Material, or Future Training Status

Survey Topics Identified Completed

Training Plan Goal 1 Completed







Requirement 4.4: Conduct a Training Needs/Preference Survey of Education Personnel in Asia and Middle East Regions Status

Conduct Survey and Analysis Completed

Requirement 4.5: Produce a Training Plan Status

Full Training Plan Completed

Requirement 4.6: Develop Goal 1 Training Modules or Courses for Classroom and Online Training Status

Research Goal 1 Impact Evaluation Completed

Design Training Modules On-line Course Completed

Requirement 4.7: Develop Goal 2 Training Modules or Courses for Classroom, Online, and/or Resource Materials Status

Rolled into Goal 1 Impact Evaluation above in 4.6 NA

Requirement 4.8: Develop Goal 3 Training Modules or Courses for Classroom and Online Training Status

Rolled into Goal 1 Impact Evaluation above in 4.6 NA

Requirement 4.9: Deliver Classroom Training for Goals 1, 2, and 3 Status

Research Completed Completed



Result 4: Education Personnel Knowledge of Selected Education and Evaluation Topics and Processes Strengthened

Course Designed Underway

Training Summer 2013 in Washington DC Cancelled

Training Planned November 2014 (Bangkok, Thailand) Pending

Requirement 4.10: Deliver Online Training Modules or Courses Status

Research Completed Completed

Course Designed Completed

Training On-Line Completed

Requirement 4.11: Package Informational/Resource Material for Online Access Status

Design Completed Completed

Recording and Packaging Completed

Transfer to USAID University Completed



Progress toward Objectives (by Result)


Overview of Activities and Achievements

DEP-AME activities this year focused on six USAID Mission analytic activities under

Result 1.

Bangladesh Institutional Assessment of Early Grade Reading

In May 2013 USAID/Bangladesh asked RTI to assist with analyzing programs, policies,

practices, and institutional capabilities for improving early grade reading in Bangladesh.

The resulting Institutional Assessment of Early Grade Reading, launched this fiscal year

after delays due to security concerns. Once fieldwork was approved, DEP-AME

mobilized the team in January 2014 to Dhaka to conduct the institutional assessment of

the early grade reading sector.

The Institutional Assessment of Early Grade Reading served as a rapid education sector

assessment and related desk study. It examined major aspects of the education sector

(government and nongovernment), and relevant elements of the Government of

Bangladesh (GOB) outside the education sector that impact upon early grade reading

either directly or indirectly. The objective was to identify specific strengths, weaknesses,

and leverage points, and to offer a set of recommendations and an overarching conclusion

regarding the Bangladeshi education sector’s potential to scale-up and sustain an early

grade reading (EGR) program.

The assessment was conducted by a team of researchers formed by RTI. The team was

comprised of Dr. Colette Chabbott (Team Leader), Dr. F. Henry Healey (Lead Writer),

Dr. Shahnaz Karim (Technical Expert on education governance and related issues), and

Dr. Nazmul Haq (Technical Expert on curriculum and teacher training). The assessment

involved both a desk study and a field visit during which a number of guided interviews

of key informants took place over the calendar period January 26–February 14, 2014. The

report reflected the findings of the assessment. The desk study itself was largely

undertaken before the field study took place.

Through the field study, the team interviewed key officials in the Ministry of Primary and

Mass Education (MOPME), the Ministry of Education (MOE), and their central and

lower level departments, offices, institutes, and centers. The team also visited schools

where they had an opportunity to talk with teachers, Head Teachers, and members of the

school management committee (SMC). Additionally, a focus group session involving

eight teachers from various schools in Dhaka was conducted. The field study also

allowed the team to meet with people from a number of non-government organizations

(NGO), international NGOs, research centers (RC), and development partners (DP).



To ensure that the information presented in this report was as accurate as possible,

measures were taken to cross check the information gathered from one informant

by asking for the same information from a number of other relevant informants.

When a number of different informants provided the same answer, the team felt

confident that the information fairly accurate, realizing of course that in most

instances our sample size was exceedingly small. In those instances where the

team was unable to find a clear answer, it is noted in the report. Results of the

assessment were summarized and analyzed in consultation with the

USAID/Bangladesh team and USAID Asia and Middle East Bureau Senior

Education Advisor Mitch Kirby. The draft report was submitted to USAID in

March for comment along with a summary brief for external audiences. A revised

version was submitted later in the year.

Next Steps

Pending revisions from USAID/Bangladesh Mission, the report will be finalized.

This will conclude this activity in the coming year.

Morocco: Reading Research Studies

DEP-AME began work on the Morocco: Reading Research Studies series this year,

submitting a proposal to USAID in March, as requested, for funding three major studies

on reading to provide the Ministry of Education and Vocational Training the needed

evidence for its upcoming reform strategy. Within weeks, DEP-AME mobilized to

conduct the research for what are the first three of an anticipated five Reading Research

Studies. Field activities drew on the technical expertise of the VarlyProject and other

experts to carry out the research.

The instrument design was conducted in April and fieldwork training, instrument piloting

and fieldwork preparation occurred during April and May. Data cleaning and analysis

occurred in May and report writing began in June. Translation of the final deliverables

occurred in July and the instruments, survey documents and final report were

subsequently submitted, concluding Phase 1. Translation of the first three deliverables

has been completed and submitted, including instruments, survey documents and the

Final Report. Dissemination workshops on the results findings were also conducted by

DEP-AME and included outreach to other implementers to facilitate wide access to the

findings. The three studies, conducted by RTI, covered the following topics:

Curriculum and Textbook Analysis: This study reviewed the official

curriculum and a selection of textbooks to determine whether they reflect

international best practices in the teaching of Arabic, and whether they provide

the framework for improving reading instruction in the early grades. The study

looked at the gaps between the official curriculum and evidence-based practices

in teaching Arabic reading. The analysis revealed the need to improve the

presentation and pedagogical quality of primary school textbooks. The study



showed that certain key competencies in reading acquisition—for example,

phonemic awareness, letter-sound identification and incremental decoding—were

not addressed sufficiently in the textbooks before requiring children to do more

complex tasks like reading phrases and understanding of texts. Monitoring of

student learning focused mostly on recitation and oral comprehension skills, and

much less on reading activities that require a significant linguistic and cognitive

effort such as decoding and free writing. Teacher guides were not formally

analyzed in this study; however, they were used as reference documents to the

textbooks to see what the teacher guides were recommending the teacher to do

with the textbook.

Pre-Service Teacher Education Analysis: The objective of this study was to

look at how teachers in training learn to teach reading in the early grades. The key

research questions revolved around whether reading is taught as something

explicit or implicit in the teacher preparation curriculum, and whether the

methods promoted in teacher training reflect best practices in the teaching of

Arabic to early readers. Several factors limit the effectiveness of pre-service

teacher training to prepare teachers to teach Arabic reading in the early grades.

The main factor is the limited amount of time required for teacher certification in

the CRMEF (about 9 months). There is not sufficient time dedicated to providing

in-depth strategies for teaching a specific subject area like reading alongside all of

the other general professional knowledge and skills required by the curriculum

(lesson planning, classroom management, etc.). The one-year program does

provide a range of courses, but not to the depth that a full three- or four-year

program would. This is compounded by the fact that teachers in training can come

from undergraduate programs in any subject area, and therefore do not always

have pre-requisite content knowledge in the subject to be taught. Finally, the

study found an important gap between theoretical knowledge and practical skills,

which needs to be addressed in order to better prepare teachers for their practice.

Teacher Attitudes, Beliefs, and Practices: This study examined the perceptions

of teachers towards the teaching of reading in Arabic and the way these

perceptions influence practice. The principal research questions were: What

beliefs have the most influence on the teaching methods used in early reading

lessons? How do these perceptions impact the professional growth of teachers?

How to teachers react to questions related to language of instruction? Are they

receptive to using local dialects to support teaching reading in Arabic?

The analysis found that teachers express a relatively optimistic view of their motivation

to teach in primary grades and to teach reading in Arabic. However, many factors make it

difficult for them. First, they do not feel they have the resources, in general, to teach in

crowded or multi-grade classrooms, regardless of pedagogical expertise. Next, there is no

consensus on the best method to use for reading instruction; some teachers maintain the

government’s recommended whole language method, while others opt for a mixed or



“syllabic” approach to teaching reading. Very few are aware of recent evidence of best

practice in teaching reading in Arabic in the early grades (for example, use of

morphology and a phonetic approach), but they do recognize that the current textbooks

are not aligned with the ability of the children. Teachers find themselves adapting on

their own to the sociolinguistic environment and to teaching materials that are too

difficult for most students especially those with no preschool education.

Phase 2 Planning and design for two remaining Reading Research Studies is underway

for the coming fiscal year. These are slated to be completed around March or April 2015.

Next Steps

Launch Phase 2 Reading Research Studies’ activities in the next reporting period.

Nepal: Early Grade Reading Assessment (EGRA), Education Management Efficiency Study and Teacher Observation Study (EMES-TOS)

The Nepal Early Grade Reading Assessment (EGRA), Education Management Efficiency

Study and Teacher Observation Study (EMES-TOS) was launched in the prior reporting

year and concluded its activity this Fiscal Year. An initial delay arose due to election-

related security concerns which necessitated a scheduling change. Once mobilized, the

team moved rapidly to launch this three-part analytic study, intended to strengthen local

skills in design, evaluation, and management of education programs, and quality data

capture and analysis to support them.

On October 1, final approval was received from USAID for the study. Three weeks later

DEP-AME was informed of a delay to the anticipated November 10-14 workshop. This

was then rescheduled until after the election. An updated schedule for the Nepal activities

was provided to USAID and field work commenced.

In November DEP-AME staff traveled to Nepal to facilitate the adaptation of the

EMES/TOS instruments and train the assessors. The full field data collection of the

EMES/TOS was completed by December 31. During the second quarter DEP-AME

completed the field data collection for the EMES, TOS and EGRA and a draft report was

submitted for USAID comment.

The focus of the EGRA study provided a nationally representative assessment of Grade

(G) 2 and Grade 3 students in Nepal public schools. In January/February of 2014, the

EGRA study was conducted, which included an EGRA for Grades 2 and 3 students in the

Nepali language, a student interview, and a teacher passage about the upcoming Early

Grade Reading Program. The study was designed to provide a useful baseline or

benchmark by which the effectiveness of the upcoming national reading program could

be measured. The report provided key results from a national study measuring the reading

skills of students in Grades 2 and 3 in Nepali across all ecobelts.

The full field data collection of the EMES/TOS was completed at the end of December.

RTI and its local partner New ERA, began the process of entering and cleaning the data



during the next quarter. RTI/New ERA visited 104 community schools across 13

districts, interviewing at least one head teachers, one Grade 2 teacher, one SMC member,

and observing one Grade 2 classroom in each school. In addition, 13 District Education

Officers (DEOs), 13 Regional Centers (RCs) and 6 ETCs were interviewed.

Informational interviews were also conducted with the Ministry of Education and it’s

Central Level Agencies (CLAs).

During the November trip, several important agreements were reached with respect to the

scope of the upcoming EGRA, namely that Grade 2 and Grade 3 learners were to be

assessed in Nepali language only; community (public) schools were to be included in the

formal study; and a small sample of private schools would be included in an annex to the

report to provide anecdotal information to the Ministry. To the extent possible, Ministry

officers from the ETCs would also be included as field assessors or supervisors. It was

agreed with the Ministry that the adaptation and training of the EGRA would take in

January and pilot data collection the first week of February. This was followed by the full

field data collection in February and March. Four key research questions were addressed

in the Final Report.

Research question 1 asked, to what extent are students in Grade 2 and 3 learning to read

in Nepali? In general, students were demonstrating emergent reading skills by Grade 3.

There were increases in mean scores between Grades 2 and 3 on all subtasks and in all

ecobelts. Girls and boys performed equally well across all subtasks, showing no

significant gender disparity in teaching and learning in typical Nepali classrooms in the

early grades. However, the mean scores achieved in Grade 3 are more indicative of

emergent reading ability rather than the more secure level of skill one would expect after

three years of schooling. The levels of reading ability this EGRA revealed in Grade 3

were what should ideally be seen at the end of Grade 1 or beginning of Grade 2. These

results suggest that more focused instruction is needed early in the primary cycle to more

quickly build literacy-related skills and reading ability. The answers to the second

research question shed additional light on this need.

Research question 2 asked, what reading-related skills are students in Grades 2 and 3

acquiring? Students in Grades 2 and 3 performed well on the letter sound knowledge

subtask, which means they were acquiring the understanding of the alphabetic principle

that is the foundation for reading. However, student performance in both grades on matra

and nonwords subtasks was lower than expected for the grade level. In addition, there

remained large percentages of students, especially in Grade 2, that were scoring zero on

the matra and nonword subtask. Poor performance in these skill areas usually indicates

that instruction does not sufficiently emphasize syllable (matra) reading or decoding

(nonword reading). In light of the results for the first two research questions, effort might

best be directed towards all students in Grade 1, as well as struggling students in Grades

2 and 3. The results show that once students gain the foundational skills needed to begin

reading, general achievement rises. Instruction in Grades 1 and 2 should target these

basic skills, with particular attention paid to learning the matras and learning how to



decode unfamiliar words. Students may benefit from explicit instruction in the sounds of

the matras, using similar strategies that have been used to teach the letter sounds, which,

results indicate, have been largely successful. In addition, students should receive

instruction in comprehension strategies beginning in Grade 1 through oral comprehension

stories.

There are large disparities in student achievement by ecobelt, with Grade 3 students in

the Terai region performing at levels similar to Grade 2 students in the Mountain, Hill,

and Kathmandu Valley ecobelts. For students in the Terai ecobelt, home language was an

important factor that may have held back their acquisition of literacy in Nepali. The

majority of students in the Terai ecobelt reported speaking a language other than Nepali.

Those students read fewer words correctly per minute of the reading passage and

answered fewer questions correctly on the listening and reading comprehension subtasks.

This suggested that their lack of familiarity with the Nepali language was not adequately

addressed during the first few years of primary school. Although not the only factor in the

different scores across ecobelts, home language is an important consideration for the

Terai ecobelt. Given the differences in results, extra support and attention should be paid

to students in the Terai ecobelt. In particular, students’ oral language skills in Nepali must

be strengthened early in their school career so they can learn to read fluently and with

comprehension in Nepali. Strategies that explicitly help children bridge from their

maternal languages to Nepali may be needed.

Research question 3 explored the factors, both in-school and out-of-school, that help

explain student performance on the EGRA. As mentioned, an important out-of-school

factor is where students reside/go to school, with Terai ecobelt demonstrating lower

scores across all subtasks and grades. Linked to ecobelt residency is the other issue

already mentioned—home language. While this may be of particular concern in Terai, in

general students whose first language was not Nepali were not performing as well as

those who came to school speaking the language of instruction. The other out-of-school

factor that correlates with reading performance is the relative socio-economic status

(SES) of a child’s family. Grade 2 students in the lowest socioeconomic quartile had a

mean oral reading fluency score equivalent to half of the mean score of those in the

highest quartile. The difference in those mean oral reading fluency scores was lower in

Grade 3. While it is a good sign that more time in school was helping students from less

wealthy families begin to catch up to better off students, schools may need to recognize

the needs of disadvantaged students earlier and intervene to help address challenges those

students may face.

While the gap in reading scores between students from the lowest and highest SES

quartiles diminished between Grades 2 and 3, the gap between Nepali and non-Nepali

speakers seemed more persistent. With that in mind, in addition to intervening early or

providing instruction in mother tongue as mentioned above, it may be worthwhile to offer

specific additional learning opportunities to students who continue to struggle with oral



comprehension of Nepali. Several in-school factors also demonstrated significant positive

correlations with reading performance.

These included aspects of teachers’ reading instructional practices, schools’ practices that

support reading remediation, parent-teacher association (PTA) support for reading, and

the availability of teaching and learning materials. These findings suggest that certain

aspects of school management, instruction, and school support from communities are

important factors to address when rolling out an Early Grade Reading Program. For

example, classrooms in which students were frequently asked to read aloud had a mean

oral reading fluency score greater than the mean of students who were not asked to read

aloud. In general, teachers who adhered to instructional practices that align more with

recognized best practice and research on literacy instruction had students who were

performing better. Getting such practices to become widespread is one fundamental

challenge that a national reading program will need to confront. Another in-school

indicator, teachers’ having reference materials or teachers’ guides, correlates with

students having mean oral reading fluency scores higher than students whose teachers did

not. This may reflect a more general relationship between availability of materials and

student performance in reading. Further research could explore the types of materials that

contribute most to improved reading outcomes. Nevertheless, it is safe to say that

increasing the availability of instructional resources and reading material should be a

cornerstone of a national reading program.

Schools that had staffs that recognized that attention must be focused on struggling

students and that in general put more effort into supporting remediation also had higher

mean reading fluency scores than those that did not. And when communities contributed

to efforts to improve reading, the students in those schools performed better. These two

findings suggest that building up greater awareness of the need to focus on assisting

struggling students and mobilizing and supporting local initiatives related to reading

improvement could be useful features of a national reading program. Another in-school

factor concerns what teachers thought and believed about several aspects of the teaching

and learning of reading. Teacher attitudes and beliefs were measured, and results revealed

patterns of beliefs amongst teachers with regard to mother-tongue instruction, reading

instructional practices, student ability, and student engagement. In general, most teacher

attitudes and beliefs fell in the middle of the ranges of the continua used in this survey.

This suggests that for most teachers, attitudes and beliefs reflected practices that will

support reading instruction—support for mother tongue, active learning, and best

practices for literacy acquisition. However, some teachers held attitudes and beliefs that

run counter to what will be needed for a national reading program to succeed.

Research question 4 asked how well teachers understand a written explanation of a

proposed early grade reading program. Results showed that a majority of teachers were

able to read the passage and successfully answer most of the questions asked about it.

This shows that most teachers had reading and comprehension abilities that would enable

them to process information written at a secondary school level. Yet, a small minority of



teachers seemed to need help processing overly complex written information. In general

however, the results pointed to the promise of designing written materials and

professional development courses to introduce new instructional strategies for the

teaching of reading, as teachers already are able to read well with comprehension. Also,

materials in which teachers were asked to read and implement a lesson can be designed

with confidence in teachers’ ability to comprehend the instructions and the lessons.

The final EMES report provided data on specific education management capacities and

teaching practices that will affect the Ministry of Education’s (MOE’s) ability to

implement and sustain a national EGRP at scale. In so doing, it examined the variety of

system structures and attributes that lend themselves to EGRP implementation, identify

existing gaps and capacities to be addressed, and highlight the strengths and opportunities

that could be leveraged by the Ministry and its financial and technical partners.

The EMES-TOS provided a description of the current instructional practices and

institutional and management capacity requirements to support a coherent national early

grade reading program. Each of these assessments and studies conducted was intended to

provide a useful baseline or benchmark by which the effectiveness of an upcoming

national reading program. The three main deliverables included:

EGRA report that contains analysis of teacher assessment and learner

performance data, disaggregated by region and ecobelt.

A brief anecdotal report on private schools’ performance viz EGRA and

EMES/TOS.

A separate report on the study testing print vs. electronic methods.

These reports presented data that addressed each of the research questions and used those

data to reach summary conclusions about the extent to which the education system in

Nepal is prepared to support implementation of a national EGRP. The Final Report was

submitted to USAID following the Policy Dialogue Workshop in June. This concluded

the activity.

Next Steps

None, this activity is completed.

Philippines: Analytic Support Services for Early Grade Reading

Under DEP-AME, the 2014 Philippines EGRA study (PhilEd Data II) was conducted to

expand the knowledge base for the Mother Tongue Based-Multilingual Education (MTB-

MLE) program by conducting a comparable exercise in three additional mother-tongue

languages—Cebuano, Hiligaynon, and Maguindanaoan. Another representative set of

data was collected for Ilokano, which provided additional information on whether quality

of MT instruction is improving with implementation experience, and how much children

improve their reading skills from Grade 1 to Grade 2. The 2014 study used the same

EGRA instrument as 2013—with appropriate adaptions—to observe and measure reading



skills in five key areas: (1) Alphabetic principle; (2) Phonemic awareness; (3)

Vocabulary; (4) Fluency; and (5) Comprehension. The field work included a local survey

research partner, TNS Global.

Finalization of the scope of work for the Philippines: Analytic Support Services for Early

Grade Reading occurred in October and included activities to build on earlier work

supported under the Philippines Ed Data task order: This included:

Conducting a sample based EGRA in 4 mother tongue languages. Suggested

languages included: Ilokano, Hiligaynon, Cebuano, and Maguindanao. Final

determination of the languages was made in consultation with

USAID/Philippines.

Data were also collected and analyzed in the regions where the mother tongue

assessments were conducted on the instructional practices and use of language in

early grade classrooms (including information on teacher facility in different

languages).

DepEd capacity was built to use Tangerine® and to manage the data generated by

Tangerine-based implementations of EGRA.

This activity began at the start of the Fiscal Year immediately after final agreement with

the Mission on the scope of work. RTI established a subcontract with Taylor Nelson

Sofres (TNS), a local NGO, to support the selection and training of assessors and the

management of all field data collection.

In November, RTI facilitated an instrument adaptation workshop during which DepED

officials, local linguists and curriculum experts worked to develop the survey instruments

for each of the four target languages: Ilokano, Hiligaynon, Cebuano, and Maguindanao.

The teacher survey and classroom observation instruments were developed in conjunction

with DepED colleagues. During December all the instruments were finalized and

rendered into electronic format using the Tangerine® open source software. All data

collection was carried out using tablet computers. Survey teams included TNS staff and

locally recruited teachers and other DepED officials. In December, TNS began

identifying and recruiting assessors, and DepED designated the department staff that

would participate in each region. The teacher survey and classroom observation

instruments were developed in conjunction with DepED colleagues.

The results provided baseline data on Grade 3 students’ reading achievement and

language readiness under the previous (prior to adopting mother tongue-based

instruction) national curriculum. PhilEd Data I also explored the relationship of these

skills to selected factors in students’ school and home environment. This evidence can be

used to inform policy and the implementation of appropriate interventions in schools and

classrooms across the country, as well as to monitor the effect of MTB-MLE in years to

come. The regional language EGRA for Ilokano was the first rigorous and large-scale

assessment of student learning in one of the mother-tongue languages.



One of the key takeaways from the study is the existence of important inequities in

achievement between boys and girls, and across regions. The challenges in the lowest-

performing regions are well-known—instability, poverty, geography, lack of access to

early childhood education, etc.—and many programs are in place to try to address these

factors. However, understanding which factors are having the most influence and what

specifically can be done most effectively and efficiently to address the achievement gaps

will be critical going forward.

The most prevalent sign of MTB-MLE being implemented in the four regions included in

this study was the consistent use of the regional mother tongues as the languages of

instruction in both reading and other lesson periods. All teachers observed were using the

mother tongue for the vast majority of their instructional activities and interactions with

students. Students were also observed to be engaged in the activities being conducted in

mother tongue. Also, Filipino and English are being taught as expected, and children are

acquiring beginning skills according to the progression expected by the curriculum.

However teachers were not exhibiting the full set of skills needed to appropriately

support literacy acquisition in the mother tongue during their reading lessons.

Many teachers indicated that they had not been sufficiently trained for mother-tongue

instruction, and more than 70% stated that they did not have sufficient materials and

resources for teaching reading. Majorities of teachers reported not having important

materials like the mother-tongue teachers’ guides in three of the four regions. A particular

concern is whether the activity of reading practice is taking up a sufficient amount of

time during reading lessons, or if teachers are disproportionally speaking and asking

children to read or recite in chorus. There were also very different expectations of when

children should be acquiring reading skills in different languages (or possibly just

different expectations of what “fluently” means).

For the vast majority of students in the regions included in this study, the language of

instruction matched their home language. However there were students who did not

speak the mother tongue of instruction at home, especially in Region 1 where as many as

12% of the students were in a school that was using a language of instruction that was not

their mother tongue. Children’s self-reports of language use in the home may be

somewhat unreliable, and assessors noted in some circumstances children were

unfamiliar with the terms ‘language’ and so additional prompting needed to be done to

make sure the children were answering accurately. This is an important finding in itself—

how are children being prepared for instruction in multiple languages, and how are

teachers bridging between the first and second (and third) languages if children are not

exposed to the meta-language associated with code switching in the classroom, the

community and the home? What are children learning about being multilingual?

Another somewhat accidental finding was that teachers have been hearing about EGRA

through a variety of different sources including K–12 workshops, Basa Pilipinas, and

different regional initiatives. In their responses it became clear that were different

understandings about what EGRA is and what it is supposed to do. In some cases the



term “EGRA” is confused with the notion of teaching reading in the mother tongue rather

than as a specific type of reading assessment tool.

As this was still early in the process of rolling out a complex reform, it is understandable

that the resources, supports, and practices for MTB-MLE were not yet all in place.

Notably, some important aspects of MTB-MLE implementation show strong

relationships with student performance in reading. For example, the 20% of students who

often used their mother-tongue book for reading were 2.8 times as likely as those who

never used it to be in the top 25% of reading performance. Similarly, the 24% of students

whose teachers often asked them to read in the mother tongue were more than three times

as likely to be in the top 25%. This indicates that if MTB-MLE implementation

conditions could be better assured across a much broader range of schools, then it is very

likely that reading performance will improve.

The uneven implementation that these data show helps explain the types of outcomes

presented in this report. According to the MTB-MLE expectations, children should be

learning to read in their mother tongues by the end of Grade 1. A large proportion of

children learning in Maguindanaoan were scoring zero on all the EGRA subtests, except

listening comprehension in Maguindanaoan. This indicates that many children were not

acquiring even the building blocks of literacy in that language during Grade 1. Ilokano

and Hiligaynon also had high proportions of children in Grade 1 with zero scores on

several subtests.

Providing this support to DepEd reinforced their capacity for nation-wide use of the

EMIS Toolbox to support generation of Basic Education Report Cards (BERCs). DEP-

AME completed implementation and concluded the activity with analysis of data from

February 2014 data collection in four regions. The Final Report was drafted and

submitted to USAID following a Policy Dialogue Workshop which was held in August.

This concluded the activity.

Next Steps

None, activity completed.

Tajikistan (GBAO) ‒ Early Grade Reading Assessment

DEP-AME received a Concept Note in January from the USAID/CAR Mission

requesting support for an EGRA to be conducted in the Gorno-Badakshan Autonomous

Oblast (GBAO) region of Tajikistan. In March, a proposal was submitted to USAID and

it was approved. Shortly thereafter, Dr. Marcia Davidson was identified to serve as the

reading expert to accompany Dr. Hank Healey on the first field visit to launch the activity

in GBAO in May 2014. Local partners were identified and dialogue, planning and

subcontracting procedures began with M-Vector as the local training and data-collection

subcontractor. The ultimate goal is to launch an EGRA exercise and to obtain a first-hand

understanding of the existing early grade reading (EGR) situation in GBAO.



Field implementation launched in May 2014 with a field visit by Drs. Hank Healey and

Marcia Davidson to GBAO plan for the anticipated Fall EGRA administration. During

this launch visit, the team met with key Ministry of Education (MOE), GBAO

Department of Education (DOE), USAID, and relevant other stakeholders, including RTI

local partner M-Vector.

The team identified and began to gather the information needed to conduct the adaptation

workshop and EGRA exercise which was scheduled to occur in September/October 2014.

This information included the primary-level reading curricula and curricula materials,

existing EGRAs that have been (in 2011) or will be (via the Quality Reading Project in

2014) conducted in Tajikistan, school data, and general knowledge gained from

conversations with key people in GBAO and Dushanbe. The team was impressed with

the number of books and learning materials available in classrooms. Teachers seemed

generally quite motivated and there was no indication of unmotivated students.

There was some indication (including teacher comments at presentation session) that

children do not learn phonology and how that knowledge of letter sounds can be integral

to a strong reading foundation. While higher level thinking skills are very much valued

and strived for in the visited schools, many of the activities observed focused on review

and examples of content previously learned. There was no evidence of procedural

knowledge being taught as it might relate to problem solving or analyzing text.

In late August, the DEP-AME team was required to cancel the planned travel and data

field collection set for September/October due to a change in Tajikistan/GBAO policy on

the issuance of travel visas. In addition, the coming winter months required that the data

collection be postponed until the spring when roads will again be passable. The planned

adaptation workshop and EGRA exercise will be rescheduled at a later date, to be

determined.

In GBAO, the team first met with the Deputy Director (DD) of the GBAO Department of

Education (DOE) in Khorog. During this meeting, the following information was

provided:

The language situation in GBAO is one in which nine local languages, or mother

tongues (MT), are spoken at home and that the language of instruction (LOI),

Tajik, is virtually a foreign language for all students.

The DD stated that multiple home languages are the greatest challenge in GBAO

since the languages are quite different from one another and Tajik.

Students are required to learn Russian and English as well as Persian script for

Tajik.

Some schools offer an early childhood development (ECD) and/or “Grade 0”

program that have been created to help transition children from Mother Tongue to

Tajik.

Students not attending these schools are immersed in Tajik upon Grade 1 entry.



The Open Society Foundation funds 103 preschool centers managed by the

Institute for Professional Development (IDP).

There is a lack of adequate texts for Grades 1-4, as anticipated.

There is a need for family literacy programs so that parents can learn ways to

support their children’s early grade reading (EGR) learning.

Currently, there are no after school support programs, yet summer camp (i.e.,

summer school) is provided for 5,000 of the total 45,000 students.

Overall enrolment has been declining for at least the last 10 years due in large part

to a significant amount of out-migration. This has resulted in some very low

pupil-teacher ratios—5-to-7 in one school as per the DD—as well as closed

schools. When schools close, he queried aloud, “Where will children learn?”

Despite these challenges, 83% of all Tajiks who study in graduate programs

abroad are from GBAO.

It was noted for the DEP-AME team members that the DOE wanted all teachers to have

more training and that they would like to channel more resources toward pre-primary and

primary realizing that such an investment has a very high public return. At the conclusion

of the meeting the team was provided with a detailed list of all the primary schools in

GBAO inclusive of each school’s Grade 1, 2, 3, and 4 enrolments. Following the meeting

with the DD, the team conducted the formal EGRA Launch event, which took place

within the facilities of IPD. Those attending the launch included educators from IPD,

teacher trainers, curriculum and curriculum materials specialists, assessment specialists,

and M&E specialists



West Bank Early Grade Reading Assessment

DEP-AME continued implementation of the West Bank: Early Grade Reading

Assessment (Grade 2) baseline this year. The activity was designed to strengthen local

skills in the design, evaluation and

management of education programs, and

quality data capture and analysis to

support them. This was the first baseline

assessment of early grade reading skills

representative of all Grade 2 students in

MOE schools of the West Bank.

USAID support for this activity is

intended to generate regional and

country-specific education data, and

analyses of those data, that can be used

by the Ministry of Education and Higher

Education (MOEHE) to prioritize

education needs and future investments.

The objective is to help inform policy

decisions and planning by the Palestinian

Authority (PA) for improved reading instruction and student learning outcomes to; (1)

infuse EGRA findings into the design and development of MOEHE curricula and

teaching resources for enhanced reading instruction in Grades 1-3; (2) establish a national

baseline of Grade 2 reading skills to measure future progress in enhanced reading

performance in MOEHE schools; and (3) strengthen the capacities and deepen the

knowledge base of MOEHE staff to implement EGRAs.

This Grade 2 baseline established the capacities and deepened the knowledge base of

MOEHE staff to implement future EGRAs. During implementation, all EGRA assessors

and assessor team leaders were MOEHE supervisors or candidate teachers and all

planning, training, implementation, and dissemination was conducted in close

collaboration with the Ministry’s Assessment and Evaluation Department (AED) and

EGRA Steering Committee. Central ministry and MOEHE districts actively supported

the field implementation with school liaison, orientation, and enrollment lists for sample

selection.

This began during a November Launch Week during which DEP-AME meetings were

held with the USAID Mission and the Palestinian Ministry of Education. DEP-AME

team members traveled to Ramallah to:

Familiarize USAID/WBG Mission and the MOE with the objectives, design and

implementation of the national EGRA baseline; identify counterpart officials and

resource persons and their contacts; review and finalize work plans and schedules;

finalize the sample size and desired stratifications; and review and agree upon the

roles, tasks, responsibilities, and deliverables of the parties.



Review the EGRA FAQs and familiarize MOE and the WBG Mission with

previous Arabic EGRAs and analytical reports.

Review and confirm the purpose and intended use of results from the EGRA.

Identify early grade literacy/language specialists and assessment specialists within

MOE (i.e. future champions) as needed and communicate contact points.

Identify local language and reading experts to provide technical and psychometric

advice.

Confirm EGRA personnel and MOE responsibility, and sufficient, appropriate

personnel available to supply EGRA assessors and assessor team leaders.

Finalize schedule with MOE and USAID/WBG for EGRA implementation.

Clarify responsibilities, roles and timeframe for EGRA tool adaptation.

Obtain Grade 2 textbooks for Arabic reading as resources for EGRA tool

adaptation.

EGRA Sample Frame and School Data with the MOE and decide the sample

frame and desired strata for the analysis of EGRA results.

Clarify the process, MOE role and responsibilities in the selection of sample

MOE schools and Grade 2 students for EGRA testing.

Adaptation of instruments to provide training for data collection in Nepal for the TOS

and EMES (to be followed by the EGRA) in November and December. Following this, a

decision was made on the a March timeframe for the assessment and dissemination of

results; selecting the final EGRA subtasks to be used; setting up a process for adapting

the Arabic EGRA for Palestine from the instruments used in Jordan and Egypt; and

preparing the database of schools to be used for creating the sampling frame and strata.

During the second quarter, DEP-AME conducted the Grade 2 EGRA itself, working

closely with the Palestinian Ministry of Education, the USAID/West Bank/Gaza Mission,

and subcontractor Infonex. (The policy dialogue workshop was then scheduled to

conclude the activity in June but this was delayed due to hostilities.) The final week of

field data collection and counterpart meetings concluded in April with a celebration event

that convened all of the assessors, assessor team leaders (supervisors), field coordinators,

USAID and MOE representatives, to mark the successful completion of this fieldwork. In

fact, the USAID Mission asked DEP-AME to produce a documentary to show at a Policy

Dialogue Workshop.

This EGRA was conducted with a representative sample of 150 Ministry of Education

and Higher Education (MOEHE) primary schools, stratified by school gender and

selected randomly from 16 districts in the West Bank. The 2,953 tested students were

randomly selected from Grade 2 enrollment lists prior to each school visit. The results are

representative of MOEHE Grade 2 students and primary schools in West Bank. The

findings of this national baseline were intended to inform the strategic design and

development of MOEHE enhancements to the curriculum, resources and teaching of



early grade reading. Implemented by MOEHE supervisors and candidate teachers, this

EGRA strengthened ministry capacities and deepened its knowledge base of early grade

reading and reading assessment. Key findings of this Grade 2 baseline included:

Good foundation of pre-reading skills: The sample Grade 2 students demonstrated

good proficiency in the basic pre-reading skills of letter sounds knowledge and

nonword / invented word decoding. Average scores on these two subtasks are, by

far, the highest of any Arabic EGRA in Grades 2 or 3; and the percentage of zero

scores are the lowest. There is a strong foundation of reading proficiency in these

pre-reading skills among MOEHE Grade 2 students. This was a strong and

positive finding.

Students struggle to read connected text in passages: Most Grade 2 students were

struggling to read familiar words in connected text (passages, stories), indicative

of low reading fluency. Average scores on the oral reading fluency subtask (with

diacritics) are low and the percentage of students scoring zero is high.

Low reading comprehension: Most Grade 2 students do not comprehend what

they are reading. Average scores on reading comprehension (60 seconds reading

with diacritics) are low and the percentage of students with zero scores is high.

The generally good results on the listening comprehension subtask, however,

suggest that most Grade 2 children capably understand formal Arabic. Hence the

low scores on reading comprehension in Grade 2 chiefly result from low reading

fluency and undeveloped reading comprehension skills. Comprehension skills

must be learned, practiced and mastered—they do not come automatically to

students who mechanically read well.

The results will be used to inform policy decisions and planning by the Palestinian

Authority (PA) for improved reading instruction and student learning outcomes in the

early grade, while the longer term objective is enhanced teacher training and learning

resources for improved reading proficiency by primary students.

Next Steps

Complete review of the Final Report for USAID acceptance and finalize a video

documentary prepared to support future training.



DEP-AME activities focused this year on two analytic activities under Result 2. These

included completion and translation of the Topical Analysis of Early Grade Reading

Instruction in Arabic and development of related Early Grade Reading Briefs for use that

the November 2013 All Children Learning Workshop in Rabat, Morocco, and the

submission of an updated Topical Analysis of Scale up and Sustainability. In October,

USAID and DEP-AME reviewed the Result 2 Trends Analyses final documents.



Agreement was reached that no further work would be conducted until updated analyses

are conducted in 2015 or 2016. Throughout October and November communication also

continued with the USAID Bureau for Economic Growth, Education, and Environment

(E3) regarding the Morocco All Children Learning Workshop.

Topical Analysis: Early Grade Reading Instruction in Arabic

In October, DEP-AME submitted a draft of the Topical Analysis: Early Grade

Reading Instruction in Arabic to USAID for review. A subsequent revision was

submitted on November 1 for review and transmittal to a translator in preparation

for the All Children Learning Workshop I Rabat, Morocco. Further inputs were

received in November from the USAID E3 bureau and the comments were

incorporated into the paper. In early November, an updated budget was submitted

to USAID for the Topical Analysis: Early Grade Reading Instruction in Arabic to

incorporate additional inputs, including preparation of Country Briefs and travel

of presenters to Rabat, Morocco to participate in the All Children Learning

Workshop.

The All Children Learning Workshop in the Middle East and North African region

was held in Rabat, Morocco from December 2-5, 2013 in order to mobilize

partners from governments, civil society, and development organizations together

to share knowledge and best practices on how to dramatically improve the

learning levels of children learning in the early grades across the region. DEP-

AME also researched and drafted Country Briefers on the status of early grade

reading in Middle East/North Africa (MENA) countries, including Egypt, Iraq,

Jordan, Lebanon, Morocco and Yemen. The Country Briefer profiles were

developed for use with country delegations. DEP-AME also supported a roster of

speakers for the Workshop, including Dr. Sylvia Linan-Thompson, Dr. Helen

Boyle and Ms. Samah Al Ajjawi of Florida State University. Dr. Amber Gove

also presented on the EGRA Barometer which is supported under Result 3.

In November, a revised draft of the Topical Analysis: Early Grade Reading

Instruction in Arabic was sent to USAID for translation, including a list of

interviewees by country. Additional feedback was received and a reorganization

of content was undertaken to incorporate the requested changes. In December,

input received from USAID and incorporated into the final report. Subsequent

revisions, including the eventual translation of the report into Arabic, concluded

this activity by the final quarter of the year.

Next Steps

None, this activity is completed.

Topical Analysis: Scale up and Sustainability

In the second quarter, DEP-AME submitted an updated draft of the Topical Analysis:

Scale and Sustainability under Result 2. The paper offered guidance on how to scale up



and sustain efforts aimed at improving early grade reading (EGR) in developing countries

around the world based on an extensive review of the scale-up and sustainability

literature. This revealed that there are several fundamental factors of successful scale-up

and sustainability:

The overall effort unfolds within a potentially hostile political economic

environment, therefore reform support measures must be taken to address it; the

effort must be demand driven; people must be held accountable, and incentives,

both positive and negative, matter;

Measures must be taken to engender widespread and deep-rooted ownership;

That which is going to scale and being sustained must be accommodated within a

coherent systemic and institutional home, the effort must then be systemic;

Trivial as it may sound, but often ignored, the effort must be doable over time, the

host country must be able to carry out the tasks and assume the recurrent and

capital costs required to keep it going;

Much of the work of reform needs to be carried out by an entity whose primary if

not sole purpose is to do this work—implementation support is necessary;

Depending on the nature of the intervention being taken to scale and being

sustained, it must, to some degree, be adaptable, flexible, and able to be

accommodated across a variety of contexts;

The effort needs to be seen as on-going—reform never really ends—and as such it

must be learning- and evidence-driven.

The reported also noted that each fundamental factor has its respective set of operational

requirements—the specific things that one must do in order for them to be realized. If, for

example, the effort must be demand driven, then specific tasks must be undertaken to

generate that demand. If people must be held accountable, then viable accountability

mechanisms need to be put in place and fully operationalized to ensure that this happens.

This is useful, as it suggests that scale-up and sustainability can be greatly abetted if

implementers pay attention to these operational requirements, but it also imposes an

obligation to do so. For this reason, listing and delineating them carefully is an important

task.

Also, a number of these fundamental factors strongly suggested that what should be taken

to scale and sustained within the education sector is the capacity to innovate as opposed

to the innovation itself. Making this distinction is itself insightful. What works in one

context may not work well in another. Therefore, it makes sense to scale-up and sustain

the capacity to innovate.

Next Steps

Pending feedback from USAID, revise the Topical Analysis: Scale and

Sustainability if desired.



Result 3: Measurement Tools for Cross-Country Use


The EGRA Barometer is an interactive tool intended to offer wide range of EGRA data

from Asia and the Middle East and ultimately to be piloted in the Asia and Middle East.

This year, DEP-AME began development of a proposal for Phase 2 to pilot the EGRA

Barometer in the Asia/Middle East region and identified five objectives for further

development next quarter:

Share the Barometer. Let target audiences know about the tool.

Teach target audiences how to use the Barometer to facilitate dialogue, inform

policy and influence program planning.

Learn how the tool is used in the field and identify its most useful features

Add more countries to, and increase the functionality of, the Barometer

Understand the resources required to add datasets resulting from assessments

completed by other contractors (e.g., not under EdData II)

Ongoing feedback loops from USAID and input from selected Mission staff continually

improved its functionality this year, during which DEP-AME:

Completed the Beta release of the EGRA Barometer for launch and submitted it to

USAID.

Developed a video to showcase the EGRA Barometer data content and use.

Presented the EGRA Barometer on January 29 to the USAID Education Sector

Council.

Presented the EGRA Barometer with USAID on March 18 at the World Bank

Community Discussion on Measurement & Data Issues in Early Grade Reading,

which focused on updates on measurement and data issues in early grade reading.

Presented the EGRA Barometer on March 31 at USAID’s Innovation Lab launch.

Began planning for Phase 2 of the EGRA Barometer for submission of a proposal

to USAID in the next quarter.



During the first quarter, RTI worked on the design and development of the EGRA

Barometer tool, which allows real time manipulation of EGRA results, provide measures

of average performance, show full

distribution of student performances,

demonstrate the relationship between

levels of performance on various EGRA

subtasks, and facilitate setting standards

for student performance in reading using

EGRA data.

In November, DEP-AME delivered the

Beta version of the EGRA Barometer

Tool with data for Egypt, Iraq, Jordan,

Morocco, and Yemen for USAID review.

Its three modules focused on using data

to (1) understand existing levels of

reading achievement, (2) generate

projections for how reading interventions

could impact learning outcomes on an

increasing scale, and (3) measure the

impact of reading programs.

During the second quarter, DEP-AME

completed the Beta version of the EGRA

Barometer, showcased it at several key

meetings for USAID, and submitted a

work plan and proposal for Phase 2

(which is now piloting the instrument for

further development and field application). Presentations on the EGRA Barometer this

quarter occurred on January 29 to the USAID Education Sector Council; presentation of

the EGRA Barometer with USAID on March 18 at the World Bank Community

Discussion on Measurement & Data Issues in Early Grade Reading; video demonstration

at the USAID/Africa Bureau Africa Regional Education Workshop (AREW) on March

24, and presentation of the EGRA Barometer on March 31 at the USAID Agency’s

Innovation Lab launch.

The Barometer was also presented in Indonesia in August by Peggy Dubeck and Alastair

Rodd; in the Philippines twice (in July and August) by Joe DeStefano and Sarah

Pouezevara. It was also presented in Colombia in June by Carmen Strigel and Amber

Gove.

Phase 2 key activities include:

EGRA Barometer Highlights:

The number of users increased in June (80) and July (110) from May (41)

The number of sessions increased in June (121) and July (152) from May (103)

While the number of users and sessions increased, the Average session duration decreased: June (3:41), July (3:19) vs. May (4:01)

The highest number of sessions each month is from the USA (81 in July)

The second highest in July is Brazil (24), followed by Nepal (8)

The second highest in June is Indonesia and Canada (9 each)

To date, the highest number of sessions is from the USA (616), followed by Brazil (26), and the Philippines (19)



Task 1: Dissemination and Outreach

This task addresses Pilot objectives 1, 2, and 3 (Share, Teach, and Learn). A robust and

multi-faceted dissemination strategy is essential to realize the potential of the Barometer

to facilitate policy dialogue and program planning. DEP-AME began undertaking

activities, beginning with the development of:

Develop and implement a dissemination plan.

Leveraging social media, such as Facebook to engage as many stakeholders as

possible.

Establishing a network of trusted dissemination partners to maximize Barometer

reach.

Conducting up to four (4) presentations in Washington, DC (or other venues) for

key audiences as requested by USAID.

Capturing profiles (vignettes) of how countries use the Barometer and creating

links to these profiles within the Barometer so that other users can see examples

of how the tool is being used. These profiles (vignettes) will be sent out as

electronic newsletter (E-News).

Gathering user testimonials for posting on the Barometer’s home page.

Preparing quarterly E-News and disseminating them.

Training. Conducting in-country training and capacity building workshops around

using the Barometer to facilitate policy dialogue. Workshop participants include

USAID education staff and other counterparts or stakeholders designated by the

identified USAID Mission.

Website Analytics. Real-time monitoring of user activity has been underway to

provide an important vehicle for tracking progress and improving the

effectiveness of the dissemination strategy. Monitoring of website activity (for

example, number of users, the pages they viewed, and so on) was collected using

Google Analytics and tracked, as well as the number of times the Take a Tour

video is played with data from Vimeo. This information has been supplemented

for USAID by summaries from the user account signup portion of the website,

and the responses to the online user survey.

Next Steps

Monitor data and prepare a monthly activity summary for USAID.

Report on dissemination efforts and outcomes.

Prepare a final report on the effectiveness of the dissemination efforts and lessons

learned.



Task 2: Data

This task addressed Pilot objectives 4, and 5 (Add and Understand). While the first data

task began incorporating additional EGRA datasets collected by RTI under EdData II,

this second involves determining the level of effort associated with incorporating

additional early grade reading assessments from other agencies contracting with USAID.

This began during the later quarters of the fiscal year. When completed, the end products

will allow USAID to assess the advantages, both in terms of cost and time to

dissemination, of requiring contractors to use a set of uniform standards for measures for

EGRA studies and delivering data files in a format optimized for easy transfer to the

Barometer. Activities begun under this task include:

New RTI Assessment Data. EGRA’s from Nepal, Philippines, Indonesia, and

Egypt were added to the Barometer. Incorporating these new data required

reviewing distributions, examining associations among subtasks, creating analytic

variables for the web tool, and comparing results to any existing reports about the

particular country’s data. The scope and number of assessment measures and the

number of grades and languages in each assessment have varied by country,

which has required modifying the Stata programs that produce the data for the

web application.

EGRA’s Conducted by Other Organizations. USAID has additional early grade

reading assessments designed and collected by other organizations. Although

there has been agreement among implementers to a common framework, efforts

to achieve such standardization are in their infancy. DEP-AME began work this

year to design a framework for data integration into the Barometer and began

investigating the level of effort required to add two non-RTI datasets to the

Barometer. While the work required is similar to those performed for adding a

new RTI EGRA dataset, since the data are from another organization these

activities will involve greater complexity. DEP-AME efforts to address these

complexities confirms the need for even greater data cleaning preparation prior to

transmitting the information to RTI.

Data Specification Document and Instructions for Data Providers. In addition

to including sample datasets in the Barometer, DEP-AME produced a draft data

specification document for use by organizations contracting with USAID to

conduct EGRAs. The document provides clear definitions of each subtask to be

incorporated into the EGRA Barometer; the format for each of these measures,

including variable names, labels, and types; and a list of, and structure for,

additional information needed for the technical and methodological appendices

included in the Barometer.

Task 3: Application Development

This task focuses on the part of Pilot objective 4 for increasing the functionality of the

Barometer. Application development involved a number of extensions to the analytic and



presentation capabilities of the tool, allowing broader and deeper examination of early

grade reading data by USAID staff and host country partners. DEP-AME planning

focused on the following critical areas:

Benchmark Projection Methodology. The EGR Barometer used data from the

Egypt Girls’ Improved Learning Outcomes (GILO) study, a grade 2 reading

intervention, to model projected changes in the percentage of students meeting

oral reading fluency targets (set by the user) through 2020. As additional EGRA

data become available, DEP-AME used the opportunity to develop alternative

benchmark projections and to build the datasets. New data for existing countries

allowed DEP-AME to begin to explore alternative approaches for calculating

growth, and to examine such options for improving and making more appropriate

the projection methodology used in the benchmarking portion of the Barometer.

Trend Analyses. As new waves of data for existing studies became available or

new studies conducted in countries that have already completed an EGRA, DEP-

AME began to analyze change over time and provided graphics and tabular

displays highlighting those areas where progress had been made and others where

it lags behind. This involved adding another feature to the Barometer that allows

the user to compare the results of assessments conducted at two points in time in

the same country.

Side-By-Side Comparisons. Currently, performance on the various EGRA

subtasks is presented separately by grade and by language. In many cases, users

may benefit from being able to compare results for different languages or grades

on the same graph. For example, the distributions of scores in oral reading fluency

for grades 2 and 3 could be displayed on the same graph, potentially illustrating

how scores shift from one year to the next and providing a concise view of a

year’s growth in the particular skill area. Similarly, side-by-side comparison of

EGRA outcomes in the different languages in which a student was assessed (like

in the Philippines) or among different language groups (where different students

are assessed in different languages) may be of particular interest to USAID and

host country staff. This has required adding considerable functionality to the

existing results feature. DEP-AME has developed features dedicated to making

comparisons.

Stand-alone Versions of the Barometer. As new datasets are added to the

Barometer, DEP-AME worked to develop a stand-alone versions that does not

require an Internet connection and which can post them for download.

More Reading Resources. In response to feedback already received, DEP-AME

expanded the Resources section to create a richer collection of information that

can be used as a reference and as a tool in advancing the progress of all children

in learning to read.

Content Management System. The capacity of nontechnical staff to modify the

Barometer text (not EGRA data) is being developed for use as a customized



Content Management System (CMS). As the Barometer is revised, it will be

updated to ensure USAID staff retain this ability.

Tool Documentation. DEP-AME prepared detailed technical documentation to

ensure the Barometer can be managed by another vendor or USAID technical

staff in the future. The documentation will be updated whenever major changes

are made to the Barometer.

Other Features. DEP-AME also began to respond to requests for additional

functionality by users and/or USAID. RTI will work with USAID to develop a

scope of work for such new features if and when they are deemed necessary to

add to the application.

Task 4: Site Maintenance and User Support

This task underlies all the Pilot objectives to ensure that the Barometer is always

available and functioning as expected, regardless of changes in web browser versions.

Throughout the year DEP-AME monitored site maintenance and user support.

Task 5: Tablet Application

This task addressed Pilot objectives 1, 2, and 3 (Share, Teach, and Learn). To ensure the

Barometer is available for a wide range of users, DEP-AME will develop a tablet version.

The tablet application will mirror the functionality of the downloadable version of the

website. It will be redesigned so that the user interface can accommodate the common

screen resolutions of tablets, write necessary iOS- and android-specific code, conduct

comprehensive quality control testing, work with Apple through application review and

approval, and launch the applications. Further work on this will continue. The application

will target the most common tablet hardware and operating systems, including: iOS: iPad

Air, iPad 2, and iPad Mini with Retina Display, Android: 7-inch tablets (similar to Nexus

7), and 10-inch tablets (similar to Galaxy Tab 10.1).

Task 6: Other

Additional areas of work were mentioned in the early rounds of feedback when the

Barometer was being demonstrated and tried out. The first concerned whether it would be

possible to add more contextual data/information about each country. Some suggested

adding information on education expenditures or other education sector information.

Adding such data would involve identifying reliable sources and creating standardized

ways to important and display such information. The second issue raised during the initial

demonstrations of the Barometer concerned making it available in languages other than

English. Thus, DEP-AME will prepare the materials necessary to obtain cost estimates

for creating an Arabic and a Russian version of the Barometer for the coming year.

Next Steps

Monitor use and refine the instrument and populate new country data throughout the pilot

period.



Example of EGRA Barometer Refinement



Barometer Data Usage Summary—May 2014



Barometer Data Usage Summary—June 2014



Result 4: USAID Staff Knowledge Strengthened


During the first quarter DEP-AME continued preparations for face-to-face Evaluation for

Education Projects Workshop, originally planned for February 2014 in Bangkok,

Thailand at the USAID Regional Training Center. In October, the proposed timeline and

update was submitted to USAID including proposed evaluation experts to be considered

as workshop presenters. Dr. Ben Castleman of the University of Virginia was selected as

the consultant to design and provide the training. A complete planned schedule was

prepared and submitted to USAID. An application for the USAID/Bangkok Regional

Training Center was also submitted to secure the date and venue. The training date for

this first classroom training was confirmed for February 24-26, 2014.

In January, confirmation was received of the postponement of the planned February

training at the USAID Regional Training Center. An Annotated Agenda was submitted at

this time in January for USAID review and feedback. Recommendations from USAID on

the importance of non-experimental methods, evaluation in program design, and results

dissemination were received and reviewed, including these:

Summarize the current and available reading research to clarify what we know

works, and to identify gaps in research and evaluation.

Expand coverage on non-experimental methods, including pair matching and

value-added modeling, and on how to incorporate evaluation into project design.

Develop an overview and summary of how to review and interpret evaluations.

Assess communication approaches for diverse audiences on how to communicate

technical info to a largely non-technical audience in a clear and compelling way.

Based upon this input, research activities continued with Dr. Ben Castleman although a

new date had not yet been confirmed. During the third quarter a November 2014

timeframe was confirmed, during which Dr. Castleman would no longer be able to serve

as the course trainer. In response, RTI identified a replacement instructor Dr. Melissa

Chiapetta of Social Impact, who was quickly contracted to provide the training. Meetings

were held throughout the latter part of this reporting period to refine and complete the

training agenda and to meet with USAID and Dr. Chiapetta, and with Allegro to complete

the training design. Throughout the fourth quarter the training materials were refined and

presentations prepared for the face-to-face training, confirmed for November 17–19,

2014.

Toward the end of the Fiscal Year the course content was being further refined, with the

goal that the participants would be able to:

Describe the importance of evaluations for education programs, projects, and

activities

Differentiate between qualitative and quantitative data collection techniques and

describe value of each



Differentiate between performance and impact evaluations and understand when

each is appropriate

Differentiate between experimental, quasi-experimental, and non-experimental

evaluation designs as well as strengths and limitations of each

Manage an evaluation from design and procurement to dissemination of results

Write an effective SOW for an evaluation and critically review evaluation

deliverables

The course designed was premised on USAID 2011’s Evaluation Policy and the ADS and

was designed to address these core considerations:

Accountability

Did the project/activity deliver the inputs and outputs it was contracted to? Why

or why not?

Were funds used efficiently?

Learning

Did the pilot project/activity prove effective at achieving the outcomes and

impacts it targeted? Why or why not?

Which aspects of the project/activity were most effective or cost effective?

In which contexts would a similar project/activity work well in the future?

How might we improve the sustainability of projects/activities?

The table below presents an excerpt from the training presentation, which will be covered

more fully in the next reporting period.

Next Steps

Complete the training course design, materials, and face-to-face instruction.



Coverage Monitoring Question Examples Evaluation Question Examples

Inputs

(Activities, Resources, etc.) How many participants were invited to the training? How was the program implemented? Was implementation in accordance with design and specifications?

How effective were those inputs? Did we target the right inputs, or were there others resources we should have provided or activities we should have delivered? How well was the program implemented? For outreach, did we use the best avenues and methods we could have? How well did we access hard-to-reach and vulnerable populations? Did we reach those with the greatest need? Who missed out, and was that fair?

Outputs

(Products, Services, Deliverables, Reach)

How many people or communities were reached or served? Were the targeted numbers reached?

How adequate was program reach? Did we reach enough people? Did we reach the right people?

Outcomes

(Usually the medium-term goals of the project; things that happen to people and communities)

What has changed since (and possibly as a result of) program implementation? How much have outcomes changed relative to targets?

How substantial and valuable were the outcomes? How well did they meet the most important needs and help realize the most important aspirations? Were they not just statistically significant, but educationally, socially, economically, and practically significant? Did they make a real difference in people’s lives? Were the outcomes worth achieving given the effort and investment put into obtaining them?

Impacts

(The longer-term, higher-level results of the project; things that happen to sectors, economies, countries)

Did impact-level measures or indices change over the same period of time as the project (cause is not evident)?

To what extent did the project contribute towards changes in impact-level measures or higher-order results?



Overarching DEP-AME Next Steps

While DEP-AME has had a highly productive year and enjoyed robust activity, some

implementation has proceeded more slowly than originally anticipated. This is due in

large part to the fact that activity plans and schedules must align with USAID Missions’

schedules and priorities. And, the close work with in-country partners, often results in

extensive delays due to scheduling, security, or simply the need for host country policy

dialogue to progress. As an example, this year delays were encountered in Bangladesh,

Nepal, Tajikistan/GBAO, and in development of the Evaluation Training Course. The on-

going activities that are slated to continue under Result 1 and to be completed in this

coming fiscal year include the following:

Impact Evaluation Training Course

Tajikistan Gorno-Badakhshan Autonomous Oblast (GBAO) Early Grade Reading

Assessment.

Early Grade Reading Assessment (EGRA) Barometer

West Bank EGRA

Philippines Reading

Other activities also remain within DEP-AME’s scope of work which have not yet been

implemented, including development or refinement of evaluation tools, and

implementation surveys and assessments in response to Missions’ data needs. Given the

need to work closely with missions, it can take several months to schedule activities and

they in turn are working closely with their ministerial counterparts for country-buy-in.

Once activities begin they routinely take from 6-9 months.

Under Result 2, this includes a second trends analysis (needed to measure the impact of

USAID’s regional efforts).

Under Result 3, this includes the EGRA Barometer.

Under Result 4, this includes completion of the existing Impact Evaluation training and

development of an additional training course.


Annual Report Year 3 (01 October 2013 – 30 September 2014) A-1

Annex A: Financial Summary

RTI International—Yearly Financial Statement, Task 15 (Year 11)

Contractor/Recipient: RTI International Data for Educational Programming in Asia and Middle East (AME)

Award No.: Prime Contract AID-OAA-BC-11-00001

Performance Period: September 29, 2011 November 30, 2014

Yearly Financial Report as of September 30, 2014

A B C D E F=C+D+E G = B-F H = A-F

Results Description

Original or Revised Total

Estimated Cost

Obligated Amt. to Date

Total Expended as of (March ‘14)

Total Expended

(Apr-June ‘14) Outstanding

Commitments

Total Expended to

Date

Balancing Remaining of

Obligated

Balance Remaining of

Total Est. Cost

Data for Education Programming in Asia and the Middle East (DEP-AME)

[ REDACTED ]

summary annual progress report - … · summary annual progress report october 1, ... tos teacher...

Documents