ISSUES IN THE EVALUATION OF A PROGRAM TO PROMOTE EDUCATIONAL ACHIEVEMENTS OF ETHIOPIAN-ISRAELI

HIGH SCHOOL STUDENTS

"Improving Education through Accountability and Evaluation: Lessons from Around the World”

Rome, Italy, October 2012

DALIA BEN-RABI, VIACHESLAV KONSTANTINOV, RUTH BARUJ-KOVARSKY, MIRIAM COHEN-NAVOT

Engelberg Center for Children and YouthEngelberg Center for Children and YouthMyers-JDC-Brookdale InstituteMyers-JDC-Brookdale Institute

Since 1984 Most parents are illiterate In 2012 – most children were born in IsraelIn spite of efforts of government, local

authorities, and voluntary organizations large gaps in many areas, including academic achievements

2

IMMIGRATION FROM ETHIOPIA TO ISRAEL

3

ACHIEVEMENTS ON NATIONAL EXAMS OF ETHIOPIAN-ISRAELI 8TH GRADERS, COMPARED TO THOSE OF ALL STUDENTS IN HEBREW SPEAKING SCHOOLS IN ISRAEL, 2004/5 (AVERAGE SCORE, ON A SCALE OF 0-100)

4

ACHIEVEMENTS ON NATIONAL MATRICULATION EXAMS OF ETHIOPIAN-ISRAELI 12TH GRADERS, COMPARED TO THOSE OF ALL STUDENTS IN HEBREW SPEAKING SCHOOLS IN ISRAEL, 2004/5 (%)

Was first implemented in 2006Aimed at Ethiopian-Israeli students in middle

schools and high schoolsPlaces special emphasis on matriculation exams Instruction in small groups for about four hours

weekly, in different school subjectsSocial activities and personal coaching At its peak in 2008 reached over a third of all

Ethiopian-Israeli students in grades 7-12 nationwide

5

THE PROGRAM

Providing regular feedback each year to support ongoing program improvement: The implementation of the program

Satisfaction of the school principals and

the participating students

The impact of the program on the students’ scholastic achievements

6

EVALUATION

Eligibility for partial matriculation certificates (enables admission to advanced professional courses)

Eligibility for matriculation certificates (a prerequisite for many jobs and some colleges)

Eligibility for matriculation certificates with a high-level of English (a prerequisite for universities)

7

MEASURES OF ACHIEVEMENT

Are the achievements of program participants better than those of non-participants?

Is the difference in achievements growing over years, as the students are exposed to the program more years before taking the exams?

Are the gaps between Ethiopian and non-Ethiopian Israeli students narrowed?

8

QUESTIONS OF IMPACT: CAN WE ATTRIBUTE THE DIFFERENCES TO THE PROGRAM?

9

ACHIEVEMENTS OF PARTICIPANTS 2009/10

10

MATRICULATION ACHIEVEMENTS OF 12TH GRADERS, 2010 (%)

Partial matriculation

matriculation

University matriculation

11

Boys

Girls

GIRLS DO MUCH BETTER THAN BOYS: % OF BOYS AND GIRLS WITH MATRICULATION CERTIFICATES, 2010

12

GIRLS DO MUCH BETTER THAN BOYS: % OF BOYS AND GIRLS WITH MATRICULATION CERTIFICATES THAT MEET UNIVERSITY ADMISSION REQUIREMENTS, 2010

Boys

Girls

13

IMPACT OF THE PROGRAM ON THE STUDENTS' MATRICULATION RESULTS

Schools and students were not randomly assigned to the program

Schools in the program are stronger than the average, and weakest students are not included

Schools came in and out of the program over the years

14

THE PROBLEM: NO CONTROL GROUP

Comparison groups:

1.Ethiopian-Israeli students from similar non-participating schools and with similar socio-demographic characteristics:

Student characteristics: country of birth (Israel/Ethiopia), genderSchool characteristics:

State or State Religious school The percentage of students eligible for matriculation certificates among

non-Ethiopian school students, in the year of the analysis The percentage of students eligible for matriculation certificates among

Ethiopian-Israeli school students in 2005, prior to the implementation of the program.

2. Non-Ethiopian Israeli students, from participating schools

15

THE SOLUTION: MATCHING STUDENTS BASED ON ADMINISTRATIVE DATA

As students have had the opportunity for more years of exposure to the program:

Differences in achievements between participants and non-participants should grow

Gaps between participants and non-Ethiopian students should be narrowed

The comparison is of the added value of the program as opposed to whatever form of other assistance may exist in program or comparison schools

16

ANALYSIS 1: CHANGES OVER TIME IN 28 SCHOOLS THAT PARTICIPATED IN THE PROGRAM FOR AT LEST 3 YEARS

17

ACHIEVEMENTS OF PARTICIPATING STUDENTS, ALL ETHIOPIAN-ISRAELI STUDENTS IN PARTICIPATING SCHOOLS AND STUDENTS IN COMPARISON GROUP, BY YEARS OF EXPOSURE TO THE PROGRAM (% OF STUDENTS ELIGIBLE FOR MATRICULATION CERTIFICATES)

Impact (difference between program participants and comparison group, divided by % in comparison group)

27%*

Years in the Program (28 schools) *p<0.01

*49% not significant

18

ACHIEVEMENTS OF PARTICIPATING STUDENTS, ALL ETHIOPIAN-ISRAELI STUDENTS IN PARTICIPATING SCHOOLS AND STUDENTS IN COMPARISON GROUP, BY YEARS OF EXPOSURE TO THE PROGRAM (% OF STUDENTS ELIGIBLE FOR MATRICULATION

CERTIFICATES MEETING UNIVERSITY ADMISSION REQUIREMENTS)

20%*

Years in the Program (28 schools) *p<0.05 **p<0.01

**63%

Impact (difference between program participants and comparison group, divided by % in comparison group)

not significant

19

THE ACHIEVEMENTS OF ETHIOPIAN AND THE NON-ETHIOPIAN ISRAELI STUDENTS IN THE SAME SCHOOLS, BY YEAR OF EXPOSURE TO THE PROGRAM (% OF STUDENTS ELIGIBLE FOR MATRICULATION CERTIFICATES)

Years in the Program

20

THE ACHIEVEMENTS OF ETHIOPIAN AND THE NON-ETHIOPIAN ISRAELI STUDENTS IN THE SAME SCHOOLS, BY YEAR OF EXPOSURE TO THE PROGRAM (% OF STUDENTS ELIGIBLE FOR MATRICULATION CERTIFICATES MEETING UNIVERSITY ADMISSION

REQUIREMENTS)

Years in the Program

21

IMPACT OF THE PROGRAM ON THE ACHIEVEMENTS OF STUDENTS WHO PARTICIPATED IN 2008-2010, TAKING ACCOUNT OF THEIR ACHIEVEMENTS IN EIGHTH GRADE

22

MATRICULATION EXAM ACHIEVEMENTS OF PARTICIPATING STUDENTS AND STUDENTS IN THE COMPARISON GROUP WITH THE LOWEST PREVIOUS ACHIEVEMENTS, AND THE IMPACT OF THE PROGRAM (2008-10, IN PERCENTAGES)

*p<0.05 **p<0.01 282 participating students

23**p<0.01 292 participating students

MATRICULATION EXAM ACHIEVEMENTS OF PARTICIPATING STUDENTS AND STUDENTS IN THE COMPARISON GROUP WITH MEDIUM PREVIOUS ACHIEVEMENTS, AND THE IMPACT OF THE PROGRAM (2008-10, IN PERCENTAGES)

24*p<0.05 167 participating students

MATRICULATION EXAM ACHIEVEMENTS OF PARTICIPATING STUDENTS AND STUDENTS IN THE COMPARISON GROUP WITH THE HIGHEST PREVIOUS ACHIEVEMENTS (2008-10), AND THE IMPACT OF THE PROGRAM (2008-10, IN PERCENTAGES)

Estimate the impact of a comprehensive scholastic assistance program for Ethiopian-Israeli high school students, while trying to overcome the obstacles resulting from the lack of random assignment of schools and students to the program by creating a comparison group of matched students, based on administrative data.

25

THE STUDY AIMED TO

Better achievements compared with non-participating students from similar schools and with similar socio-demographic characteristics

The impact of the program was also reflected in the narrowing of the gap between Ethiopian-Israeli and non-Ethiopian students

26

THE STUDY SHOWED A SIGNIFICANT PROGRAM IMPACT ON THE MATRICULATION RESULTS OF PARTICIPATING STUDENTS:

The impact has increased as students have had the opportunity for more years of exposure to the program

The most significant impact was on students with the lowest previous achievements

Nonetheless, considerable gaps still remain, among boys, and especially as regards English exams. This poses a barrier to obtaining a matriculation certificate that meets university admission requirements.

27

Cost-effectiveness – all students or weakest students?

Boys and girls – do they require different forms of assistance?

English as a third language The continuum between elementary school

assistance and high-school assistance What are realistic expectations for attainable student

achievements, given both the large gaps between them and other students and the scope of assistance that the program provides? 28

KEY ISSUES: