learner programme d2 - zenex foundation support...evaluation of the high school learner programme...

The Tertiary Access Programme: career

guidance | assistance with tertiary and

bursary applications | peer support | bridging

fi nance

Boarding and transport (when applicable)

Academic support, in English, Maths and Science, in the form

of “targeted academic intervention” provided

on a needs basis

Group and individual mentorship or

counselling, and

Learner camps in Maths, Science and

English

Placement Sponsorship of annual school fees,

books, other stationery, and uniform

1. 11 school2. Selection - disadvantaged learners from

feeder and participating schools3. 661 of learners4. KwaZulu Natal

1. 11 schools2. Selection - disadvantaged learners3. 693 of learners4. KZN, GP, WC

Increase the number of learners who obtain Batchelor passes in the NSC with quality passes above 60% in Maths, Science and English.

PROGRAMME GOALS: PROGRAMME RATIONALE:

TOTAL COSTS:

To provide an opportunity for these learners to follow careers in Maths- and Science-related subject streams at university level. The LP aimed to facilitate the entry and integration of potential learners from disadvantaged backgrounds into selected high quality public and independent schools.

Intervention: R119,347,994

Evaluation: R6,315,956

TOTAL NUMBER IN THE PROGRAMME:

SELECTION:

1354NUMBER OF LEARNERS

COMPLETED NSC:

1255(8 % attrition)

1. RACE: African, Coloured and Indian Learners.

2. ACADEMIC SELECTION: Assessments in Maths and English and critical thinking skills. Education disadvantage was taken into account:a) Learners from schools based in township and rural areas must achieve at least 50% b) Learners selected from ex model c/HOD/HOR schools must achieve at least 60% on the entrance tests.

3. SOCIO ECONOMIC CIRCUMSTANCES: Household income must be below R15 000 per month or R180 000 per annum (proof of income to be provided). Number of dependents is taken into account as well as learners who are cared for by grandparents, single parents or in child-headed households or who are orphans.

Opportunities for Black learners to follow careers in Maths and/or Science at university.

Increase the number of Black graduates who can contribute to scarce skills in the country.

123

PROGRAMME COMPONENTS

IMPLEMENTED IN TWO MAIN SCHOOL TYPES:

PUBLIC SCHOOLS INDEPENDENT SCHOOLS

Zenex Learner Programme:WHAT WE LEARNT

MATRIC COHORTSIdependent Schools: 2009 & 2010

Public Schools: 2011 & 2013

• • Selection of learners at Grade

8 level was not very successfull. All learners were channeled into Maths and Science at Grade 10

but at least half could not meet academic targets.

• While the outcomes were the same for each school, the interventions in schools varied - complex project design and evaluation.

• Initial cohorts scored way above the selection criteria and this may have had a positive effect on learner achievement. By the third cohort, selection for potential

included learners who scored between 45-65% on the selection tests

The basic purpose of the evaluation was to:• Document the programme in detail.• To provide feedback and recommendations to improve

implementation.• Assess the merits and worth of programme.

In-depth case studies were completed in four selected Public

Schools

• Matched classmates not on programme (Public Schools)

• Classmates from original schools (Independent Schools)

A sample of 106 from Independent and 159 from Public Schools -alumni

who responded i.e. self-selected

1. School Marks: Academic Progress and Achievement in Maths, Science and

Language 2. NSC Results

• Learner and alumni surveys• Interviews with learners, teachers and

project co-ordinators• Focus group discussions with learners• Baseline testing by evaluators

Evaluation Focus

Evaluation Components

Implementation: Determine if the essential components of the intervention, according to the logic framework, are in place

Progress: Assessing progress toward meeting the primary goals of the intervention

Summative: Determine to what extent the intervention has impacted on learner achievement

Tracer study: Determine the impact of the intervention on student access, retention and achievement at tertiary studies.

Evaluation DesignLearner Project Goals Findings and Recommendations

TRACER STUDY

COUNTERFACTUAL

CASE STUDIES

DATASETS

PRIMARY DATA

1. Project Documents and Monitoring Reports

2. Selection Tests (marks, reports on selection)

SECONDARY DATA

Mixed Method Convergent Parallel

Can the results be attributed to the Learner programme itself?

And what would have happened to those learners if they had stayed at their school?

2007 - 2012Most learners from poorly performing low quintile schools achieved considerably better than their classmates left behind even if they did not achieve the programme targets of 60% in Maths, Science and English.

Project learners were more likely to study English FAL, core Mathematics and Physical Science than classmates left behind. Retention of project learners in core Mathematics and Physical Science is a significant success.

The tracer study included alumni who were in their first/second year of study - no information on completion rate. However, positive early

indications - very few first year failures as opposed to national statistics.

The tracked Independent Schools alumni identified many benefits of being on the programme, notably their good English language

proficiency and their academic skills and work ethic.

The Public Schools alumni were generally positive about the academic support.

In terms of personal development both Independent and Public Schools developed grit and resilience

and the confidence and self-esteem.

• School culture influences delivery.

• Implementation of academic support unstructured and uneven.

• School project champions critical for implementation success.

WHAT WE LEARNT

• Target group: Over 80% from previously disadvantaged Black communities.

• Socio-economic: Difficulty in assessing due to complex financial arrangements in extended families

• Educational disadvantage: Inconsistently applied for selection of learners from within the schools and from feeder schools

• Selection: Focus on previous school academic achievement. Adapted upwards over time to meet project goals more easily

2009 – 2013 & 2011 -2015There was a great variation in NSC achievement - between schools, within schools, and between subjects.

Nearly all the learners met the English target of 60%, but typically only about half met the target in Maths and Physical Science.

2007 - 2012The percentage of learners achieving the project goal of a Bachelors pass had nearly doubled from 2009 to 91% in 2012

Likewise, the proportion of learners reaching the project target had greatly increased to 81% for English, 56% for Maths and 63% for Physical Science.

Finance: By 2012 (of the tracked alumni) 16% had bursaries, 35% had NSFAS loans, 49% relied on

family or self support

Finance: By 2014 (of the tracked learners) 31% had bursaries, 17% had NSFAS loans, 43% relied on family or self support and 9% had fees owing

NSC 2009 - 2012The field of study reported are approximately evenly divided between Humanities and Law;

Commerce and Accounting; and STEM fields i.e. about 34 % of

chosen courses.

NSC 2011 – 2013Over the four cohorts, the most

frequent field of study reported is engineering, followed by health

and life sciences. These fields together with mathematical and

computing sciences, and physical sciences account for STEM of 69%

of the fields of study chosen.

2007 - 2012• Quality instruction within school: YES• After school support: uneven

implementation from compulsory to voluntary and from regular to occasional

• Support strategy: Traditional revision and repeat teaching models

• Targeted academic remediation: different understanding and interpretations across schools

2009 - 2013• Quality instruction in school: YES• Learner support: Uneven in dosage,

tutor recruitment and payment.• Support strategy: Traditional revision

and repeat teaching models2011 – 2015

• Quality instruction in school: YES• Learner support: More structured

with prescribed dosage and tutor payment rates and tutor professional development programme.

• Support strategy: Differentiated pedagogy to support learners from

4 schools - included teacher training

IMPLEMENTATION EVALUATIONIndependent schools 2007 - 2012

Public Schools 2009 - 2015

ASSESSING THE COUNTERFACTUAL FIELDS OF STUDY LEARNERS PURSUE

TIMEFRAMES IN WHICH LEARNERS COMPLETE DEGREES

PROJECT LEGACY

IMPROVED ACADEMIC PERFORMANCE

POST-SCHOOL OUTCOMES EVALUATION

DELIVERY OF LEARNER PROGRAMME TO THE INTENDED TARGET GROUP

IN-SCHOOL OUTCOMES EVALUATION

WHAT WE LEARNT

ACCESSED TERTIARY STUDY AND FINANCE

2011 – 2015Cohort of classmates tested at Gr 8 level,

compared to project cohort. Comparative data was analysed three ways and in each case

there was a positive finding for the project. The effects were not large but significant (despite

project targets not being met).

The retention of project learners in core Mathematics and Physical Science was

a positive project effect.

MATRIC COHORTSIndependent schools: 2009 & 2012

Public Schools: 2009, 2011, 2013 & 2015

INDEPENDENT SCHOOLS

PUBLIC SCHOOLS

PUBLIC SCHOOLSINDEPENDENTSCHOOLS

PUBLIC SCHOOLS

PUBLIC SCHOOLS

INDEPENDENT SCHOOLS

INDEPENDENT SCHOOLS

INDEPENDENT SCHOOLS

PUBLIC SCHOOLS

OVERALL IMPLEMENTATION QUALITY

NSC 2009 – 2010 (2 cohorts, 2 or 3 years out of school)

NSC 2011 – 2013 (4 cohorts; 1, 2 or 3 years out of school)

Sample: 106 out of 241 (44%) spread across 16 of

18 schools

Sample: 159 out of 409 (39%) spread across all 11

schools.

Accessed tertiary study: Although many of the tracked alumni had slow starts to their

studies (usually due to financial obstacles), and had some

changes in direction, by 2012, 78% of the tracked alumni were engaged in study across a wide

range of courses.

Accessed tertiary study: By 2014, 147 out of 159 alumni who had responded (92%) were engaged

with tertiary studies across a range of institutions, with a further 7 (4%)

upgrading their marks in preparation to study. Most of the

alumni were on straight trajectories towards completion of their

degrees with few changes of course of study or institutions reported

Interventions

Placement in Public Schools

Placement in Independent Schools

All Project Learners Independent Schools:

322 learners in two cohortsPublic Schools:

478 learners in four cohorts

SAMPLE

DESIGN TYPE

Evaluation of the High School Learner Programme 2007 - 2015

Evaluation by Quality Projects in Education (QPiE)

Increase the number of learners who obtain Bachelor passes in the NSC with quality passes above 60% in Maths, Science and English.



• Outcome Achievement at tertiary level not in sphere of influence of project.

• Exposing low SES students to effective schooling for even a few years, had merits beyond their improvement in academic

achievement. They went into tertiary study good English language skills, academic study skills, feelings of selfworth and positive attitudes, allowing

them to more easily ‘affiliate with’ academic life.1

1 Hob

den,

S, &

Hob

den,

P. (

2015

). A

stud

y of

the

tran

sitio

n pa

thw

ays o

f sch

ool l

evel

scho

lars

hip

reci

pien

ts in

to w

ork

and

tert

iary

edu

catio

n So

uth

Afric

an J

ourn

al o

f Edu

catio

n, 3

5(3)

.

Learner Support Programme

• Outcome Achievement at tertiary level not in sphere of influence of project

• Adjust targets for STEM-related degree enrolment• Over the project period there was improved

access to variety of funding sources for tertiary education

• Gap between NSC Bachelor pass and tertiary study enrollment needs further investigation

WHAT WE LEARNT

Increase the number of learners who obtain Bachelor passes in the NSC with quality passes above 60% in Maths, Science and English.


Implementation and outcomes evaluation of the Project.

Learners Cohort 2014-2016 from ten Project schools


Evaluation Components

Clarificatory workshop: Refine Project Theory of Change and develop logic model.

Process evaluation: Assessed whether all activities of LP were implemented as intended.

In-school outcomes evaluation: Were outcomes achieved? Can they be attributed to the Project?

Post-school outcomes evaluation: Tracer Study to track progress of alumni who completed NSC in 2012, 2014, 2016.

Evaluation DesignLearner Project Goals

Evaluation by Creative Consulting & Development Works (CC&DW)

Socio-economic background, gender and geographic spread.

Matched group of learners served as a comparison to Zenex LP learners

Counterfactual.

Mixed-methods: Qualitative and quantitative.

CRITICAL SAMPLE VARIABLES

COUNTERFACTUAL

PROPORTIONAL SAMPLE

Face-to-face and telephonic interviews, focus group discussions,

and an online survey.

PRIMARY DATA

PUBLIC SCHOOLS

INDEPENDENT SCHOOLS

Project documents and monitoring records.

SECONDARY DATA

Total Sample

150 91

2012 Grade 9 ANAs matched to learners who completed NSC in 2015

2013 Grade 9 ANAs matched to learners who completed NSC in 2016

ANAs and NSC results systematically matched from combined datasets - a first

Can the results be attributed to the Learner Support Project itself?

Target not achieved: 80% of learners successfully enroll in STEM-related degrees

Target achieved: 80% of learners pursue a tertiary education qualification

Sources of funding:Bursaries: 21%-32% of alumniNSFAS loans: 14%-23% of alumniFamily support: 21%-43% of alumni

Variation between cohorts

2012 cohort: 23% enrolment in STEM2014 cohort: 56% enrolment in STEM2016 cohort: 63% enrolment in STEM

Alumni who responded in the Tracer Study2012 cohort: 100% of alumni2014 cohort: 97% of alumni2016 cohort: 77% of alumni

What would’ve happened to those learners if they’d stayed at their schools?

Delta analysis revealed no statistically significant differences in average change in

Zenex Learner results from ANA to NSC in comparison to average change in matched

controls.

Evaluation unable to assess changes in learner results from ANA to NSC in Science.

ANAs do not assess Science.

Proportion of alumni repeated at least one year of studies. Consistent with

national trends and not abnormal within tertiary education system

in South Africa.

DATASETS

• School and learner context matters• Dosage not achieved: Should be realistic

within context• Implementing agent must be involved in

project design imperative

• Race: YES • Socio-economic: Difficulty in

assessment. Consider Household income +number of siblings. No verification tools

• Educational disadvantage: school location & type not a consistent proxy

• Academic selection: Inconsistently applied. Criteria lowered for placement in public schools - learner backlogs given education disadvantage

Variations in the selection process influenced the Project’s ability to achieve outcomes.

80% of learners achieve at least 60% or above in Maths, Science and English NSC.

Learners who entered the project with a selection test score of under 50% must achieve 10% improvement from selection to NSC.

% of learners who achieved 60% target

EnglishMathsScience

PublicSchools

IndependentSchools

91% 94%

52% 65%

51% 53%

Independent Schools (N=2) Achieved Public Schools (N=37) Percentage of learners who achieved Maths (65%) Science (68%) English (100%)

• Implementation challenges in dosage of after-school and mentorship support components.

• Differentiated pedagogy not applied regularly in after school support and classroom

• School clubs and the Tertiary Access Support implemented as intended.

• Implementing partner worked independently and monitoring data not consistently kept

• Targeted academic intervention, mentorship and the tertiary access support components applied on a needs basis.

• Given selection issues the targets were not feasible • Review Targets annually -set ambitious but

reasonably achievable targets • Investigate benchmarks to set targets for

academic performance

PROCESS EVALUATION

ZENEX TARGETS RESULTS

FIELDS OF STUDY LEARNERS PURSUE

TIMEFRAMES IN WHICH LEARNERS COMPLETE DEGREES

All percentages represent a sample of 126 traced LP alumni

(N=126) 2012 (62) 2014 (64) and 2016 (87)

POST-SCHOOL OUTCOMES EVALUATION

DELIVERY OF LEARNER SUPPORT PROJECT TO THE INTENDED TARGET GROUP?

IN-SCHOOL OUTCOMES EVALUATION

Schools: Independent (4) Public (6): 2014-2016

Independent & Public Schools: 2016

Matric Cohorts: 2012, 2014, 2016

Placement in Public Schools

Placement in Independent Schools

Tertiary Access Support

Evaluation FocusInterventions

Findings and Recommendations

ASSESSING THE COUNTERFACTUAL

ACCESSED TERTIARY STUDY AND FINANCE

WHAT WE LEARNT

WHAT WE LEARNT

IMPROVED ACADEMIC PERFORMANCE

Evaluation of the High School Learner Programme 2014 - 2017

80% of learners passed the NSC with a Bachelor's pass.

Target achieved: 86% of Public School and 88% Independent School learners passed NSC with Bachelor’s pass.

OVERALL IMPLEMENTATION QUALITY