impact evaluation of mathcloud in sri lanka doc… · 7 mc in sri lanka 1 pilot treatment tests...
TRANSCRIPT
Impact Evaluation of MathCloud in Sri Lanka
South Asia Regional Symposium on ICT for EducationShangriLa Hotel, Colombo. February 28, 2018
Bilesha Weeraratne, PhD.
Institute of Policy Studies of Sri Lanka.
7
MC in Sri Lanka
1 Pilot
Treatment
Tests
Implementation
Methodology
Results
Bilesha Weeraratne, PhD.
MC Pilot Project
I 1383 Grade 8 Students
I MathCloud (MC) ran for 7 months - May to Nov. 2014.
I 3 Rounds of testing and surveying of students.
I Pre test and survey in May 2014
I Post Test 1 and survey Oct. 2014
I Post Test 2 and survey Nov. 2014
7
MC in Sri Lanka
Pilot
2 Treatment
Tests
Implementation
Methodology
Results
Bilesha Weeraratne, PhD.
MC Treatment
Control (C) Group
I 674 Students
I 5 days of the week
I All 5 days regular classroom environmentwith a mathematicsteacher
I Faced all 3 tests andsurveys
Treatment (T) Group
I 709 Students
I 5 days of the week
I 2-3 days : computerbased instructions by themath teacher using MCin a computer lab
I 2-3 days : regular classroom environment with amathematics teacher
I Faced all 3 tests and surveys
7
MC in Sri Lanka
Pilot
2 Treatment
Tests
Implementation
Methodology
Results
Bilesha Weeraratne, PhD.
MC Treatment
Control (C) Group
I 674 Students
I 5 days of the weekI All 5 days regular class
room environmentwith a mathematicsteacher
I Faced all 3 tests andsurveys
Treatment (T) Group
I 709 Students
I 5 days of the week
I 2-3 days : computerbased instructions by themath teacher using MCin a computer lab
I 2-3 days : regular classroom environment with amathematics teacher
I Faced all 3 tests and surveys
7
MC in Sri Lanka
Pilot
2 Treatment
Tests
Implementation
Methodology
Results
Bilesha Weeraratne, PhD.
MC Treatment
Control (C) Group
I 674 Students
I 5 days of the weekI All 5 days regular class
room environmentwith a mathematicsteacher
I Faced all 3 tests andsurveys
Treatment (T) Group
I 709 Students
I 5 days of the week
I 2-3 days : computerbased instructions by themath teacher using MCin a computer lab
I 2-3 days : regular classroom environment with amathematics teacher
I Faced all 3 tests and surveys
7
MC in Sri Lanka
Pilot
Treatment
3 Tests
Implementation
Methodology
Results
Bilesha Weeraratne, PhD.
Testing
I 30 Test items in each test
I Pre Test: both 2nd and 3rd academic syllabus
I Post-Test 1 : only 2nd academic content
I Post-Test 2 : both 2nd and 3rd academic content
7
MC in Sri Lanka
Pilot
Treatment
Tests
4 Implementation
Methodology
Results
Bilesha Weeraratne, PhD.
Implementation Experience
I SamplingI Planned for randomization but ended up being purposive
sampling.I Duration
I Planned for 1 years ended up with 7 months.I During intervention -
I shared computers, used MC outside their scheduled MChours.
I MethodologyI Planned for Difference in difference model - but couldn’t
collect all required previous test scores.
7
MC in Sri Lanka
Pilot
Treatment
Tests
Implementation
5 Methodology
Results
Bilesha Weeraratne, PhD.
Impact Evaluation Methodology
Propensity Score Matching (PSM)
I PSM controls for possible selection bias
I Makes T/C more comparable based on observable char.
I Similarity between subjects is based on estimated treatmentprobabilities = propensity scores.
3 versions of Test Scores
I Raw scores
I Standardized scores
I Item Response Theory adjusted Scaled Scores
7
MC in Sri Lanka
Pilot
Treatment
Tests
Implementation
Methodology
6 Results
Bilesha Weeraratne, PhD.
Impact Estimates of Change in Test Scores
Period Pre-Test to Post Test-1 Pre-Test to Post Test-2Outcome Raw Std Scaled Raw Std Scaled
MainATE 1.45*** 0.27*** 1.10*** 0.58* 0.11 0.65***ATT 1.46*** 0.27*** 1.13*** 0.47 0.09 0.59**
Rob 1ATE 0.97** 0.18** 0.35 0.49 0.09 0.37ATT 0.87** 0.16** 0.18 0.36 0.04 0.23
Rob 2ATE 1.38*** 0.25*** 1.08** -0.27 -0.05 0.55ATT 1.62*** 0.30*** 1.09** -0.04 -0.01 0.58
Notes :
Standard errors in parentheses
* p<0.10 ** p<0.05 *** p<0.01
Main Models : Matching based on students’ characteristics
Rob 1: Matching based on students’, teachers’ and school characteristics
Rob 2: Matching based on students’, teachers’ and school characteristics +
Sample restricted to common mathematics teachers for T/C
7
MC in Sri Lanka
Pilot
Treatment
Tests
Implementation
Methodology
6 Results
Bilesha Weeraratne, PhD.
Impact Estimates of Change in Test Scores
Period Pre-Test to Post Test-1 Pre-Test to Post Test-2Outcome Raw Std Scaled Raw Std Scaled
MainATE 1.45*** 0.27*** 1.10*** 0.58* 0.11 0.65***ATT 1.46*** 0.27*** 1.13*** 0.47 0.09 0.59**
Rob 1ATE 0.97** 0.18** 0.35 0.49 0.09 0.37ATT 0.87** 0.16** 0.18 0.36 0.04 0.23
Rob 2ATE 1.38*** 0.25*** 1.08** -0.27 -0.05 0.55ATT 1.62*** 0.30*** 1.09** -0.04 -0.01 0.58
Notes :
Standard errors in parentheses
* p<0.10 ** p<0.05 *** p<0.01
Main Models : Matching based on students’ characteristics
Rob 1: Matching based on students’, teachers’ and school characteristics
Rob 2: Matching based on students’, teachers’ and school characteristics +
Sample restricted to common mathematics teachers for T/C
7
MC in Sri Lanka
Pilot
Treatment
Tests
Implementation
Methodology
6 Results
Bilesha Weeraratne, PhD.
Impact Estimates of Change in Test Scores
Period Pre-Test to Post Test-1 Pre-Test to Post Test-2Outcome Raw Std Scaled Raw Std Scaled
MainATE 1.45*** 0.27*** 1.10*** 0.58* 0.11 0.65***ATT 1.46*** 0.27*** 1.13*** 0.47 0.09 0.59**
Rob 1ATE 0.97** 0.18** 0.35 0.49 0.09 0.37ATT 0.87** 0.16** 0.18 0.36 0.04 0.23
Rob 2ATE 1.38*** 0.25*** 1.08** -0.27 -0.05 0.55ATT 1.62*** 0.30*** 1.09** -0.04 -0.01 0.58
Notes :
Standard errors in parentheses
* p<0.10 ** p<0.05 *** p<0.01
Main Models : Matching based on students’ characteristics
Rob 1: Matching based on students’, teachers’ and school characteristics
Rob 2: Matching based on students’, teachers’ and school characteristics +
Sample restricted to common mathematics teachers for T/C
7
MC in Sri Lanka
Pilot
Treatment
Tests
Implementation
Methodology
6 Results
Bilesha Weeraratne, PhD.
Impact Estimates of Change in Test Scores
Period Pre-Test to Post Test-1 Pre-Test to Post Test-2Outcome Raw Std Scaled Raw Std Scaled
MainATE 1.45*** 0.27*** 1.10*** 0.58* 0.11 0.65***ATT 1.46*** 0.27*** 1.13*** 0.47 0.09 0.59**
Rob 1ATE 0.97** 0.18** 0.35 0.49 0.09 0.37ATT 0.87** 0.16** 0.18 0.36 0.04 0.23
Rob 2ATE 1.38*** 0.25*** 1.08** -0.27 -0.05 0.55ATT 1.62*** 0.30*** 1.09** -0.04 -0.01 0.58
Notes :
Standard errors in parentheses
* p<0.10 ** p<0.05 *** p<0.01
Main Models : Matching based on students’ characteristics
Rob 1: Matching based on students’, teachers’ and school characteristics
Rob 2: Matching based on students’, teachers’ and school characteristics +
Sample restricted to common mathematics teachers for T/C
7
MC in Sri Lanka
Pilot
Treatment
Tests
Implementation
Methodology
7 Results
Bilesha Weeraratne, PhD.
Takeaway Messages
I MC ⇑ scaled scores by approx 3.5 percentage points afterPost Test 1
I MC ⇑ scaled scores by approx 2 percentage points after PostTest 2