vietnamese youth results. 2 overview of research studies conducted to date cfc programs analyzedcfc...

Vietnamese Youth Results

2

Overview of Research Studies Conducted to Date

CFC Programs Analyzed CFC Sites Province Method Study Type Status Date of Completion Slides2008-2009 HA Hau Giang Observational Longitudinal Completed July 2010 3-8

2012 TH Hau Giang Randomized Controlled Trial Cross-Sectional Completed Nov. 2012 9-162008-2012 HA Hau Giang Discrete Time Survival Analysis Longitudinal Completed Apr. 2013 17-24

2013 TQK, PVD Da Nang Randomized Controlled Trial Cross-Sectional Completed May 2014 25-362013 HA, TH Hau Giang Randomized Controlled Trial Cross-Sectional Completed May 2014 25-362013 TQK, PVD Da Nang Randomized Controlled Trial Longitudinal In Progress 2013 HA, TH Hau Giang Randomized Controlled Trial Longitudinal In Progress 2014 TQK, PVD, NPH Da Nang Randomized Controlled Trial Cross-Sectional In Progress

HA = Hoa An school. TH = Thuan Hung school. TQK = Tran Quang Khai school. PVD = Pham Van Dong school. NPH = Nguyen Phu Huong school.

3

Observational Study, Hoa An Site, Summer 2008-2009 (July 2010)

4


5


6


7


8


9

Randomized Controlled Trial – Thuan Hung Site, Summer 2012 (Nov. 2012)

Completed Pre-Survey (N = 307)

Pre-SurveyMay 17 -- 21, 2012

Treatment (N = 135)

Control (N = 143)

Late (N = 29)

Completed Post-Survey (N = 307)

Lost/Missing (N = 87)

Randomized Offer within School-by-Grade Cluster

CFC ProgramJune 30 – July 22:

9th and 8th Grade Camp

July 14 – Aug. 5: 7th and 6th Grade

Camp

Treatment (N = 65)

Control (N = 43)

Treatment (N = 70)

Control (N = 100)

Post-SurveySept. 13 – Oct. 8,

2012Avg Time Between Pre and Post Surveys

136.08 days (Treatment)

128.33 days (Control)

Participated (N = 108) Did NOT Participate (N = 170)

10


Effect of Intent-to-Treat (Offer): OLS Regression

School-by-Grade Fixed Effects

Covariates: ACAD_PERF_LOW, PARENT_ED, FEMALE, CFC_PRE

Full Sample

Reduced Sample (No 6th Grade)

11


Instrumental Variables Estimation: First Stage

Offer is a significant predictor of program participation

Prior academic performance is also a significant predictor of program participation

12


Effect of Treatment on the Treated: Instrumental Variables Estimation

School-by-Grade Fixed Effects

Covariates: ACAD_PERF_LOW, PARENT_ED, FEMALE, CFC_PRE

Full Sample

Reduced Sample (No 6th Grade)

13


• For a female who participated in the CFC program previously, was in the 7th grade at Thuan Hung School in 2011-2012, and had parents with an average parent education level of 10.45 (between 9th and 10th grade).

Effect of Offer (Intent to Treat)

Effect of Program Participation (Treatment on the Treated)

100 Point Scale Metric

14


• For a female who participated in the CFC program previously, was in the 7th grade at Thuan Hung School in 2011-2012, and had parents with an average parent education level of 10.45 (between 9th and 10th grade).

Participating in the program increases higher educational goals by one grade level, while not participating in the program decreases higher educational

goals by about one grade level.

Original Scale Metric

This effect is observed between 1-2 months after the program.

15


Full Sample

Reduced Sample

• Small to medium effect of offer on the difference in higher education goals after the program compared to before the program

16


• Controlling for school-by-grade cluster, prior academic performance, prior program participation, gender, and average parent education level

Participating in the program leads to an increase in higher educational goals that is 22.9% of the maximum scale score

for those in the reference group (TH6TH).

Participating in the program leads to an increase in higher educational goals that is 25.3% of the maximum scale score

for those in the reference group (TH7TH).

Full Sample

Reduced Sample

17

Dropout Analysis, Hoa An Site, Summers 2008-2012 (Apr. 2013)

Target Population• All low-income, rural Vietnamese adolescents about

to enter 6th grade in fall 2008 or fall 2009

Sample• 351 students about to enter 6th grade at the Hoa An

Lower Secondary School in fall 2008 or fall 2009 (two cohorts).

• To include GPA as a covariate, excluded 13 who dropped out in period 1, as well as 4 people who had missing GPA information.

• Final sample of 334.

46.2%Percentage of the full sample which drop out at some point during middle school

18

RQ#2: Do adolescents who participate in the CFC program have a lower risk of dropout in the semesters following their participation? Does participation in CFC over multiple summers have a greater effect on the relevant hazard probabilities?

RQ#3: Do adolescents who participate in the CFC program have a greater probability of remaining in school past the 9th grade? Does participation in CFC over multiple summers have a greater effect on the relevant survival probabilities?

Discrete Time-Hazard Model which includes the main effect of time, the main effect of 6 th grade fall GPA, and the main effect of total days of CFC participation, assuming a short-term effect (Model 6a)


Discrete time survival analysis, with a completely general specification for time (no intercept):

RQ#1: What is the risk of dropout for rural Vietnamese adolescents in the Hoa An community during each semester of middle school? When are such students most at risk of dropping out of middle school?

Life Table Analysis (Sample Hazard and Survivor Functions) Discrete Time-Hazard Model which includes only the main effect of time (Model 1b)

Differences in Logit--Risk of Dropout by CFC Participation Status

For person iIn period j

19


Life table describing the number of dropouts in each semester of middle school in a sample of rural, Vietnamese adolescents

Risk of dropout varies over time, with the greatest risk occurring in the first semester of 8th grade.

The probability of staying in school declines steadily over time, with only 54% remaining at the end of middle school (9th grade).

Median Lifetime > 8 semesters

20


The discrete time hazard model replicates the life table analysis.

13 students who dropped out in period 1 removed from the dataset because their baseline GPA was 0 by definition

Discrete time hazard model includes data from 334 adolescents, and main effects of P2-P8

21


Controlling for CFC participation in previous summers and 6th grade fall GPA, an adolescent who fully attended a CFC camp (17 days) has odds of dropout in the subsequent academic year that are 79% less than the odds of dropout for an adolescent who did not participate in the CFC camp that summer.

Do adolescents who participate in the Coach for College program have a lower risk of dropout in the semesters following their participation?

~p< 0.10; * p < 0.05; ** p < 0.01; ***p < 0.001

22


Does participation in Coach for College over multiple summers have a greater effect on the relevant hazard probabilities?

By participating in CFC over multiple summers (green and purple lines), students can maintain a lower risk of dropout over many semesters, but the effect of any one camp dissipates.

Given that he did not drop out previously, there is nearly a 30% chance that a student who earned a low GPA in the first semester of 6th grade will drop out in the first semester of 8th grade. However, if such a student attends the 8th grade CFC camp, his risk of dropout in that semester drops to 7.9% (a 73% decrease).

23


The survival probability for three categories of CFC participants substantially exceeds the survival probability of low GPA non-participants (blue line) in semesters after their CFC participation.

On average, 25 more kids will make it through middle school who would not otherwise do so, if they participate in the CFC programming offered in the summers before the 7th and 8th grade.

Do adolescents who participate in the Coach for College program have a greater probability of remaining in school past the 9th grade?

24


The High-Low gap is reduced by 87% when students participate in the 7th and 8th CFC camps, and 95% when students participate in the 6th, 7th, and 8th CFC camps.

The Average-Low gap is reduced by 96% when students complete the 7th, & 8th grade CFC camps and almost completely eliminated when students complete all CFC camps offered in the first three summers of middle school

Period 8

Period 6

Do adolescents who participate in the Coach for College program have a greater probability of remaining in school past the 9 th grade?

Survival Gap in the Absence of CFC

25

Summer 2013 RCT: Comparison of RCT Designs by Province

Delayed Control = First half of crossover design

Procedures

Province Da Nang Hau Giang

Overview

Random Assignment to Treatment Yes YesNature of Random Assignment Within School Class Within School Class Encouragement to Participate Encouragement to ParticipateNo. of Sites 2 2Control Group Intervention Normal Summer Activities Normal Summer ActivitiesTrial Design Delayed Control Parallel GroupSampling Frame Entire Class List Self-Selected VolunteersCFC Camp Dates May 25 - June 29, 2013 June 29 - Aug. 11, 2013No. of Camp Sessions evaluated 4 6No. of Grades 2 3No. of Schools 2 12

Superior Features of Each Trial Design

Minimum Detectable Effect Size (power = 0.80) 0.30-0.32 0.21-0.22No. of School Classes 25 79No. of Students in Study 307 469Average No. of Study Participants Per Class 12.28 5.97Control Group Involvement in Treatment Delayed Participation No ParticipationControl-Group Compliance 99% 90%Treatment-Group Compliance 79% 92%Proportion of Treatment Assessed 67% 100%

Timing of Outcome Assessment Immediately after end of treatment 9-12 weeks after end of treatment

26

Summer 2013 RCT: Measures – Psychological Constructs

Middle School Higher EducationGoals Desired Educational Attainment Expected Educational AttainmentMotivation Intrinsic Academic Motivation College Motivation Extrinsic Academic Motivation College EfficacySelf-Efficacy Academic Efficacy Self-Regulation Self-Regulatory Efficacy

Efficacy-Related

(Expectancy)Motivation-Related

(Value)

Middle School Self-Regulatory EfficacyIntrinsic Academic

Motivation

Academic EfficacyExtrinsic Academic

Motivation

Higher Education College Efficacy College Motivation

Expected Educational

AttainmentDesired Educational

Attainment

For POST (Outcomes) and PRE Surveys (Baseline Covariates):

Cronbach’s alpha for all multi-indicator scales > 0.8

Single Indicator Multi-Indicator Scale

FRAME 1: Educational Period

FRAME 2: Expectancy-Value Theory

Measures

27

Summer 2013 RCT: Measures - Predictors

Descriptive Statistics for Major Predictors in the Study, by Province and Completion of Pre Survey

Province

Da Nang Hau Giang

Variable DescriptionFull

SamplePre

Survey Full

SampleMay Pre Survey

Pre-Camp Survey

Instrument (TOT) and Question Predictor (ITT) OFFER Randomly assigned to Treatment 0.51 0.51 0.59 0.66 0.62

Treatment Status Variables/Second Stage Endogenous Predictors (TOT)

CFC_PARTIC_0 Attended at least some days in the CFC program designated for the Treatment group

0.45 0.45 0.60 0.67 0.62

CFC_PARTIC_80 Attended at least 80% of the days in the CFC program designated for the Treatment group

0.38 0.39 0.51 0.60 0.55

Key Predictors for Interactions

PRE_GPAStudent's cumulative GPA in the 2012-2013 school year 7.68 7.71 7.32 7.37 7.29

(1.13) (1.10) (1.17) (1.13) (1.18)

PRE_CFC_PARTIC_0 Participated at least some days in CFC program(s) before summer 2013 0.00 0.00 0.48 0.54 0.52

Baseline Demographic Covariates

HOUSEHOLD_WEALTHStudent's self-reported household wealth, derived from a composite of indicators given in surveys 51.80 51.80 51.82 51.91 52.66

(15.23) (15.23) (14.68) (14.32) (14.46)

DISTANCE_from_CFCStudent's estimate of traveling time between student's house and relevant CFC site in minutes 15.29 15.29 28.40 27.18 28.17

(10.61) (10.61) (16.87) (15.83) (17.13)

Parent_Ed_MHighest level of education obtained by student's mother 10.59 10.59 9.51 9.51 9.50

(3.95) (3.95) (3.92) (3.82) (3.93)

Parent_Ed_FHighest level of education obtained by student's father 10.63 10.63 10.26 10.24 10.24

(4.11) (4.11) (4.33) (4.21) (4.32)FEMALE Gender is female 0.64 0.66 0.54 0.55 0.54

N Sample Size 307 291 469 281 416

Measures

28

Summer 2013 RCT: Empirical Strategy

Multi-level random intercepts regression, with random intercepts for school class i and student j in class i

Hypothesis #1: Main effect of the CFC program on adolescents’ academic psychology with respect to middle school and higher education.

Students who receive an offer to participate in the program have higher goals, motivation, efficacy, and self-regulation with respect to each educational period after the program (intent-to-treat [ITT]).

Students who participate in the program have higher goals, motivation, efficacy, and self-regulation with respect to each educational period after the program (treatment-on-the-treated [TOT]).

Stage 1:

Stage 2:

OLS

2SLS

Ceiling effects in Hau Giang required Tobit regression

Key

Parameter

Empirical Strategy

TOT effects should be greater than ITT effects:

Positive

29



Hypothesis #2: Interaction between involvement in the CFC program and key exogenous covariates.

There is a negative interaction between the offer of treatment and prior GPA, with the largest intent-to-treat (ITT) effects being observed for those with the lowest GPA.

There is a negative interaction between participation in the CFC program and prior GPA, with the largest effects of participation (TOT) occurring for those with the lowest GPA.

Stage 1a:

Stage 1b:

Stage 2:

OLS

2SLS

Pre-Program GPA (PRE_GPA)

Empirical Strategy

Key Parameter Negative

30



Hypothesis #2: Interaction between involvement in the CFC program and key exogenous covariates.

At sites where the CFC program has been offered previously, there is a negative interaction between the offer of treatment and previous CFC participation, with the largest intent-to-treat (ITT) effects being observed for those who have not participated in any prior CFC camp.

At sites where the CFC program has been offered previously, there is a negative interaction between participation in the program and previous CFC participation, with the largest effects of participation (TOT) being observed for those who have not participated in any prior CFC camp.

Stage 1a:

Stage 1b:

Stage 2:

OLS

2SLS

Previous CFC Participation (PRE_CFC_PARTIC_0)

Empirical Strategy

Key Parameter Negative

31

Summer 2013 RCT: Similarities in Main Effects Between Provinces

Hypothesis 1: Main Effect of the CFC Program on Adolescents’ Academic Psychology Desired Educational Attainment

0 1 2 3 42.00

2.20

2.40

2.60

2.80

3.00

3.20

3.40

3.60

3.80

4.00

ITT and TOT Effects of the CFC Program on Desired Educational Attainment Above Parents' for the Average Student in Each Prov-

ince, Controlling for May Pre Score

Hau Giang

Da Nang

Pre

dic

ted

Gai

n i

n D

esir

ed A

ttai

nm

ent

Received offer Some attendance 80% attendance

ITT Effect TOT Effect

Effect Sizes for Desired Educational Attainment in Each Province, for Students with Average Levels of Baseline Covariates, Controlling for May Pre Score Level of Involvement in CFCProvince Treatment Type Offer Some CFC 80% CFCHau Giang 3 weeks Actual 0.15 0.17 0.19Da Nang 2 weeks Actual 0.14 0.15 0.17

Da Nang 3 weeksEstimate 0.20 0.22 0.26

0.00

5.00

10.00

15.00

20.00

25.00

30.00

The Effect of the CFC Program on the Probability of Desiring to Graduate University, Controlling for

Demographic Covariates and May Pre Score

Hau Giang - % Increase Da Nang - % Increase

Hau Giang - % Point Diff Da Nang - % Point Diff

Per

cent

age

Received offer Some attendance 80% attendance

Results

32

Summer 2013 RCT: Differences in Main Effects Between Provinces

Largest Effects for Motivation Outcomes in Hau Giang

Effect Sizes for the Average Student in Hau Giang Province, by Take-up of Randomized Offer,Controlling for May Pre-Score

Level of Involvement in CFC Treatment Received

Offer Some CFC

Participation 80% CFC

ParticipationLevel Construct Dependent variable

Middle School Motivation Intrinsic Academic Motivation 0.22 0.23 0.26

Middle School Motivation Extrinsic Academic Motivation 0.19 0.21 0.23

Higher Education Motivation Desired Attainment Above Parents 0.15 0.17 0.19

Middle School Efficacy Academic Efficacy 0.10 0.14 0.16

Higher Education Motivation College Motivation 0.04 0.05 0.05

Middle School Efficacy Self-Regulatory Efficacy -0.01 0.00 0.00

Higher Education Efficacy College Efficacy -0.02 -0.03 -0.03

Higher Education Efficacy Expected Attainment Above Parents -0.09 -0.09 -0.10

Hypothesis 1: Main Effect of the CFC Program on Adolescents’ Academic Psychology

Comparison of Predicted Probabilities of Desiring and Expecting to Graduate University at Post Test for Hau Giang Students, Controlling for Demographic Covariates and May Pre Score

Level of Involvement in CFC Treatment During Summer 2013

Offer (Logit) Some CFC (ivProbit) At Least 80% CFC (ivProbit)

Dependent Variable C T T - C (T - C)/C OR N Y Y - N (Y - N)/N N Y Y - N (Y - N)/N

Setting Pre-Score to Average

Wish to graduate from university 0.72 0.83 0.11 15.6% 1.94 0.70 0.83 0.13 18.7% 0.70 0.85 0.14 20.2%

Expect to graduate from university 0.77 0.76 -0.01 -0.8% 0.97 0.77 0.73 -0.03 -4.3% 0.77 0.73 -0.03 -4.4%

Students with Goals Less Ambitious Than University Graduation Before the Program


Expect to graduate from university 0.53 0.52 -0.01 -1.7% 0.97 0.53 0.49 -0.04 -7.8% 0.53 0.49 -0.04 -8.1%

C = Control. T = Treatment. N = No. Y = Yes. OR = Odds-ratio.

Results

33

Summer 2013 RCT: Differences in Main Effects Between Provinces

Largest Effects for Higher Education Outcomes in Da Nang

Hypothesis 1: Main Effect of the CFC Program on Adolescents’ Academic Psychology

Effect Sizes for the Average Student in Da Nang Province, by Takeup of Randomized Offer, Controlling for May Pre-Score

Level of Involvement in CFC Treatment Received

Offer Some CFC


ParticipationLevel Construct Dependent variable

Higher Education Efficacy College Efficacy 0.15 0.17 0.19

Higher Education Motivation Desired Attainment Above Parents 0.14 0.15 0.17


Middle School Motivation Intrinsic Academic Motivation 0.09 0.11 0.12

Higher Education Efficacy Expected Attainment Above Parents 0.02 0.01 0.01


Middle School Efficacy Self-Regulatory Efficacy -0.01 -0.02 -0.02

Middle School Motivation Extrinsic Academic Motivation -0.07 -0.07 -0.09

Comparison of Predicted Probabilities of Desiring and Expecting to Graduate University at Post Test for Da Nang Students, Controlling for Demographic Covariates and May Pre Score

Level of Involvement in CFC Treatment During Summer 2013

Offer (Logit) Some CFC (ivProbit) At Least 80% CFC (ivProbit)

Dependent Variable C T T - C (T - C)/C OR N Y Y - N (Y - N)/N N Y Y - N (Y - N)/N

Setting Pre-Score to Average


Expect to graduate from university 0.56 0.64 0.08 13.5% 1.37 0.55 0.66 0.11 20.2% 0.55 0.67 0.12 22.6%

Students with Goals Less Ambitious Than University Graduation Before the Program


Expect to graduate from university 0.30 0.37 0.07 23.5% 1.37 0.31 0.42 0.11 34.7% 0.31 0.44 0.12 39.0%

C = Control. T = Treatment. N = No. Y = Yes. OR = Odds-ratio.

Results

34

Summer 2013 RCT: Effects of the CFC Program for Key Subgroups

Hypothesis 2: Interaction between CFC Program and Key Exogenous Covariates Largest Effects for Lower-GPA Students in Da Nang

Predicted Probability of Desiring to Graduate from University, Controlling for May Pre-Score Level of Attendance in the CFC

Some CFC (ivProbit) >= 80% CFC (ivProbit)

Dependent Variable N Y Y - N(Y - N)/

N N Y Y - N(Y - N)/

N Setting Pre-Score to Average

Low Pre-GPA 0.38 0.67 0.29 75.3% 0.39 0.71 0.32 83.6%

Average Pre-GPA 0.60 0.73 0.13 22.0% 0.60 0.74 0.15 24.8%

Students with Pre-Program Goals Less Ambitious Than University Low Pre-GPA 0.16 0.41 0.24 148.9% 0.17 0.45 0.28 168.7%Average Pre-GPA 0.26 0.39 0.13 49.2% 0.26 0.41 0.15 57.2%

Predicted Probability of Expecting to Graduate from University, Controlling for May Pre-Score

Level of Attendance in the CFC

Some CFC (ivProbit) >= 80% CFC (ivProbit)

Dependent Variable N Y Y - N(Y - N)/

N N Y Y - N(Y - N)/

N Setting Pre-Score to Average

Low Pre-GPA 0.32 0.62 0.30 93.7% 0.32 0.66 0.33 103.1%

Average Pre-GPA 0.55 0.66 0.11 20.2% 0.55 0.67 0.12 22.6%

Students with Pre-Program Goals Less Ambitious Than University

Low Pre-GPA 0.16 0.41 0.25 156.2% 0.17 0.46 0.29 174.8%Average Pre-GPA 0.31 0.42 0.11 34.7% 0.31 0.44 0.12 39.0%

-0.05

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

Effect Sizes for Post-Program College Motivation in Da Nang, by Level of CFC Involvement in Summer 2013 and Pre-

Program GPA

Low Pre-GPA Sample Average

High Pre-GPAE

ffec

t S

ize

Offer >0% >=80%

Results

35


Hypothesis 2: Interaction between CFC Program and Key Exogenous CovariatesLargest Effects for First-Time Participants in Hau Giang

Effect Sizes for First-Time Participants in Hau Giang Province, by Takeup of Randomized Offer, Controlling for May Pre-Score

Level of Involvement in CFC Treatment Received Offer

to Participate Some CFC


ParticipationLevel Construct Dependent variable Middle School Motivation Intrinsic Academic Motivation 0.53 0.63 0.74Middle School Motivation Extrinsic Academic Motivation 0.42 0.52 0.61



Higher Education Efficacy College Efficacy 0.12 0.18 0.21Middle School Efficacy Self-Regulatory Efficacy 0.07 0.12 0.14

-0.20

-0.10

0.00

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.80

Effect Sizes for Post-Program Intrinsic Academic Motivation in Hau Giang, by Level of CFC Involvement in Summer 2013

and Previous CFC Participation

First-Time Partic-ipants

Sample Average

Returning Partic-ipants

Eff

ect

Siz

e

Offer Some CFC 80% CFC

First Time - Returning Received Offer

to ParticipateSome CFC

Participation80% CFC

Participation Intrinsic Academic Motivation 0.59 0.70 0.81

Extrinsic Academic Motivation 0.45 0.55 0.64

College Motivation 0.53 0.67 0.75

Academic Efficacy 0.34 0.42 0.49

College Efficacy 0.29 0.36 0.40

Self-Regulatory Efficacy 0.14 0.20 0.23

Results

36


Largest Effects for 9th Grade Students in Hau Giang with Average Levels of Baseline Covariates

Similar effects were observed when participation was defined as at least 80% attendance during the treatment period.

-6.00

-4.00

-2.00

0.00

2.00

4.00

6.00

8.00

10.00

12.00

The Effect of Some Participation in the CFC Program on Higher Education Outcomes in Hau Giang, by

Grade in School

9th Grade 7th & 8th Grade

Pre

dic

ted

Eff

ect

of

Par

tici

pat

ion

on

P

ost

Sco

res

-10.00

-5.00

0.00

5.00

10.00

15.00

The Effect of Some Participation in the CFC Program on Middle School Outcomes in Hau Giang, by Grade in

School

9th Grade 7th & 8th Grade

Pre

dic

ted

Eff

ect

of

Par

tici

pat

ion

on

P

ost

Sco

res

Results

vietnamese youth results. 2 overview of research studies conducted to date cfc programs analyzedcfc...

Documents

th grade slide

thuan hung school

th grade camp treatment

control n

lostmissing n

late n

grade cluster cfc program

postsurvey n