the use of administrative data for program evaluation rio
DESCRIPTION
The use of administrative data for program evaluation RioTRANSCRIPT
The use of administrative data for program evaluation
Emanuela GalassoDevelopment Research GroupThe World Bank
The role of Administrative records and complex surveys in the monitoring and evaluation of public policiesRio de Janeiro Nov 2014
Using administrative data: the question
• Administrative data: main objective is monitoring for decision making– can trace and describe trends of key inputs and outcomes of interest
• Question: how can it be used for impact evaluations?
• Complementary to survey data collection
• For either type of data sources, a pre-requisite for embarking in an impact evaluation is to have a clear identification strategy
• Call for expanding access to administrative data prominent US economists
Using administrative data for impact evaluations: the potential
• Size: Universe of the target population
• Best placed to assess programs targeted to very specific groups – to be oversampled in surveys (the extreme poor, migrants), or geographic areas
• By-product of the existing monitoring of the social policy: cost
• time: characterize dynamics of impact
• Flexibility: can potentially look at different target groups, assess new variants
Using administrative data for impact evaluations: the cons
• Data up-to-date? Differential capacity of municipalities or agencies in data collection
• Representativeness? ex.registries of the poor (as CadUnico) covers the entire target population?
• Data quality? Income (imputations), scope for over/under reporting (ex. Colombia PMT)
• Scope: limited set of variables. May be missing key outcomes or determinants of pathway of impact(ex. noncognitive skills, mental health, parenting practices)
Using administrative data for impact evaluation: the experience of Chile Solidario
• Non-experimental evaluation: method of regression discontinuity design
• Planned after the program had started
• Exploited the clear assignment :
• Eligibility based on proxy means score (CAS). Official cutoff by municipality
• Households invited from the bottom up of the CAS distribution in each municipality
• Program gradually rolled out, 4 cohorts 2002-2005, became law in 2004
Chile Solidario: objectives and approach
1. Social inclusion
• Targeting the extreme poor (and vulnerable): bottom 5% CAS. From explicit demand for services to identification and invitation
• Central role of psychosocial wellbeing in social policy: 2 years of psychosocial support, local social worker
Chile Solidario: objectives and approach
2. Social protection
• Short term cash transfer + guaranteed monetary transfers
• Preferential access for social services (demand)
• Making the supply side available (supply) and tailored to the needs of the poorest
• Beyond access to health and education: multiple dimensions and ‘minimum conditions’
Chile Solidario: objectives and approach
3. Promotion
•.Identification of key skills and endowments to sustain exit from poverty (demand)
• Human capital of young and current generation
• Psychosocial capital
• Making supply side available (supply) and tailored to the needs of the poorest
Data: Panel Survey Chile Solidario• CASEN 2003: identify a representative sample of ChS
participants and an observationally ‘comparable’ sample of non-participants • Rich set of outcomes (health, education, housing, income,
poverty/indigence) CASEN + added new modules: intergenerational mobility, psychosocial dimension (since 2006)• Representativeness? cohorts 2002-2003
• Mistake sampling scheme in 2004 sampled better off non-participants
• Followed sample 2004, 2006, 2007
• No baseline data
Data: administrative data
• Panel of the administrative registry: Ficha CAS
• Demographics, housing/durables, employment, income, monetary subsidies
• In 2007 FPS : expanded into attendance to preschool, school, health visits, more detailed employment module,
• Detailed geographical information: 13 regions, 346 municipalities, 7000 neighborhoods.
• Can distinguish families vs households
• Coverage: 1/3 in 2000/1, to 2/3 after 2007
• Ficha vigente – updated every 2 years. Substantial churning families in/out
The key role of the national ID
• Registro Unico Tributario RUT. use it to merge it to a large set of complementary data (SIIS presentation yesterday)
• Merged registry with:
• Administrative data from the program (identity participants)
• administrative data on social workers, caseloads (quality of implementation)
• participation on training/employment programs (unit level records available at the time of the evaluation)
Descriptives: target of the bottom 5% in each municipality led to under-coverage nationwide
Proportion of participant among eligible households:
Cumulative entry around 50-60% of the potential target population
Descriptives: social exclusion was higher among the poorest
Note: Only families eligible for SUF are considered (heads 20-50 years old).
CAS Distribution
SUF take-up rate
regression discontinuity design: ‘effective’ cutoffs
• Exploit gradual roll out of the program over time to estimate effective thresholds that vary by municipality, over time
• inference caveat: window CAS within municipality, larger support nationwide
Participation to the program by cohorts• Eligibility defined by the ‘effective’ cutoff
Main result: take-up 2 years
11% 1.2%
2.3% 4%
Main results: employment 2 years
Going beyond average impacts to understand mechanisms
• Program is tailored to the needs: it is important to account for the initial conditions of beneficiaries
• compare households who had vs had not satisfied outcomes before 2002)
• Program evolved over time (supply side):
• compare cohorts before/after 2004
• In theory with current Ficha Social+ expanded historical information on programs could look at 6-10 years impact
Supply side articulation: employmentPreferential access to:
• Self-employment programs (AAE)
• Employability (PNCL)
Exclusive access to:
• Job placement (wage subsidy SENCE, PROFOCAP)
• Self-employment programs (PAME)
• employability (habilitacion sociolaboral SENCE, competencia laboral mujeres PRODEMU, apoyo jovenes SENCE)
• Share programs exclusive to CS increased over time
• Lion share of programs for self-employment (75%)
Results by initial conditions: take-up• Impact driven mainly by those who were previously
disconnected from social services (SUF)
Results by initial conditions: take-up• Take-up Employment programs driven by those
inactive/unemployed before CHS
Final employment outcomes show the importance of the supply side
• Spouse: activation employment by 20% only for those cohorts exposed to the supply side expansion
To summarize• Social protection: activation of the demand for a large array of
social programs
• Important demand side constraints (information, transaction costs, psychic costs activation)
• Impact persists over time
• Social inclusion: activation demand for programs for those who were previously disconnected
• Take-up of monetary transfers
• Take-up of labor market programs
• Promotion: No. Activation demand does not translate into final outcomes if not met by supply side:
• Need comprehensive activation component: technical and soft skills training, internship, job intermediation and placement
Quality of the social worker associated with better take-up rates and employment
Quantile of SW quality
Average SUF take-up rate
10 0.3602
25 0.5692
50 0.6141
75 0.7019
90 0.8824
Quantile of SW quality
Avg. prop. Head emp.
10 0.5414
25 0.7059
50 0.7222
75 0.8700
90 0.9816
Conclusions• Social registries starting data source for impact evaluation:
move beyond monitoring
• Cross check with unit record data on access to services, use of services)
• Can go to fine level of disaggregation, exploit time dimension
• Might be missing key outcomes (income, psychosocial dimensions)
• Caveat representativeness, quality
• Even programs are already at scale, refine questions of impact: quality of implementation, complementarity of services to target groups, experimentation