using longitudinal administrative data to evaluate area-based initiatives george smith & david...
TRANSCRIPT
Using longitudinal administrative data to evaluate area-based
initiatives
George Smith & David McLennan
Outline
Background Brief history Example 1: Evaluating the Kent Supporting
Independence Programme (SIP) Example 2: Evaluating the national Neighbourhood
Nurseries Initiative (NNI) Example 3: Evaluating the New Deal for
Communities (NDC) Future developments? (Not covering ethical, legal, contractual and access
issues)
Background: Using Administrative Data for Social Research
Administrative data: data primarily collected for administrative purposes, converted for use as a research tool
Such data traditionally released in an aggregated format but increasingly available in individual format
Increasingly linkable laterally (across different datasets) and over time (but still a very long way short of what is possible in Denmark and Sweden).
Key strengths: potentially virtual universal coverage of many of these datasets, and regular (sometimes continuous) updating; also may be virtually the only way to study some social phenomena
Weaknesses: may only contain data that is needed for administrative purposes and there may be ad hoc changes in criteria or definitions (through policy or admin changes).
Administrative data and survey data: very powerful combination?
Brief history – where are we coming from?
Phase 1: late 1980s: impact of national 1988 welfare benefit changes assessed by using an individual claimant extract from an LA housing benefit (HB) system taken before the reforms to estimate the post 1988 reform ‘gains and losses’.
Phase 2: mid 1990s: local longitudinal studies (e.g. Lone Mothers Moving On and Off Benefit), using linked HB extracts to plot benefit dynamics. Such data increasingly well postcoded (and linkable through LA specific ‘HBref’).
Phase 3: 1999 onwards: access by SDRC to national data systems at individual level (e.g. DSS/DWP) for the new Index of Multiple Deprivation (ID2000), led on to longitudinal linkage via encrypted NINOs e.g. for the SEU (e.g. Changing Fortunes , 2001, Growing Together or Growing Apart? 2002, which drew on individually linked welfare benefit data over 5 years, to give both national and local trends).
Phase 4: 2002 onwards: using such linked datasets to assess local programmes. Key strengths of such data are that it is already collected in a consistent form nationally. Is typically more than 99% postcoded and can be linked over time.
IMD work gave access to a very wide range of individual level admin data (welfare benefits, employment, education, health, crime). Strong focus on local areas but within a setting where we needed consistent information across all areas (a prime requirement for the IMD).
Previous experience with attempts to evaluate local area based programmes suggests that unless programmes are highly focussed and targeted, any effects are likely to be very small.
Example 1: Evaluating the Kent ‘Supporting Independence Programme’ (SIP)
Study commissioned by HM Treasury 2002-5 (and follow up to 2006/7) to evaluate impact of Kent’s SIP programme as part of the local Public Service Agreement (LPSA) - Target 12 ‘to reduce dependency and increase employment .. across the whole of Kent with an initial focus on particular districts’.
SIP was essentially an umbrella programme of local initiatives more or less targeted at Kent’s ‘priority wards’ (initially in East Kent) and later expanded to 33 wards in all 12 districts. These had a claimant population of about 18,000 in 2001.
The SIP programme had already begun by the time the evaluation was commissioned; it was a series of loosely targeted initiatives, in some cases with new facilities e.g. increased childcare; in other cases with advice centres or enhanced coordination between local and national agencies.
Some of the key outcomes could be directly measured using national welfare benefits data. So an opportunity to test the use of admin data in a local evaluation. Was it possible?
The Kent LPSA Evaluation
Stage 1: Baseline of ‘benefit spend’, based on data extract for April 2001, for all wards and districts in Kent. Numbers of cases and actual ‘spend’.
Stage 2: Repeated cross-sectional extracts at yearly intervals of major national welfare benefits (IS, JSA. IB/ SDA), allowing trends over time for the target and other areas.
Stage 3: Monitoring individual change in claiming patterns over time, and tracking geographical mobility on benefit.Key to uncovering how far area change is affected by
movement into or out of benefit by claimants. Also effects of geographical mobility on claiming patterns
Stage 4: Comparing the pattern of change by claimants in the Kent priority wards, with the pattern of change in the rest of the South East of England over the same periodModelling the pattern of exits for a cohort followed from 2001-2005 on IS, JSA, IB and SDA.
Some interim results
Limited number of explanatory variables in the admin data, or available from other sources at the local area level.
No clear cut overall effects, though possible improvement in 2003 as claimants in the most highly targeted wards had exit rates in line with other areas in the SE; also slightly higher than expected exit rates for some groups in these areas e.g. younger lone parents with one child under five.
Admin data comes up with numbers large enough to identify such groups, even at a local level. But problem of attributing this apparent change to the programme, without more evidence of a direct link. Issue may not be simply ‘does it work? But for whom, when and where?’
Further two years of data analysed to explore these patterns further. Report in preparation.
Evaluating the Neighbourhood Nursery Initiative (NNI)
NNI – national programme to increase childcare in the most disadvantaged areas (approximately 45,000 places in about 1400 centres by 2005)
Evaluation ( by groups at Oxford, Nat/Cen and IFS). covered implementation, quality of provision, neighbourhood change, and the impact of NNI provision on entry to employment,. Study made extensive use of admin data, including longitudinal DWP benefit data, WPLS and DfES PLASC data.
NNI did not serve tightly specified areas. So necessary to identify potential target areas.
NNI Rich and NNI Poor Areas
Identified all census output areas (OAs) in the 20% most disadvantaged Super Output Areas (SOAs) using the IDAC measure (itself generated from admin data).
Using individual level DfES PLASC data, analysed the distance travelled to primary school by children aged 5-7 in each area to establish the accessibility criterion for an NNI nursery.
Used this information to identify disadvantaged areas with reasonable access to NNI provision and places (‘NNI rich’) and those without (‘NNI poor’).
Propensity score matching techniques used to identify two matched groups of families with preschool children drawn from the child benefit system resident in NNI rich and NNI poor areas. Samples invited to take part in a screening survey to assess their intentions to (re)enter the labour market and make use of childcare.
Subsequent impact study (Nat/Cen and IFS) drew on the ‘work ready’ subset but was able to compare results from four different sources; those actually using NNI provision; their own self assessment of impact; ‘work ready’ families in NNI rich and poor areas, and the overall population of families with young children in these two areas. This included drawing on WPLS to check on the subsequent entry to labour market outcomes. The advantage is that the study includes both the effects on users and at the community level. One key point that emerged that even in the most highly targeted areas (NNI rich) the take up of NNI childcare was around 10% at best. So any effect on ‘users’ is significantly reduced when looking ‘community wide’. Report to be published shortly.
Tackling worklessness in NDC areas
Worklessness
Worklessness:“involuntary exclusion from the labour market and receipt of selected out-of-work benefits”
UnemploymentJob Seekers Allowance (JSA)
Work-limiting illnessIncapacity Benefit (IB)Severe Disablement Allowance (SDA)
Work and Pensions Longitudinal Study (DWP)
Tackling worklessness
Has the NDC programme reduced worklessness in the 39 partnership areas?
(1) Area level change:Reduction in area-level worklessness rates?
(2) Individual level change:Increased likelihood of NDC residents moving
out of worklessness and into employment?
Cross-Sectional Worklessness
24 of the 39 partnerships saw a reduction in the absolute percentage point ‘gap’ with local authority worklessness rate between 1999 and 2005
31 out of 39 saw a reduction in absolute gap with local authority on unemployment rate
4 out of 39 saw a reduction in absolute gap with local authority on work-limiting illness rate
So are 31 NDCs successfully tackling unemployment?
Unemployment example
Change in unemployment rate could be due to…
Improvement for area?
Improvement for unemployed people living in
area?
Unemployed people in the NDC area moving into jobs
Unemployed people in the NDC becoming unable to work due to illness
New people moving into the NDC area who are not unemployed
Unemployed people moving out of the area but remaining unemployed
Unemployment example
Change in unemployment rate could be due to…
Improvement for area?
Improvement for unemployed people living in
area?
Unemployed people in the NDC area moving into jobs
Yes
Unemployed people in the NDC becoming unable to work due to illness
New people moving into the NDC area who are not unemployed
Unemployed people moving out of the area but remaining unemployed
Unemployment example
Change in unemployment rate could be due to…
Improvement for area?
Improvement for unemployed people living in
area?
Unemployed people in the NDC area moving into jobs
Yes Yes
Unemployed people in the NDC becoming unable to work due to illness
New people moving into the NDC area who are not unemployed
Unemployed people moving out of the area but remaining unemployed
Unemployment example
Change in unemployment rate could be due to…
Improvement for area?
Improvement for unemployed people living in
area?
Unemployed people in the NDC area moving into jobs
Yes Yes
Unemployed people in the NDC becoming unable to work due to illness
No
New people moving into the NDC area who are not unemployed
Unemployed people moving out of the area but remaining unemployed
Unemployment example
Change in unemployment rate could be due to…
Improvement for area?
Improvement for unemployed people living in
area?
Unemployed people in the NDC area moving into jobs
Yes Yes
Unemployed people in the NDC becoming unable to work due to illness
No No
New people moving into the NDC area who are not unemployed
Unemployed people moving out of the area but remaining unemployed
Unemployment example
Change in unemployment rate could be due to…
Improvement for area?
Improvement for unemployed people living in
area?
Unemployed people in the NDC area moving into jobs
Yes Yes
Unemployed people in the NDC becoming unable to work due to illness
No No
New people moving into the NDC area who are not unemployed
Yes
Unemployed people moving out of the area but remaining unemployed
Unemployment example
Change in unemployment rate could be due to…
Improvement for area?
Improvement for unemployed people living in
area?
Unemployed people in the NDC area moving into jobs
Yes Yes
Unemployed people in the NDC becoming unable to work due to illness
No No
New people moving into the NDC area who are not unemployed
Yes No
Unemployed people moving out of the area but remaining unemployed
Unemployment example
Change in unemployment rate could be due to…
Improvement for area?
Improvement for unemployed people living in
area?
Unemployed people in the NDC area moving into jobs
Yes Yes
Unemployed people in the NDC becoming unable to work due to illness
No No
New people moving into the NDC area who are not unemployed
Yes No
Unemployed people moving out of the area but remaining unemployed
Yes
Unemployment example
Change in unemployment rate could be due to…
Improvement for area?
Improvement for unemployed people living in
area?
Unemployed people in the NDC area moving into jobs
Yes Yes
Unemployed people in the NDC becoming unable to work due to illness
No No
New people moving into the NDC area who are not unemployed
Yes No
Unemployed people moving out of the area but remaining unemployed
Yes No
Longitudinal analyses
Work and Pensions Longitudinal Study enables individuals to be tracked over time:
into, out of and between benefits and employment
geographically when claiming benefits
Enables analysis of the individual level dynamics driving area level changes
NDC Worklessness ‘inflows’ and ‘outflows’
Programme impact?
Has the NDC programme increased the likelihood of people leaving workless? And entering employment?
Need to track people over time and control for individual, household and area level factors to ‘isolate’ any programme effect
Propensity score matching and Difference-in-Difference modelling using WPLS
Previous results
NDC Evaluation Phase 1 results (June 2005):
Unemployed NDC residents were 1.1 times more likely to exit workless benefits compared to control group
NDC residents unable to work due to limiting illness were 1.6 times more likely to exit workless benefits compared to control group
But… Used GMS-ONE Unable to track people into employment Control group matched across whole of England
Current developments
NDC Evaluation Phase 2 analysis:
Using WPLSTracking people into employmentControl group matched to people in similarly deprived local
neighbourhoods
Results forthcoming: spring/summer 2007
Next Steps / Future Developments
Next Steps: WPLS
1. Add evaluations: E.g. Neighbourhood Renewal Fund
NRF areas
Non-NRF areas
Future possibilities: WPLS
2. Compare (people within) programmes:
NRF areas
EmploymentZones
NDC areas
Possible outcome measures include:
Likelihood of:1. Exiting to employment2. Exiting to sustained employment3. Returning to workless benefits
Future possibilities : WPLS
3. Find cumulative impact of people and place programmes
EmploymentZones NRF areas
New Deal25+
Future possibilities: WPLS
4. Find cumulative impact of combinations people and place programmes
No programme
People based programme Place based
programmeBoth
programmes
Future possibilities:Multi-Source Data Linkage
5. Assessing the impact of change in one outcome measure on change in another- Linking data on the same person
Access to Higher Education (UCAS)
Pupil Level Annual School Census &National Pupil Database (DfES)
Post 16 Qualifications, such as ILR, SERAP, NVQD (DfES)
Future possibilities:Multi-Source Data Linkage
6. Assessing the impact of change in one outcome measure on change in another- Linking data on the same person
Work and Pensions Longitudinal Study
Health data: e.g. GP Prescription Data?
Future possibilities:Multi-Source Data Linkage
7. Assessing the impact of change in one outcome measure on change in another- Linking data on different people
Work and Pensions Longitudinal Study(DWP)
Pupil Level Annual School Census & National Pupil Database (DfES)
Child Benefit records (HMRC)
Conclusion
Huge increase in interest and usage of longitudinal linkage of administrative data
Invaluable for unpicking individual-level dynamics driving area-level changes
Invaluable for small area analyses – surveys rarely robust at neighbourhood level
Great deal of untapped potential – especially for targeting, monitoring and evaluating interventions to tackle social disadvantage