the labour force survey - data quality issuesthe labour force survey – data quality issues matthew...
TRANSCRIPT
The Labour Force Survey – Data Quality Issues
Matthew Steel
• Response rates • Factors • Improving response • Research • Refusal follow-up study • Future work
Data Quality Issues
Response rates
• Falling contact rates • Increasing interview length • Respondents less willing • External shocks • Reduced budgets • Change in survey design
Factors affecting response
• Interviewer training • Incentives • Questionnaire reviews • Varying calling patterns • Survey materials • IT infrastructure
Improving response
• Analysis of attrition bias • Analysis on key estimates • Analysis of paradata • Refusal follow-up study
Research
Refusal follow-up study - Introduction Study:
Eurostat funded project - result of one of the recommendations of the Quality Task Force Review of the LFS, to collect information on the characteristics of non-respondents in order to assess and adjust for non-response bias and to improve fieldwork strategies.
Issue
• Declining response rates:
- August 2011 64% response rate at wave 1 - 10 years ago 75% in wave 1
• Key LFS estimates - sensitive to ethnicity, household type and nationality But – these are not included in the calibration totals
Refusal follow-up study - Introduction
• Therefore, if achieved sample proportions of… - ethnic groups - household groups - national groups… …are not representative of those proportions in… - allocated sample - population… …then the estimates derived from the LFS may be biased.
• RFU survey aimed to:
- identify whether key characteristics of responders and non-responders differentiate - identify if RFU group reasons for refusal differ to general refusing group
Refusal follow-up study – Data Collection
Mainstage RFU exercise: - Main and boost LFS survey - Wave 1 and outright/HQ refusals only - April to June 2011 LFS questionnaire
Data collected in 4 ways… 1. Face-to-face 2. Telephone Unit (wave 1) 3. Survey Enquiry Line (SEL) 4. Self completion
♦ Face-to-face (left at doorstep) ♦ SEL (posted to respondent)
Questionnaire process: - Interviewer/ SEL attempt achieve LFS interview - Not possible, offer RFU as last resort
RFU Letter RFU Questionnaire
Refusal follow-up study – Response Rate
• Total AJ11 outright refusals and HQ refusals = 4,137
• LFS RFU respondents = 787
• LFS RFU response rate = 19 %
• ONS received responses from each of the 4 methods • No interviewer estimate data was allowed, for:
- Ethical reasons - Accuracy/ validity reasons
Refusal follow-up study – Analysis
• LFS collected data on all members of household • LFS RFU collected data:
- Fully on one member of household - Partially on all other members
• Therefore, comparisons between LFS and LFS RFU enabled by… - Selection of one random adult from each LFS responding household - Selection of full data respondent from LFS RFU - Created analysis dataset containing both
Percentage comparison of reasons for RFU and LFS
Refusal follow-up study – Reasons for refusal
Refusal follow-up study - Characteristic comparisons of LFS and RFU respondents
Age
Refusal follow-up study - Characteristic comparisons of LFS and RFU respondents
Sex
Refusal follow-up study - Characteristic comparisons of LFS and RFU respondents
Nationality
Refusal follow-up study - Characteristic comparisons of LFS and RFU respondents
Ethnicity
Refusal follow-up study - Characteristic comparisons of LFS and RFU respondents
Household type
Refusal follow-up study - Characteristic comparisons of LFS and RFU respondents
Workless households
Refusal follow-up study - bias
• Aim? - to determine whether bias exists in LFS data
• How? - Created 1 weight for LFS responders and RFU combined dataset, and - Created 1 weight for LFS responders dataset only (excluding RFU cases)
• Analysis? - Investigated whether proportions in data differed when RFU group included
This indicates: - potential bias in the current estimates from a sample that included these ‘refusers’ in the LFS - or that the current LFS estimates are themselves potentially biased.
Dataset Indicator Status LFSRA only Unemployed No bias
LFSRA plus LFSRFU Employed No bias
LFSRA plus LFSRFU Unemployed Bias
LFSRA plus LFSRFU Inactive Bias
LFSRA only Employed Bias
LFSRA only Inactive Bias
Refusal follow-up study – bias investigations
Refusal follow-up study - Caveats
• LFSRFU data collected from one adult per residence
• LFSRFU sample may not be representative of all ‘refusers’ to LFS
• Not possible to produce standard errors in comparing the two weighted datasets.
• Continue to monitor • Census non-response link study • Regularly review length of survey • Data collection modes • Non-response weights
Future work
Contact details LFS Research Team: [email protected] [email protected] [email protected] Social Survey Data Service: [email protected]
Questions?