Advancing ParadataParadata and the UK Poverty and Social

Exclusion surveyDavid Gordon & Eldin Fahmy

Townsend Centre for International Poverty ResearchUniversity of Bristol

Web Site http://www.bristol.ac.uk/poverty

Web Site http://www.poverty.ac.uk

Poverty in the UK: Advancing Paradata Analysis and Open Access

University of SouthamptonRos Edwards

Institute of EducationAnn PhoenixHeather ElliottMichelle Cage

University of BristolDavid GordonEldin FahmyKaren BellViliami Fifita

The research aims

1.To improve the measurement of poverty, deprivation, social exclusion and standard of living.

2.To measure the change in the nature and extent of poverty and social exclusion over the past ten years.

3.To produce policy-relevant results about the causes and outcomes of poverty and social exclusion.

Background

Every decade since the late 1960s, UK social scientists have attempted to carry out an independent poverty survey to test out new ideas and incorporate current state of the art methods into UK poverty research.

•1968-69 Poverty in the UK survey (Peter Townsend and colleagues),

•1983 Poor Britain survey (Joanna Mack, Stewart Lansley)

•1990 Breadline Britain survey (Joanna Mack, Stewart Lansley)

•1999 Poverty and Social Exclusion Survey (Jonathan Bradshaw and colleagues) and its 2002 counterpart in Northern Ireland (Paddy Hillyard and colleagues)

•2012 Poverty and Social Exclusion in the UK

Survey Data

Omnibus Survey: Necessities of Life – A systematic random achieved sample of adults (16+) of over 1,900 interviews in Britain and Northern Ireland.

Main Survey: Poverty & Social Exclusion – A follow-up survey to the 2010/11 Family Resources Survey (circa 47,000 households). The main survey in Britain has an achieved sample of over 4,000 households and 9,500 individuals – with approximately 1,000 households in the ‘ethnic’ strata and 1,000 households in Scotland. In Northern Ireland, the achieved sample was just under be 1,000 households and over 2,000 individuals.

The survey was divided into a household questionnaire which was answered by the Household Reference Person (HRP) and an individual questionnaire which was answered by all adult household members (aged 18 and over).

Qualitative Data

British Impoverishment Study, The qualitative study, Life on a Low Income in Austere times sought to provide insights into the experiences of poverty during the recent recession and ongoing programme of austerity. The project collected 62 video/audio testimonies during 2012-2013, in Birmingham, Glasgow and Gloucestershire.

Northern Ireland Family Solidarity Study, a qualitative survey of 50 parents to explore the role of family in coping with poverty.

The study had a material focus – examining the extent to which resources are transferred among family members (both nuclear and extended) - and it also explores family cultures and relationships as factors affecting poverty and social exclusion. This helps to understand the extent and limits of family solidarity

The Bristol Social Exclusion Matrix (B-SEM)

Focus groups

• Exploring public perceptions of deprivation, living standards and social exclusion in the UK today to inform the NatCen Omnibus and PSE Main Stage surveys

Cognitive Interviews

• Qualitative pre-testing of selected indicators for potential inclusion in the PSE Main Stage survey

Survey pre-test

•Conventional survey pilot with interviewer de-briefing and analysis of non-response

Expert review

•Input from team members and UK and International Advisory Groups

Survey paradata

•Behaviour coding; analysis of question timings and interviewer characteristics

2012 PSE-UK Question development and testing

http://www.poverty.ac.uk/content/other-research

Was the interviewee suspicious about the study? 11 Not at all 11 To some extent, yes 11 Absolutely

69%28%4%

How do you rate the interviewee's understanding of the questions?

11 Excellent 11 Good 11 Fair 11 Poor

24%51%24%2%

Overall, how great was his/her interest in the interview? 11 Very high 11 Above average 11 Average 11 Below average 11 Very low

3%27%53%14%4%

How do you rate the reliability of the information provided by the interviewee about their household’s income?

11 Accurate 11 Fair 11 Inaccurate

60%36%4%

The PSEHK survey Interviewers thought that only 60% of respondents provided ‘accurate’ information about their household incomes. This could be problematic in a poverty survey if only income information is used to measure ‘poverty’

Why behaviour coding?

• No consensus on what is involved in adequate testing survey questionnaires

• Pretesting often limited to interviewers perceptions of the response process

• Non-response analysis uninformative about potential for response problems not resulting in item non-response

• Application of systematic coding framework to survey interview transcripts to identify problems questionnaire administration

• Assumes analysis of interview behaviour provides a reliable means of identifying question problems without the need to speak directly to respondents• Assumes that deviations from the paradigmatic, standardised survey interaction are a source of measurement error to be minimised• No consensus on what is coded and how: implications for evaluation of survey questionnaires poorly understood

• Household survey conducted separately in Britain and Northern Ireland by NatCen (Britain) and NISRA. Funded by the ESRC

• Sample of respondents in 2010/11 FRS giving permission to be re-contacted. PSE fieldwork conducted Feb-Sept. 2012 using CAPI with all adults within selected households

• Target interview duration of 50 mins for single person hhld (+30 mins. for each additional adult.

• 23 interviews audio-recorded and transcribed verbatim. Sample selected by the survey organisation from experienced interviewers. Subsequent analysis of 111 question items

About the PSE-UK Main stage survey design

•

• Observations are Resp*Qitems. In theory 2,533 observations (23*111)• 1,689 valid observations in total. Selected respondents provided codeable

responses to between 62 and 84 of the 111 items (mean=73.4; sd=6.3)

Behaviour coding framework

Interview problem codes: frequency counts (%)

Respondent problem codes: frequency counts (%)

Respondent problems by interviewer behaviourRow percentages

Mean number of question problems by interviewer and respondent characteristics

Interviewer and respondent problems by question item: standardised mean scores

• How useful (if at all) are different questionnaire pretesting techniques?

• To what extent do they give comparable results? What are the main differences in the results they give?

• How should we explain and respond to inconsistent results between methods?

NEXT STEPS…

•Comparison of behaviour coding with interaction analysis of textual data

•Comparison of PSE-UK behaviour coding with:

– Cognitive interview question testing

– Survey response and question timing paradata

– ‘Traditional’ survey pilot results and expert review

Concluding thoughts