scotland’s large scale surveys and survey harmonisation
DESCRIPTION
SCOTLAND’S LARGE SCALE SURVEYS AND SURVEY HARMONISATION. 1 JULY 2008. Scottish Household Survey. Introduction. Started in February 1999, as a result of devolution c.15,000 households per year Two part interview Highest Income Householder or spouse/partner - PowerPoint PPT PresentationTRANSCRIPT
SCOTLAND’S LARGE SCALE SURVEYS AND
SURVEY HARMONISATION
1 JULY 2008
Scottish Household Survey
Introduction
• Started in February 1999, as a result of devolution
• c.15,000 households per year• Two part interview
– Highest Income Householder or spouse/partner• Information about the households in Scotland• Characteristics of HIH help characterise the household as a
whole
– Randomly Selected Adult• Information about adults in Scotland
• 45 minutes long
Topics Covered
• 3 Priority Topic Areas– Transport – Local Government – Communities
• Many other areas– Health & Care– Anti-Social Behaviour– Culture and Sport– Recycling
Reporting
• Can produce – Results for Scotland every quarter– Results for larger LAs every year– Results for the urban/rural classification every
year– Results for all LAs every two years
Survey Data
• The SHS is an important source of data to aid policy development and evaluation
• The survey provides the data for 5 of the SG’s Spending Review 07 indicators:– Neighbourhood Rating– Satisfaction with local services– Smoking– Journeys affected by congestion– Travel to work
Smokers in Scotland
Adults (16+): Percentage who smoke
0
5
10
15
20
25
30
35
1999 2000 2001 2002 2003 2004 2005 2006
Year
% S
mo
kers
Target – 22% by year 2010
Neighbourhood RatingAdults (16+): Pecentage who rate their neighbourhood as a Very Good or Fairly Good place to live
0
10
20
30
40
50
60
70
80
90
100
1 2 3 4 5 6 7 8 9 10
SIMD deciles
%
Very good Fairly good
Recent Changes
• Significant de-clustering of the sample
• Introduction of modularisation
• Consortium of three companies for 2007-2010– Ipsos Mori– TNS– Scottish Centre
Looking forward
• Questionnaire Review for 2009– Harmonisation– Policy Requirements under the new
administration– Improving interview “flow”
Future Plans
• Methodology Developments– Weighting– Income Imputation
• Dissemination & Promotion– Training– New Publications– Web-based analysis
The Scottish House Condition Survey
The SHCS – basics
• 3,000 occupied households every year;
• Simple Random sample of Local Authorities - unclustered
• Minimum of 80 surveys in each Local Authority
• 2-stage Survey• Social Survey; followed by • Physical Survey
• Purpose: Measures the quality of people’s experience of housing against the quality of housing in Scotland;
The SHCS – Basic outputs
Surveyor expertise
-
50
100
150
200
250
pre-1919 1919-1944 1945-1964 1965-1982 post-1982
Age of dwelling
Num
ber
of
dw
elli
ngs
(thousa
nds)
DetachedSemi-detachedTerracedTenementOther flats
The SHCS – Basic outputsSocial Survey expertise
Tenure by age group of highest income householder (%)
0
10
20
30
40
50
60
70
80
16-30 31-44 45-54 55-64 65-74 75+
Grouped age of HIH
Perc
enta
ge
Owner-occupierLA/other publicHA/co-opPrivate renter
The SHCS – Basic outputsSocial and Physical Survey – number of bedrooms and
household relationships, genders and age of residents
3 or more bedrooms above
standard7%
2 bedrooms above standard
19%
I bedroom above standard
39%
Compliance with bedroom standard
32%
Below Standard3%
The SHCS – Key Outputs Domestic Dwellings Energy The SHCS – Key Outputs Domestic Dwellings Energy EfficiencyEfficiencyDwellings becoming more energy efficient – but still a long way to Dwellings becoming more energy efficient – but still a long way to gogo
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
1991 1996 2002 2003/2004
2004/2005
2005/2006 Survey period
Good 7-10
Moderate 3-6
Poor 0-2
% of dwellings
National Home Energy Rating
The SHCS – Key outputs:
Fuel Poverty (Income, dwelling heating characteristics, size and fuel costs)
Hhlds in Fuel Poverty 1996-2005/6 (% and CIs)
1996 2002 2003/42004/5 2005/60
5
10
15
20
25
30
35
40
Survey year
Perc
enta
ge
The SHCS – Key outputs:
Scottish Housing Quality Standard
•be above the statutory Tolerable Standard
•be free from serious disrepair
•be energy efficient
•have modern facilities and services
•be healthy, safe and secure.
The SHCS – Key outputs: Scottish Housing Quality Standard
0
500
1000
1500
2000
2500
No of dw
ellings 000s
2002 2003/4 2004/5 2005/6
>4 Failures
4 Failures
3 Failures
2 failures
1 Failure
0 Failures
The SHCS medical variables:
• 2002 Health Report
http://www.scotland.gov.uk/Resource/Doc/933/0039909.pdf
• Nothing since – handout with current health questions.
• Abandoned Mental health module but in survey in 1991, 1996, 2002, 2003/04, 2004/05 2005/06 and 2007.
• Abandoned Coughing and wheezing but in survey in 1991, 1996, 2002, 2003/04, 2004/05, 2005/06 2007.
• Now have simplified asthma type questions (2008 onwards)
Mental Health Module GH1 Have you recently been able to concentrate on whatever you're doing?
GH2 Have you recently lost much sleep over worry?GH3 Have you recently felt that you are playing a useful part in things?
GH4 Have you recently felt capable of making decisions about things?
GH5 Have you recently felt constantly under strain?
GH6 Have you recently felt you couldn't overcome your difficulties?
GH7 Have you recently been able to enjoy your normal day-to-day activities?
GH8 Have you recently been able to face up to your problems?
GH9 Have you recently been feeling unhappy and depressed?
GH10 Have you recently been losing confidence in yourself?
GH11 Have you recently been thinking of yourself as a worthless person?
GH12 Have you recently been feeling reasonably happy, all things considered?
Coughing and Wheezing Module
HE2 Have you had any wheezing or whistling in your chest at any time in the last 12 months?
HE3 Have you been at all breathless when wheezing noise was present?
HE4 Have you had this wheezing or whistling when you did not have a cold?
HE5 Have you been woken up with a feeling of tightness in your chest at any time in the last 12 months?
HE6 Have you been woken by an attack of shortness of breath at any time in the last 12 months?
HE7 Are you currently taking any medicine (including inhalers, aerosols, or tablets) for asthma?
Coughing and Wheezing Module
HE8 Have you ever had asthma?HE9 Was this confirmed by a doctor?HE10 Have you had an attack of asthma in the last 12 months?HE11 How old were you when you had your first attack of asthma?HE12 How old were you when you had your most recent attack of
asthma?HE13 How many attacks of asthma have you had in the last 12 months?HE14 Do you have any nasal allergies including hay feverHE15 Have you ever suffered a two year period or more during which
you experienced coughing and phlegm on most days?
HE16 And would you say that you suffered that problem for periods of 3 months or more each of those years?
Coughing and Wheezing Module (HE17-26 asked about children in HH)
HE17 Has {sub} had any wheezing or whistling in the chest in the last 12 months?
HE18 How many attacks of wheezing has {sub} had in the last 12 months?
HE19 In the last 12 months how often, on average, has {sub2} sleep been disturbed due to wheezing?
HE20 In the last 12 months has wheezing ever been severe enough to limit {sub}'s speech to only one or two words at a time between breaths?
HE21 Has {sub} been woken by an attack of wheezing in the last 12 months?
HE22 In the last 12 months has {sub} chest sounded wheezy during or after exercise?
Coughing and Wheezing Module (HE17-26 asked about children in HH)
HE23 In the last 12 months has {sub} had a dry cough at night, apart from a cough associated with a cold or chest infection?
HE24 Has {sub} had more than 3 courses of antibiotics for respiratory infections (chest, ears or throat) in the last 12 months?
HE25 Is {sub} currently taking any medicine (including inhalers, aerosols or tablets) for asthma?
HE26 Has {sub} had an attack of asthma in the last 12 months?
Scottish Health Survey
History
• Designed to provide information on population health, health-related behaviour and use of health services in Scotland.
• Unique aspect of survey is nurse visit where measurements such as BP, lung function and blood samples are taken
• Previously run in 1995, 1998 and 2003
• Reviewed in 2005/2006
Review Recommendations
• Continuous
• Core and modular structure
• Scotland results annually, HB results at least every 4 years
• Unclustered sample
• NHS boards able to purchase boosts to their samples
Survey Design
• Annual sample of around 6,400 adults and 2,000 children
• HB minimum sample of 784 after 4 years
• Sample stratified by HB and SIMD
• Unclustered design (after 4 years)
• HBs able to purchase boosts (3 have for 2008)
Survey Design - Structure
• Core and modular structure• Core questions (around 35 mins) to full sample• Two modules of questions (around 10 mins) to
partial sample– Module A – range of topics – some questions being
asked annually, some biennially Given to one third of the full sample
– Module B - Knowledge, Attitudes and Motivations module (previously HEPS) – asked annuallyGiven to two thirds of the full sample but only asked of one adult per household
What Happens Now?
• Contractor appointed July 2007
• 2008 questionnaire finalised end December 2007
• Fieldwork began end January 2008
• First publication planned for summer 2009
Labour Force Survey &
Annual Population Survey
LFS & APS
• Survey of households• Covers:
– Employment hours worked, industry, occupation, earnings
– Unemployment and economic activity – Education and training:
qualifications held and current education/training
- Health problems limiting ability to work
UK-wide LFS
• Quarterly survey (since 1992) • Sample size – 52,000 households in UK
– Approx 8,000 in Scotland (in five ‘waves’)
• Panel design – Individuals stay in sample for five quarters– Each quarter a fifth of sample replaced– Sample contains five ‘waves’ of c.11,000 households
• Unclustered sample – small bias in Scotland: partial coverage north of
Caledonian Canal
UK-wide LFS
• Changed in 2006 from seasonal quarters (Mar-May, Jun-Aug etc) to calendar quarters (Jan-Mar, etc) in line with EU
• Publications – monthly Labour Market First Release (with
regional data)Headline figures employment, unemployment, economic activity
– quarterly supplement to Labour Market Trends
Employment, unemployment and economic inactivity rates by gender, Scotland, 1992-2006
0
10
20
30
40
50
60
70
80
90
1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006
Year
Rat
e
Employment Male Employment Female Employment Male
Unemployment Female Inactive Male Inactive Female
Annual Population Survey
• SG funds an annual boost to LFS in Scotland – From 8,000 (unique households) to 23,000
• Calendar years 2004 to 2006 – plus quarterly rolling annual (Apr-Mar, Jul-Jun etc.)
• More reliable estimates for local authorities• Most headline estimates available each year for
LAs• Annual publication by SG
Future Developments
• Looking at including communal establishments
• Harmonisation of questions – UK Integrated Household Survey
• Pilot of ethnicity boost (this Summer, funded by SG)
Further Information
Data available from Nomis & data archive
Websites
SG:http://www.scotland.gov.uk/Topics/Statistics/Browse/Labour-Market
ONS:http://www.statistics.gov.uk/
Nomis:https://www.nomisweb.co.uk/Default.asp
E-mail: [email protected]
The Scottish Crime The Scottish Crime and Justice Surveyand Justice Survey
About the Scottish Crime and About the Scottish Crime and Justice SurveyJustice Survey
Face to face household surveyFace to face household survey Random sample (some stratification, Random sample (some stratification,
clustering)clustering) Up to 2006:Up to 2006:
– Every 2 to 3 yearsEvery 2 to 3 years– Sample of 5,000Sample of 5,000
2008 onwards2008 onwards– ContinuousContinuous– Sample of 16,000Sample of 16,000
Rationale and limitationsRationale and limitations
Helping fill the gap in official statistics Helping fill the gap in official statistics on crimeon crime– Victim focusVictim focus– Unreported crimeUnreported crime
BUTBUT Private households onlyPrivate households only 16+ only16+ only Language and disability may be Language and disability may be
barriersbarriers
Counting crimeCounting crime
How much crime?How much crime? What type of What type of
crime?crime? How many victims?How many victims?
Prevalence of victimisation 1992 to 2005/06
0
5
10
15
20
25
1995 1999 2002 2003/4 2005/6
Year
Per
cen
t of p
opul
atio
n
Household crimes Personal crimes All SCVS crimes
Measuring attitudesMeasuring attitudes
Once secondary; Once secondary; becoming just as becoming just as important?important?
Keeping policy Keeping policy makers in tune makers in tune with people’s with people’s perceptionsperceptions
Measuring Measuring performanceperformance
Perceptions of change to the local crime rate
Less crime than 2
years ago, 8% More crime
than 2 years ago,
33%
About the same, 59%
SCVS 2006
Counting crime since 1970sCounting crime since 1970s Scottish Crime Survey (SCS), Scottish Crime and Scottish Crime Survey (SCS), Scottish Crime and
Victimisation Survey (SCVS) and Scottish Crime and Justice Victimisation Survey (SCVS) and Scottish Crime and Justice Survey (SCJS) Survey (SCJS)
Face to faceFace to face ‘‘British’ Crime Survey (BCS) since 1982 (now covering British’ Crime Survey (BCS) since 1982 (now covering
England and Wales only)England and Wales only)Face to faceFace to face
Northern Ireland Crime Survey (NICS)Northern Ireland Crime Survey (NICS)Face to faceFace to face
International Crime and Victim Survey/ European Crime and International Crime and Victim Survey/ European Crime and Victim Survey (ICVS)Victim Survey (ICVS)
Mixed methods (phone)Mixed methods (phone) United States National Crime Survey (NCS)United States National Crime Survey (NCS)
PhonePhone
Crime surveys in ScotlandCrime surveys in Scotland
1982 1988 1982 1988 Lowland Scotland covered in the ‘British’ Crime SurveyLowland Scotland covered in the ‘British’ Crime Survey Face to face, pen and paperFace to face, pen and paper19931993 1996 2000 2003 1996 2000 2003 Scottish Crime SurveyScottish Crime Survey Face to face, pen and paperFace to face, pen and paper 5,0005,000
REVIEW in 2003REVIEW in 200320042004 20062006 Scottish Crime and Victimisation SurveyScottish Crime and Victimisation Survey Telephone interviewing (CATI) in 2004 discontinuedTelephone interviewing (CATI) in 2004 discontinued Back to face to face, CAPI/CASI in 2006Back to face to face, CAPI/CASI in 2006 3,000 (2004) and 5,000 (2006)3,000 (2004) and 5,000 (2006)20082008 Scottish Crime and Justice SurveyScottish Crime and Justice Survey
The ‘new’ Scottish Crime and The ‘new’ Scottish Crime and Justice SurveyJustice Survey
A balance – meeting stakeholder needs, but A balance – meeting stakeholder needs, but affordable and using gold standard survey affordable and using gold standard survey methodmethod
Continuous – data every yearContinuous – data every year 16,000 per year – data at police force area 16,000 per year – data at police force area
levellevel Bigger sample – more modules means more Bigger sample – more modules means more
space for questionsspace for questions Face to face household interviews using Face to face household interviews using
CAPI/CASICAPI/CASI In the field in April 2008In the field in April 2008
SCJS todaySCJS today
Cognitive testing of new sections Cognitive testing of new sections Small scale pilot to test script and Small scale pilot to test script and
systemssystemso Main pilot completed February 2008:Main pilot completed February 2008:
o Stage 1 (800 quota)Stage 1 (800 quota)o Stage 2 (200 RPS)Stage 2 (200 RPS)
Next steps….into the field!Next steps….into the field! Data? Data?
Survey Harmonisation in Survey Harmonisation in Scotland Scotland
an overview of the theoretical and the an overview of the theoretical and the practicalpractical
The Background…….The Background…….
• Many Government surveysMany Government surveys• Independent/isolated developmentIndependent/isolated development• One purposeOne purpose
Which meant……Which meant……
• Poor comparability of survey outputsPoor comparability of survey outputs• Confusion for usersConfusion for users• Potential embarrassment for Scottish Potential embarrassment for Scottish
GovernmentGovernment• Timings offTimings off• Duplicated costs Duplicated costs
The Theory………The Theory………
Survey IntegrationSurvey Integration
OROR
Survey HarmonisationSurvey Harmonisation
Survey Integration……Survey Integration……
• Merge 4-5 key surveys into 1 integrated surveyMerge 4-5 key surveys into 1 integrated survey• Ask core set of economic and social questions of whole Ask core set of economic and social questions of whole
samplesample• Ask non-core questions across subsets of the sampleAsk non-core questions across subsets of the sample• Ask specialist topic questions in modules across subsets Ask specialist topic questions in modules across subsets
of the populationof the population• Re-brand component surveysRe-brand component surveys• e.g. ONS Integrated Household Surveye.g. ONS Integrated Household Survey
Drawbacks………
•Costly•Complicated•Case management system•Component Surveys may lose their identity•Slow to disseminate (like the Census)•No way back?
Survey harmonisation………
•Identify key surveys with harmonisation potential
•Embed standardised questions and outputs into each survey
•Each survey should adopt a core and modular structure
•Each survey should demonstrate unclustered sampling
•Policy driven dissemination from larger (pooled) samples
Benefits………
•More flexible than Integrated Survey model
•Surveys retain their unique identities and purpose
•Avoid duplication of effort and costs
•Topic experts e.g. Health Survey Statistician definitive authority on health related questions.
•Singing from same song sheet
•Communication within Scottish Government improved
•Harmonisation potential continues to be identified
Survey harmonisation appears to be the more effective model for Scotland’s large scale surveys although survey integration has not been ruled out.
A decision has yet to be made but in the meantime survey harmonisation continues.
The story so far…….
Main focus has been on Scotland’s 5 large scale Surveys:
•Scottish Household Survey•Scottish House Condition Survey•Scottish Crime and Justice Survey•Scottish Health Survey•Labour Force Survey
Other surveys involved GUS, FRS, SSAS and SEABs.
The Census 2011 is important in the harmonisation process.
What is practical?What is practical?
• Socio-economic coreSocio-economic core• Survey designSurvey design
Harmonisation potential:Harmonisation potential:
• Respondent permission/consent incl. follow-upsRespondent permission/consent incl. follow-ups• Survey letters and leafletsSurvey letters and leaflets• ProcurementProcurement• Complaint handlingComplaint handling• Data Access PanelData Access Panel
Socio-economic coreSocio-economic core
• Established a 20 question core by examining Established a 20 question core by examining common survey questions and important common survey questions and important questionsquestions
• Set up Scottish Harmonisation Working Group Set up Scottish Harmonisation Working Group to review each question and its outputs to to review each question and its outputs to build a harmonised question bank build a harmonised question bank
• Increasing focus on harmonised outputs Increasing focus on harmonised outputs – survey to survey across Scotlandsurvey to survey across Scotland– UK wide UK wide – European European – Comparability with economically similar countries Comparability with economically similar countries
P e o p l e a n d P l a c e s
Gender Age/DOB Marital Status Household Relationships Residence one year ago
I d e n t I t y
Cultural/Ethnic Group Religion
H e a l t h
Limiting long term illness/disability Self perception of general health
H o u s I n g
Tenure Accommodation Type Accommodation Size (no. bedrooms, overcrowding)
E m p l o y m e n t
Economic status (ILO) Full/part time Banded Household Income Attendance on government training scheme
E d u c a t I o n
Education status Highest qualification held
T r a n s p o r t
Access to car Mode of transport to work/education/school
Survey designSurvey design
• Adoption of a core and modular structureAdoption of a core and modular structure
• Unclustered samplingUnclustered sampling
• Methodology Forum Sub-group (MFSG)Methodology Forum Sub-group (MFSG) •samplingsampling
•weightingweighting
• time seriestime series
• sample additivity sample additivity
• omnibus etc.omnibus etc.
Harmonisation Programme Work Flow
SURVEY MANAGERS' NETWORK
----------SCOTTISH POPULATION SURVEYS CO-ORDINATING COMMITTEE
METHODOLOGY FORUM GROUP
SCOTTISH HARMONISATION WORKING GROUP
METHODOLOGY FORUM SUB-GROUP (SURVEY HARMONISATION)
What next?What next?
• Website : Question bank Website : Question bank • Website : Surveys’ Customer Service sectionWebsite : Surveys’ Customer Service section• Promote core questions and survey design to smaller Promote core questions and survey design to smaller
Government surveys, local authorities, health boards etcGovernment surveys, local authorities, health boards etc• Adopt a naming convention for variablesAdopt a naming convention for variables• Administrative Sources …… data linking?Administrative Sources …… data linking?• Work on non-core questionsWork on non-core questions• Continue to add specialist topic modules onto existing Continue to add specialist topic modules onto existing
surveys rather than developing new surveys surveys rather than developing new surveys (e.g. culture and (e.g. culture and sport module)sport module)
• Decommission standalone surveys if they will sit within one Decommission standalone surveys if they will sit within one of the five large scale surveys as a module (e.g. HEPS)of the five large scale surveys as a module (e.g. HEPS)
• Survey Integration or Harmonisation?Survey Integration or Harmonisation?
Issues to considerIssues to consider
• Pooled survey data? Pooled survey data? – What will/can we do with it?What will/can we do with it?– Who will disseminate it?Who will disseminate it?– Will pooled sample be stored in data archive?Will pooled sample be stored in data archive?– Will results from it be classed a National Statistic?Will results from it be classed a National Statistic?– Do we need respondent permission to pool?Do we need respondent permission to pool?– Who/how do we check for double counting?Who/how do we check for double counting?
• Respondent Burden Respondent Burden – Case management system?Case management system?– Fieldwork capacity issuesFieldwork capacity issues