thursday, 19 february 2009ntts2009, 18-20 february 2009, brussels1 getting data for (business)...
TRANSCRIPT
Thursday, 19 February 2009 NTTS2009, 18-20 February 2009, Brussels 1
Getting Data for (Business) Statistics:
What’s new? What’s next?
Ger Snijkers
Statistics NetherlandsUtrecht University
Thursday, 19 February 2009 NTTS2009, 18-20 February 2009, Brussels 2
Getting Data for Business Statistics
How do we get the data we needfor business statistics?
Yesterday, today, tomorrow
Data• In time• Complete• Correct
Statistical picture of a country
NSI
Survey Parameters
in and out of control
Respondent
Parameters
Thursday, 19 February 2009 NTTS2009, 18-20 February 2009, Brussels 3
Getting Data for Business Statistics
Over the years:1. Yesterday: ICES-I* 1993
ICES-II 2000CASM** 1980’s
2. Today: ICES-III 2007• Challenges and developments• A few examples
3. Tomorrow• What’s next ?
* International Conference on Establishment Surveys** Cognitive Aspects of Survey Methodology
Thursday, 19 February 2009 NTTS2009, 18-20 February 2009, Brussels 4
Getting Data for Business Statistics Yesterday
ICES-I (1993):
1. Surveying various branches of industry:agriculture, energy, health care, trade, finance, education, manufacturing industry
2. Quality of business frames & sampling
3. Data analysis & Estimation
4. Data collection methodology:data quality, registers, non-response, Q-design
‘Stove-pipe’ approach Single-mode survey designs
Thursday, 19 February 2009 NTTS2009, 18-20 February 2009, Brussels 5
Response• In time• Complete• Correct
De
cis
ion
to p
articip
ate
An
sw
erin
g b
eh
av
iou
r
Motivation
Respondentburden
• De facto • Perception
Internal business factors• Policy• Data• Resources• Market position
Informant:• Mandate• Data knowledge• Job priority
External business factors• Econ. climate• Regulatory requirements• Political climate
The survey:• Topic• Population and sample• Sponsor / Survey organisation• Resources• Planning• Authority/confidentiality
The survey design
Co
nta
cts
trateg
y
Qu
es
tion
naire
Mo
de
s of d
atac
olle
ction
NSI
Black box
A business
Pa
pe
r
Da
ta W
E w
an
t
Le
tters:M
an
dato
ry
Survey designs not coordinated:• ‘Stove-pipe’ approach
NSI
Single mode
Thursday, 19 February 2009 NTTS2009, 18-20 February 2009, Brussels 6
Getting Data for Business Statistics Yesterday
CASM (started in 1980’s; USA, Germany):
Cognitive Aspects of Survey Methodology• From simple stimulus-response model to
modelling Question-Answer Process:- comprehension- retrieval- evaluation- response
• Pre-testing facilities
Thursday, 19 February 2009 NTTS2009, 18-20 February 2009, Brussels 7
Getting Data for Business Statistics Today
ICES-III (2007):
1. Survey data collection methodology:• questionnaire design & pre-testing • survey participation: non-response reduction,
response burden, bias • mixed-mode designs & e-data collection• understanding the response process in bus’s
2. Using administrative data
3. Business frames & Sampling
4. Weighting, Outlier detection, Estimation & Data analysis
Thursday, 19 February 2009 NTTS2009, 18-20 February 2009, Brussels 8
Response• In time• Complete• Correct
De
cis
ion
to p
articip
ate
An
sw
erin
g b
eh
av
iou
r
Motivation
Respondentburden
• De facto • Perception
• More than one survey• More than once• In other ways: ○ Registers ○ EDI
Internal business factors• Policy• Data• Resources• Market position
Informant:• Mandate• Data knowledge• Job priority
External business factors• Econ. climate• Regulatory requirements• Political climate
Image
The survey:• Topic• Population and sample• Sponsor / Survey organisation• Resources• Planning• Authority/confidentiality
The survey design
Co
nta
cts
trateg
y
Qu
es
tion
naire
Mo
de
s of d
atac
olle
ction
NSI NSI
Registerdata
Statistical picture of a country
Black box
A business
Thursday, 19 February 2009 NTTS2009, 18-20 February 2009, Brussels 9
Getting Data for Business Statistics Over the years
General picture:• 1993:
• 2007:
• ‘Stove-pipe’ approach• Single-mode designs• Survey organisation is central
• Systematisation andstandardisation of methods
• Towards multi-source/mixed-mode designs• Respondent is central: tailoring
• 2000: • Transition
Thursday, 19 February 2009 NTTS2009, 18-20 February 2009, Brussels 10
Getting Data for Business StatisticsThe data collection design today
Challenges:• Good statistics:
• relevant• more & integrated information• faster
• Less money• Less compliance costs:
• providing data only once to government
• New technologies:• powerful computers, access to the internet
Consequences for the data collection …
Thursday, 19 February 2009 NTTS2009, 18-20 February 2009, Brussels 11
Getting Data for Business StatisticsThe data collection design today
• Use of administrative data:• Coordination of definitions:
- variables- units
• Quality of register data:- timeliness
• Data collection without questionnaires:• EDI: XBRL• GPS
• Surveys:• If other sources are not possible or insufficient
• Process measurement and quality control• Getting insight in the data collection process
Thursday, 19 February 2009 NTTS2009, 18-20 February 2009, Brussels 12
Getting Data for Business StatisticsThe data collection design today
• Surveys:• Sampling:
- controlling for overlap across surveys - controlling for rotation over time (survey holiday) one statistical business register
In order to avoid this:
Thursday, 19 February 2009 NTTS2009, 18-20 February 2009, Brussels 13
Getting Data for Business StatisticsThe data collection design today
• Surveys:• Sampling:
- controlling for overlap across surveys - controlling for rotation over time (survey holiday) one statistical business register
• Mode:- Mixed-mode designs: paper, internet, CATI- Computer-assisted
• Questionnaires for web data collection:- Customization (tailoring)- Controlling the completion process (routing, checks)
Thursday, 19 February 2009 NTTS2009, 18-20 February 2009, Brussels 14
Getting Data for Business StatisticsThe data collection design today
• Surveys:• Contact strategy:
- Mixed-mode: .. paper letters, brochures, telephone, .. e-mails, website information
- Message: .. Cooperation = mandatory!.. What, how, who, when?
- Cooperation no longer taken for granted:.. Motivating and stimulating respondents:
. Cialdini: Compliance (persuasion) principles
. Dillman: Social Exchange Theory
- Two-way communication via the internet
Thursday, 19 February 2009 NTTS2009, 18-20 February 2009, Brussels 15
Getting Data for Business StatisticsThe data collection design today
• Process measurement and quality control:• Paradata – process data:
- Macro paradata (survey process data):.. Process summaries:
response rates, timeliness of response,quality of response over time
- Micro paradata (process data at R level):.. Completion process: audit trails
Thursday, 19 February 2009 NTTS2009, 18-20 February 2009, Brussels 16
Getting Data for Business StatisticsMacro paradata
• Timeliness of response (Monthly Survey)
120
117
114
111
108
105
102
99
96
93
90
87
84
81
78
75
72
69
66
63
60
57
54
51
48
45
42
39
36
33
30
27
24
21
18
15
12
9630
responstijd afgerond in dagen
10000
8000
6000
4000
2000
0
fre
qu
en
tie
Responstijd voor papieren vragenlijsten
210
aantal keer gerappelleerd
120
117
114
111
108
105
102
99
96
93
90
87
84
81
78
75
72
69
66
63
60
57
54
51
48
45
42
39
36
33
30
27
24
21
18
15
12
9630
responstijd afgerond in dagen
10000
8000
6000
4000
2000
0
fre
qu
en
tie
Responstijd voor html-vragenlijsten
210
aantal keer gerappelleerd
Paper (letter + Q) Online (e-mail + e-Q)
Num
ber
of r
espo
nses
Days Days
Reminder 1 Reminder 1
Reminder 2 Reminder 2
Thursday, 19 February 2009 NTTS2009, 18-20 February 2009, Brussels 17
Getting Data for Business StatisticsMacro paradata
• R-indicator to monitor fieldwork of business surveys• The representativity of the Monthly Survey for
industry and retail trade by number of fieldwork days.
0,5
0,6
0,7
0,8
0,9
1
0 10 20 30 40 50 60 70
Days
R-in
dica
tor
0,0
0,1
0,2
0,3
0,4
0,5
0 10 20 30 40 50 60 70
Days
Max
imum
abs
olut
e bi
as
Retail
IndustryIndustry
Industry
Retail
Retail
Thursday, 19 February 2009 NTTS2009, 18-20 February 2009, Brussels 18
Getting Data for Business StatisticsMicro paradata – audit trails
• Completion process e-SBS: conscientious R
0
100
200
300
400
500
600
700
800
0:00 0:17 0:22 0:30 1:31 1:55 2:02 2:06 2:19 2:26 2:42 2:53
time (hour:min)
'qu
es
tio
n n
um
be
r'
session no.
day no.
help
info
calculator
save
send
action
Thursday, 19 February 2009 NTTS2009, 18-20 February 2009, Brussels 19
Getting Data for Business StatisticsMicro paradata: audit trails
• Completion process e-SBS: quick ‘n’ dirty R
0
100
200
300
400
500
600
700
800
00:00 02:17 02:57 03:46 03:51 04:02 04:16
time (min)
'qu
esti
on
nu
mb
er'
session no.
day no.
help
info
calculator
save
send
action
Thursday, 19 February 2009 NTTS2009, 18-20 February 2009, Brussels 20
Completion Min. Max. Average
N times started 1 37 1.7
Completion time 00:01:27 11:29:51 01:07:30
Used functionalities of the Questionnaire
Used by % of R’s
How many times used
mean min.-max.
Print button 43 1.7 1-21
Save button 28 2.6 1-95
Thursday, 19 February 2009 NTTS2009, 18-20 February 2009, Brussels 21
Getting Data for Business Statistics The data collection design today
More complex than yesterday:
• More data sources• Dependent on providers of registers• Integration of sources
• Mixed-mode surveys• Coordinated developments over modes• Tailoring to mode
• Tailoring to respondents• Tailoring to target populations • Coordination over surveys (samples and Q’s)
Tomorrow, even more complex
Thursday, 19 February 2009 NTTS2009, 18-20 February 2009, Brussels 22
Getting Data for Business Statistics Tomorrow
• Multi-source/mixed-mode data collection • Managing integrated sets of statistics (not stove-pipes)
• Advanced statistical modelling and estimation• Coordinated data collection designs: - not single-purpose, but multi-purpose surveys• Advanced questionnaire design:
- images, spoken language, animations, video pictures • Methodologists: competent in all modes
• Opening the survey process • Process measurement and quality control:
- continuous measurement using paradata - responsive adaptive designs• Tailoring to the internal business’s processes • Improved communication with businesses
• Opening the businesses• Insight in the internal response processes
Thursday, 19 February 2009 NTTS2009, 18-20 February 2009, Brussels 23
Getting Data for Business Statistics What’s next?
• Opening the businesses• Insight in the response processes
A Business CASM movement: Communicative Aspects of Business Survey Methodology
• Communication sciences• Administrative sciences• Organisational sciences• Psychology (organisational, work and social, cognitive)
Thursday, 19 February 2009 NTTS2009, 18-20 February 2009, Brussels 24
Response• In time• Complete• Correct
De
cis
ion
to p
articip
ate
An
sw
erin
g b
eh
av
iou
r
Motivation
Respondentburden
• De facto • Perception
• More than one survey• More than once• In other ways: ○ Registers ○ EDI
Internal business factors• Policy• Data• Resources• Market position
Informant:• Mandate• Data knowledge• Job priority
External business factors• Econ. climate• Regulatory requirements• Political climate
Image
The survey:• Topic• Population and sample• Sponsor / Survey organisation• Resources• Planning• Authority/confidentiality
The survey design
Co
nta
cts
trateg
y
Qu
es
tion
naire
Mo
de
s of d
atac
olle
ction
Statistical picture of a country
NSI NSI
Registerdata
Black box
A business
Thursday, 19 February 2009 NTTS2009, 18-20 February 2009, Brussels 25
Getting Data for Business Statistics Communication model
Direct communication
Indirect communication
Image NSI
Response• in time, • complete,• correct
Decision toparticipate
One coherent strategywith regard to
tone-of-voice, lay-out, andcompliance principles
Communication we cannot control
Thursday, 19 February 2009 NTTS2009, 18-20 February 2009, Brussels 26
Referencesin addition to proceedings paper
Bethlehem, J., F. Cobben, and B. Schouten (2008), Indicators for the Represen-tativity of Survey Response. Presentation at the 24th International Methodology Symposium of Statistics Canada: “Data Collection: Challenges, Achievements and New Directions”, 28-31 October 2008, Gatineau, Canada.
De Nooij, G. (2008), Representativity of Short Term Statistics. Statistics Netherlands, The Hague.
Groves, R.M. (2008), Dynamic Survey Design managed by modelled Paradata. Presentation at the 24th International Methodology Symposium of Statistics Canada: “Data Collection: Challenges, Achievements and New Directions”, 28-31 October 2008, Gatineau, Canada.
Scheuren, F. (2001), Macro and Micro Paradata for Survey Assessment. Urban Institute: unpublished paper, Washington D.C., USA.
Snijkers, G. (2007), Collecting Data for Business Statistics: Yesterday, Today, Tomorrow. Presentation at 56th Meeting of the ISI, 22-29 August 2007, Lisbon, Portugal.
Snijkers, G. (2008), Getting Data for Business Statistics: A Response Model for Business Surveys. Presentation at the 4th European Conference on Quality in Official Statistics, 8-11 July 2008, Rome, Italy.