developing establishment survey questionnaire design guidelines at the u.s. census bureau rebecca l....

Developing Establishment Survey Questionnaire Design Guidelines at the U.S. Census Bureau

Rebecca L. Morrison

2

Outline

Introduction & Background

Development of the Guidelines

Preliminary Guidelines

Application of Guidelines

Conclusion & Next Steps

4


U.S. Census Bureau:– “…leading source of quality data…nation’s

people and economy”– Tries to minimize burden of data collection

Burden = time, level of effort

Reduce cognitive burden through visual design, structure of data request

5

Introduction & Background (cont’d)

Paper does not focus on specific question wording and order issues.

Paper presents preliminary guidelines in questionnaire design – formatting, layout, navigation, instructions, and data request phrasing and style.

6

Development of the Guidelines

7

Development of the Guidelines: Census Bureau

Agency standards for various survey quality issues

No standard for design of questionnaires

Economic surveys moving toward consistent “look and feel”

8

Development of the Guidelines: ESMS

Establishment Survey Methods Staff– Group of survey methodologists– In-house consultants to economic

programs– Noticed differences across questionnaires– Survey programs work with ESMS on

questionnaire design and pretesting– Catalyst towards consistency

9

Development of the Guidelines: Methods

Questionnaire design principles from standard texts

Special attention to literature on visual design

Based on pretesting a variety of questionnaires with respondents

10

Development of the Guidelines: Considerations

List of design elements to consider

Used to assess tradeoffs within specific survey conditions

Not a cookbook

Preliminary Guidelines

12

Guideline A: Text styles

Be aware of text styles, and how they are used for emphasis within a survey instrument.

13

Guideline A: Example

2004 Annual Survey of Local Government Finances

14

Guideline A: Example

15


These are favored by respondents:– Print item numbers in reverse-print

bubbles, e.g., , , – Print questions in bold black text.– Print instructions in plain text or italics.– Use an 8-point font or larger.

16


Answer spaces:– Open, non-delineated vs. delineated

– Respondents do not seem to have a strong preference

– Be consistent

17


Key codes / Punch codes:– De-emphasize processing codes for

respondents– Print in a darker shade of background

color– Example:

18

Guideline B: Reduce visual clutter. Use natural navigational paths.

Reduce clutter on the page. Use navigational paths and layouts that are natural and readable for respondents.

19


20

Guideline B: Reduce visual clutter. Use natural navigational paths.Format in 1997 Economic Census

confusing:– Lines were “speedbumps” to navigation– Not clear how respondent was to navigate

two adjoined columns

Format changed for 2002– Single column of questions– Generally, a single column of response

options

21


Two columns of questions– Occasionally done when questions:

• Are shorter• Do not involve extensive instructions• Do not ask for numerical information

– Survey of Business Owners (SBO)• Information collected is categorical or ordinal,

not interval or ratio

22


Spread out the questions across more pages

Use bulleted lists

More open space = more “user-friendly”

23

Guideline C: Instructions

Place instructions close to questions, or incorporate them into questions. Where possible, convert instructions into questions.

24


Instructions convey specifications, intent of question

Respondents tend not to pay attention to instructions, or only look when they think they need them

25


2 Goals for instructions:– Eliminate, or reduce, amount of

instructions located separate from question

– Place instructions/information where it is most needed

26


Convert instructions into questions so respondents attend to them– When content critical to correct

interpretation of later questions– When it helps clarify/correct reported data

Example (2004 Medical Expenditure Panel Survey):

27

Guideline D: Avoid sentence fragments and key words

Phrase data requests as questions or imperative statements, not as sentence fragments or key words.

28


Types of data requests– Question (question word, question mark)– Imperative statement (report, enter, add)– Sentence fragment (key words, no verb)

29


Example, 2002 Economic Census:– Is this establishment physically located

inside the legal boundaries of the city, town, village, etc.?

Versus:– Type of municipality where this

establishment is physically located

30


Cognitive interview study: respondents prefer questions over sentence fragments

Survey methodology grad students: questions more effective, imply respondent has to do something

31

Guideline E: Simple vs. complicated questions

Ask additional, simple questions, rather than fewer, more complicated ones.

32


Length and complexity of question affects how long it takes for respondent to understand it

More complicated questions might also be more likely to be double-barreled

May be easier for respondents to answer series of shorter, simpler questions

33


Example: 2002 Industrial Research & Development Survey

34


2006 R&D Survey

35

Guideline F: Use matrices judiciously

Use matrices judiciously, and consider the likely respondents’ background when deciding whether or not to use them.

36

Guideline G: Be consistent!

Items should be numbered in the same way

Bold and italicized print should always mean the same thing

Navigation path should remain constant

Data requests should be in the same form

Variation can be confusing for respondents

37

Application of Guidelines

38

Guideline A: Text Styles

Work in progress among estab surveys at Census Bureau

Economic Census automation required consistent design features– Plain text: questions– Italics: instructions, definitions,

include/exclude lists– Bold: Separating items into categories, sums

of added lines, emphasis within questions

39

Guideline A: Text Styles

Survey of Business Owners:

– Questions: plain– Instructions: bold and/or italics– Emphasis withinquestions: bold and/or italics

40


Survey of Residential Alterations and Repairs (SORAR)– Two columns to one column– Removed or lightened lines– Use of bulleted lists– More open space

41


Old

SORAR

42

Guideline B: Reduce visual clutter. Use natural navigational paths.New SORAR

43

Guideline B: Reduce visual clutter. Use natural navigational paths.Two columns:

Survey of

Business

Owners

44

Guideline B: Reduce visual clutter. Use natural navigational paths.BEA quarterly foreign direct investment (FDI)

Complex navigational path

Crowded text

45

BEA form

revised


46


Economic census continues to use separate instruction sheets and booklets

Two questionnaires moved many instructions to questionnaire:– 2007 Commodity Flow Survey– BEA quarterly FDI form

47


48


Mixed success in application across surveys

49


Commodity Flow Survey uses mix of questions and imperative statements

50


Example:

2002 Survey

of Business

Owners

51


Example: 2007

Survey of Business

Owners

(still in cognitive

pretesting)

52


Example:

BEA

quarterly

FDI

53


54


55


Guidelines are preliminary, based on principles of questionnaire design

Further refinement is necessary before they can be applied to establishment surveys at Census Bureau

56


Other agencies/organizations may develop guidelines that work better for their environment

There is no cookbook for questionnaire design

57

Thank you!

Rebecca L. Morrison

301-763-7595

[email protected]

developing establishment survey questionnaire design guidelines at the u.s. census bureau rebecca l....

Documents

located slide

consistent slide

morrison slide

verb slide

userfriendly slide

consistency slide

cookbook slide

ratio slide