display matters: a test of visual display options in a web-based survey
DESCRIPTION
Surveys are increasingly being conducted online, and it is pertinent to establish clear guidelines for presenting self-administered survey items on a computer screen. Goals should include reducing respondent burden and measurement error. Web survey designers often need to decide how to best present long lists of information and where to place navigation buttons (i.e., next and previous). We evaluated the usability of web-based National Survey of College Graduates (NSCG) questionnaire prototypes. In the first round of testing (n=8), respondents had difficulties proceeding through the survey because the ‘Next’ button was on the left side of the screen and the ‘Previous’ button was on the right. In the second round of testing (n=30), four versions of the survey were tested to assess usability of the ‘Next’ and ‘Previous’ buttons based on their placement on the screen (right or left side) and additionally, to assess the display format of a long list of occupation options (one-column versus two-columns). Dependent measures included participants’ comments from a think aloud protocol, self-reported ratings of satisfaction with the survey, responses to qualitative debriefing questions, time required to complete the job code item, and eye-tracking data focusing on which button (i.e., ‘Next’ or ‘Previous’) and which half of the job code list participants looked at first and more often. Qualitative and quantitative results revealed that participants preferred and performed quicker when the long list of occupation options was displayed in two columns rather than one and when ‘Next’ was displayed to the right of the ‘Previous’ button rather than vice versa. Our findings support usability best practice guidelines that recommend eliminating excessive scrolling on Web sites, and following reading conventions (i.e., looking to the right to move forward, as if turning a page in a book).TRANSCRIPT
Display Ma*ers: A Test of Visual Display Op6ons in a
Web-‐Based Survey Jennifer C. Romano Bergstrom1, Jennifer M. Chen1,
Timothy R. Gilbert2 & Ma* Jans1
1 Center for Survey Measurement 2 Demographic Surveys Division
U.S. Census Bureau
AAPOR 66th Annual Conference May 13, 2011
Current Survey Environment
• Increasing number of surveys online • Design considera6ons – Naviga6on methods
– Presenta6on of response op6ons
2
Current Survey Environment
• Increasing number of surveys online • Design considera6ons – Naviga6on methods
– Presenta6on of response op6ons
3
Background on Next and Previous
• Next should be on the leU – Reduces the amount of 6me to move cursor to primary naviga6on bu*on (Couper, 2008)
– Frequency of use (Dillman et al., 2009; Faulkner, 1998; Koyani et al., 2004; Wroblewski, 2008)
4
Background on Next and Previous
• Previous should be on the leU – Web applica6on order
– Everyday devices – Logical reading order
5
Background on Next and Previous
• Previous should be on the leU – Web applica6on order
– Everyday devices – Logical reading order
6
Background on Next and Previous
• Previous should be below Next – Bu*ons can be closer (Couper et al., 2011; Wroblewski, 2008)
7
Background on Long Lists
• One column – Visually appear to belong to one group – When there are two columns, 2nd one may not be seen (Smyth et al., 1997)
• Two columns: Double banked – No scrolling – See all op6ons at once – Appears shorter
8
Measuring “Best” Design
• Typical: In the Field – Drop-‐off rates – Keystrokes – Survey comple6on 6mes
• Our Study: In the Lab – User sa6sfac6on – Eye-‐tracking data – Usability metrics
9
Usability • The extent to which a product can be used by specified users to achieve specified goals with effec6veness, efficiency, and sa6sfac6on. ISO/TR 16982:2002
10
• For web-‐based surveys, the design must – Meet respondents’ needs – Facilitate easy comple6on – Provide a sa6sfying experience – Reduce respondent burden – Produce high-‐quality data
Na6onal Survey of College Graduates (NSCG)
• Collects educa6on and job informa6on • Respondents have Bachelor’s degree • Was available in PAPI and CATI
• Usability study for a web-‐based self-‐administered instrument
11
Method
• Lab-‐based usability study • TA read introduc6on and leU le*er on desk • Separate rooms • R read le*er and logged in to survey • Think Aloud (Olmsted-‐Hawala et al., 2010) • Eye Tracking • Sa6sfac6on Ques6onnaire • Debriefing
12
Par6cipants
Gender N Age N Educa.on N
Male 14 < 30 8 Bachelor’s 21
Female 16 31-‐45 7 Master’s 6
46-‐60 10 Ph.D. 3
> 60 5
Mean: 46
13
14
Eye-‐Tracking Apparatus
Ques6ons Eye Tracking Can Answer
• Do respondents look at Next and Previous? • What do they look at first?
• Is it distrac6ng when Previous is located in a par6cular place on the screen?
• How long does it take respondents to see the Next bu*on?
• Does presenta6on of long lists affect what users look at on the list?
15
Previous and Next Bu*ons
16
One Column vs. Two Columns
17
4 Versions
N_P1 Next bu*on on leU, 1-‐column job code
N_P2 Next bu*on on leU, 2-‐column job code
PN1 Previous bu*on on leU, 1-‐column job code
PN2 Previous bu*on on leU, 2-‐column job code
18
Results: Sa6sfac6on I
19
* p < 0.0001
Results: Sa6sfac6on II
Overall reac6on to the survey: terrible – wonderful. p < 0.05.
Informa6on displayed on the screens: inadequate – adequate. p = 0.07.
Arrangement of informa6on on the screens: illogical – logical. p = 0.19.
Forward naviga6on: impossible – easy. p = 0.13.
20
6
6.5
7
7.5
8
8.5
Mean N_P PN
Mean Sa.sfac.on
Ra
.ng
6
6.5
7
7.5
8
8.5
Mean N_P PN
Mean Sa.sfac.on
Ra
.ng
6
6.5
7
7.5
8
8.5
Mean N_P PN
Mean Sa.sfac.on
Ra
.ng
6
6.5
7
7.5
8
8.5
Mean N_P PN
Mean Sa.sfac.on
Ra
.ng
Eye Tracking: Next / Previous
21
Eye Tracking: Previous / Next
22
Eye Tracking: N_P vs. PN
• Par6cipants looked at Previous and Next in PN condi6ons
• Many par6cipants looked at Previous in the N_P condi6ons – Consistent with Couper et al. (2011): Previous gets used more when it is on the right
23
Eye Tracking: Time to First Fixa6on
Mean 6me to first look at the naviga6on bu*on
24
4
4.5
5
5.5
6
6.5
7
7.5
8
Next Previous
Second
s
PN
N_P
N_P vs. PN: Respondent Debriefing
• N_P version – Counterintui6ve – Don’t like the “bu*ons being flipped.” – Next on the leU is “really irrita6ng.” – Order is “opposite of what most people would design.”
• PN version – “Pre*y standard, like what you typically see.” – The loca6on is “logical.”
25
1 Column vs. 2 Column
26
Time to First Fixa6on
* p < 0.01
27
0
5
10
15
20
25
First half of list Second half of list
Second
s
1 col
2 col
Total Number of Fixa6ons
28
0
5
10
15
20
25
30
35
40
First half of list Second half of list
Num
ber of Fixa.
ons
1 col
2 col
Time to Complete Item
29
0
20
40
60
80
100
120
Mean Min Max
Second
s
1 col
2 col
1 Col. vs. 2 Col.: Debriefing
• 25 had a preference – 6 preferred one column
• They had received the one-‐column version
– 19 preferred 2 columns • 7 had received the one-‐column version • Prefer not to scroll • Want to see and compare everything at once • It is easier to “look through,” to scan, to read • Re one column, “How long is this list going to be?”
30
Conclusions
• Par6cipants were more sa6sfied when Previous was on the leU.
• Par6cipants preferred the long lists in two columns.
• Par6cipants looked at the first half of the list sooner than the second half when in one column.
• Par6cipants looked at the second half of the list more when it was in two columns.
31
Bigger Picture: Recap on Next and Previous • Next should be on the leU
– Reduces the amount of 6me to move cursor to primary naviga6on bu*on
– Tab order – Frequency of use
• Previous should be on the leU – Web applica6on order – Everyday devices – Logical reading order – People are more sa6sfied – It takes longer to first look at Previous when on the right
32
Bigger Picture: Recap on Long Lists
• One column – Visually appear to belong to one group
• Two columns: Double banked – No scrolling – See all op6ons at once – Appears shorter – Second column may not be seen – People look at the second half more – People look at the first half sooner when it is in one column
– People prefer two columns
33
Future Direc6ons
• This is just a small nugget. • N_P vs. P_N study in progress – Same layout
– No skip pa*erns – Efficiency measure
• Long list of items condi6on – Which items do people pick? – Alphabe6zed vs. random order
34
Thank you!
For more informa6on, please contact Jennifer Romano Bergstrom
Twi*er: @romanocog
35