Display Ma*ers: A Test of Visual Display Op6ons in a
Web-‐Based Survey Jennifer C. Romano Bergstrom1, Jennifer M. Chen1,
Timothy R. Gilbert2 & Ma* Jans1
1 Center for Survey Measurement 2 Demographic Surveys Division
U.S. Census Bureau
AAPOR 66th Annual Conference May 13, 2011
Current Survey Environment
• Increasing number of surveys online • Design considera6ons – Naviga6on methods
– Presenta6on of response op6ons
2
Current Survey Environment
• Increasing number of surveys online • Design considera6ons – Naviga6on methods
– Presenta6on of response op6ons
3
Background on Next and Previous
• Next should be on the leU – Reduces the amount of 6me to move cursor to primary naviga6on bu*on (Couper, 2008)
– Frequency of use (Dillman et al., 2009; Faulkner, 1998; Koyani et al., 2004; Wroblewski, 2008)
4
Background on Next and Previous
• Previous should be on the leU – Web applica6on order
– Everyday devices – Logical reading order
5
Background on Next and Previous
• Previous should be on the leU – Web applica6on order
– Everyday devices – Logical reading order
6
Background on Next and Previous
• Previous should be below Next – Bu*ons can be closer (Couper et al., 2011; Wroblewski, 2008)
7
Background on Long Lists
• One column – Visually appear to belong to one group – When there are two columns, 2nd one may not be seen (Smyth et al., 1997)
• Two columns: Double banked – No scrolling – See all op6ons at once – Appears shorter
8
Measuring “Best” Design
• Typical: In the Field – Drop-‐off rates – Keystrokes – Survey comple6on 6mes
• Our Study: In the Lab – User sa6sfac6on – Eye-‐tracking data – Usability metrics
9
Usability • The extent to which a product can be used by specified users to achieve specified goals with effec6veness, efficiency, and sa6sfac6on. ISO/TR 16982:2002
10
• For web-‐based surveys, the design must – Meet respondents’ needs – Facilitate easy comple6on – Provide a sa6sfying experience – Reduce respondent burden – Produce high-‐quality data
Na6onal Survey of College Graduates (NSCG)
• Collects educa6on and job informa6on • Respondents have Bachelor’s degree • Was available in PAPI and CATI
• Usability study for a web-‐based self-‐administered instrument
11
Method
• Lab-‐based usability study • TA read introduc6on and leU le*er on desk • Separate rooms • R read le*er and logged in to survey • Think Aloud (Olmsted-‐Hawala et al., 2010) • Eye Tracking • Sa6sfac6on Ques6onnaire • Debriefing
12
Par6cipants
Gender N Age N Educa.on N
Male 14 < 30 8 Bachelor’s 21
Female 16 31-‐45 7 Master’s 6
46-‐60 10 Ph.D. 3
> 60 5
Mean: 46
13
14
Eye-‐Tracking Apparatus
Ques6ons Eye Tracking Can Answer
• Do respondents look at Next and Previous? • What do they look at first?
• Is it distrac6ng when Previous is located in a par6cular place on the screen?
• How long does it take respondents to see the Next bu*on?
• Does presenta6on of long lists affect what users look at on the list?
15
Previous and Next Bu*ons
16
One Column vs. Two Columns
17
4 Versions
N_P1 Next bu*on on leU, 1-‐column job code
N_P2 Next bu*on on leU, 2-‐column job code
PN1 Previous bu*on on leU, 1-‐column job code
PN2 Previous bu*on on leU, 2-‐column job code
18
Results: Sa6sfac6on I
19
* p < 0.0001
Results: Sa6sfac6on II
Overall reac6on to the survey: terrible – wonderful. p < 0.05.
Informa6on displayed on the screens: inadequate – adequate. p = 0.07.
Arrangement of informa6on on the screens: illogical – logical. p = 0.19.
Forward naviga6on: impossible – easy. p = 0.13.
20
6
6.5
7
7.5
8
8.5
Mean N_P PN
Mean Sa.sfac.on
Ra
.ng
6
6.5
7
7.5
8
8.5
Mean N_P PN
Mean Sa.sfac.on
Ra
.ng
6
6.5
7
7.5
8
8.5
Mean N_P PN
Mean Sa.sfac.on
Ra
.ng
6
6.5
7
7.5
8
8.5
Mean N_P PN
Mean Sa.sfac.on
Ra
.ng
Eye Tracking: Next / Previous
21
Eye Tracking: Previous / Next
22
Eye Tracking: N_P vs. PN
• Par6cipants looked at Previous and Next in PN condi6ons
• Many par6cipants looked at Previous in the N_P condi6ons – Consistent with Couper et al. (2011): Previous gets used more when it is on the right
23
Eye Tracking: Time to First Fixa6on
Mean 6me to first look at the naviga6on bu*on
24
4
4.5
5
5.5
6
6.5
7
7.5
8
Next Previous
Second
s
PN
N_P
N_P vs. PN: Respondent Debriefing
• N_P version – Counterintui6ve – Don’t like the “bu*ons being flipped.” – Next on the leU is “really irrita6ng.” – Order is “opposite of what most people would design.”
• PN version – “Pre*y standard, like what you typically see.” – The loca6on is “logical.”
25
1 Column vs. 2 Column
26
Time to First Fixa6on
* p < 0.01
27
0
5
10
15
20
25
First half of list Second half of list
Second
s
1 col
2 col
Total Number of Fixa6ons
28
0
5
10
15
20
25
30
35
40
First half of list Second half of list
Num
ber of Fixa.
ons
1 col
2 col
Time to Complete Item
29
0
20
40
60
80
100
120
Mean Min Max
Second
s
1 col
2 col
1 Col. vs. 2 Col.: Debriefing
• 25 had a preference – 6 preferred one column
• They had received the one-‐column version
– 19 preferred 2 columns • 7 had received the one-‐column version • Prefer not to scroll • Want to see and compare everything at once • It is easier to “look through,” to scan, to read • Re one column, “How long is this list going to be?”
30
Conclusions
• Par6cipants were more sa6sfied when Previous was on the leU.
• Par6cipants preferred the long lists in two columns.
• Par6cipants looked at the first half of the list sooner than the second half when in one column.
• Par6cipants looked at the second half of the list more when it was in two columns.
31
Bigger Picture: Recap on Next and Previous • Next should be on the leU
– Reduces the amount of 6me to move cursor to primary naviga6on bu*on
– Tab order – Frequency of use
• Previous should be on the leU – Web applica6on order – Everyday devices – Logical reading order – People are more sa6sfied – It takes longer to first look at Previous when on the right
32
Bigger Picture: Recap on Long Lists
• One column – Visually appear to belong to one group
• Two columns: Double banked – No scrolling – See all op6ons at once – Appears shorter – Second column may not be seen – People look at the second half more – People look at the first half sooner when it is in one column
– People prefer two columns
33
Future Direc6ons
• This is just a small nugget. • N_P vs. P_N study in progress – Same layout
– No skip pa*erns – Efficiency measure
• Long list of items condi6on – Which items do people pick? – Alphabe6zed vs. random order
34
Thank you!
For more informa6on, please contact Jennifer Romano Bergstrom
Twi*er: @romanocog
35
