what they can’t see can hurt you: improving grids for ... · in 3 different grids, varying by...
TRANSCRIPT
What They Can’t See Can Hurt You:Improving Grids for Online Surveys
on Mobile Devices
Randall K. Thomas, Frances M. Barlas,Patricia Graham & Thomas Subias
GfK Custom Research
Acknowledgements
2
Special thanks go to thefollowing people from GfKResearch for their inspiration,design, dedicated hard work, andinnovative programming skillsrequired to develop the gridalternatives examined in thisstudy:
Jason Knight,Tetyana Shvets,Eugene Kagan, andLiju James
IntroductionScreen real estate is a key determinant ofsuccessful measurement in online surveys. The screen size we use in developing our
surveys on desktops or laptops is often not assmall as what participants use when they takeour online surveys: 25% to 35% of surveys are being started
with a mobile device. 20% to 30% of surveys are being completed
on a mobile device (higher levels of break-offs occur with mobile devices).
3
IntroductionSingle items on a single screen requiring a singleresponse are common in online surveys but arenot very efficient because they…• Take time to read and understand the item
stem and the unique responses• Take time for unique manual effort to select
the appropriate response• Take time to select next to advance to next
question
4
IntroductionThe grid format evolved to be more efficientthan single items across screens. Typically, agrid presents responses in columns with theelements to evaluate in the rows.
• The grid obtains efficiency because:
Manual responding is standardized
Response meanings are standardized
5
IntroductionResearchers and statisticians have somematerially misguided beliefs:
• MYTH 1: Increasing the number of responsecategories from 3 to as much as 11 responsecategories increases true score variancerelative to total variance.
• MYTH 2: Aggregate estimates are improvedby increasing the number of itemsmeasuring the same concept.
6
IntroductionAs researchers developed grids with largenumbers of items and a high number of responses,grids have become less efficient and counter-productive - causing higher levels of participantfatigue and reactance: Break-offs Speeding Non-differentiation – straightlining Lower response rates in subsequent surveys
Time to STOP the Grid Insanity!!!7
IntroductionWith the significant rise in mobile devices usedfor online surveys, especially smartphones,participants often run into two problems withgrids that have many response columns:
If the grid reformats to be readable, not allcolumns can be seen simultaneously, or…
If the grid reformats so that all items andresponse columns are seen simultaneously(by shrinking the display), the items andresponses become unreadable andresponses are harder to make.
8
IntroductionTo improve the grid for mobile presentation, weexplored two possibilities in this study:
• Simplify grid presentation - Simplify thenature of the presentation of grids and/orresponses in order to reduce the total realestate required.
• Make grid measurement more mobilefriendly - Reduce the number of responsecategories to make the survey more mobilefriendly.
9
10
Method
MethodSample was drawn from GfK's probability-basedKnowledgePanel®, the largest web-based paneldesigned to be representative of the U.S.population. Respondents for this panel were first selected
randomly using phone or mail surveymethods, and later interviewed online.
People selected for KnowledgePanel whodidn't otherwise have access to the Internetwere provided access at no cost to them.
11
MethodTo be eligible for this study, participants had tohave both a desktop or laptop computer and oneor both of a smartphone or a tablet. Once selected, participants were randomly
assigned to take the survey on a specificdevice type when selected, which wasconfirmed within the survey.
10,672 were selected to participate 4,555 completed on assigned device About 1,500 completed on each device type
(desktop/laptop, tablet, smartphone)12
Research Question 1:Are there simpler and better gridpresentation formats than theTraditional Grid?
13
Method – Alternative Grid FormatsWe developed 3 different grid questions and randomlyassigned respondents 1 of 4 formats, 3 that were newand we believed would simplify the presentation andresponse***:
1. Traditional Grid - row and column matrix
14
Grid Format – Traditional Grid
15
Method – Alternative Grid FormatsWe developed 3 different grid questions and randomlyassigned respondents 1 of 4 formats, 3 that were newand we believed would simplify the presentation andresponse:
1. Traditional Grid - row and column matrix2. Banked Items - items with the responses
presented horizontally under each element.
16
Grid Format – Banked Items
17
Method – Alternative Grid FormatsWe developed 3 different grid questions and randomlyassigned respondents 1 of 4 formats, 3 that were newand we believed would simplify the presentation andresponse:
1. Traditional Grid - row and column matrix2. Banked Items - items with the responses
presented horizontally under each element.3. Responses on Right - Elements presented along
the left in rows and response options presentedvertically on the right.
18
Grid Format – Responses on Right
19
Method – Alternative Grid FormatsWe developed 3 different grid questions and randomlyassigned respondents 1 of 4 formats, 3 that were newand we believed would simplify the presentation andresponse:
1. Traditional Grid - row and column matrix2. Banked Items - items with the responses
presented horizontally under each element.3. Responses on Right - Elements presented along
the left in rows and response options presentedvertically on the right.
4. Focal element – one element presented at a timewith the responses presented underneath.
20
Grid Format – Focal Element
21
22
Grid Presentation Results
Grid 1 Results – Time to CompleteGrid 1 presented 8 beverages and asked for liking of each using a 3category scale. The traditional grid format took the least amount oftime to complete, the focal element presentation took the most time.
23
Grid 1 Results – Item MeansMean liking of beverages were not substantially affectedby grid format.
24
Grid 2 Results – Time to CompleteGrid 2 asked people to rate the importance of 7 different aspects ofbeverages in choosing one for between meals. Smartphones tooklonger to complete, the Traditional Grid took the least amount of time.
25
Grid 2 Results – Item MeansMean ratings of importance were not affected by Response Formator Device.
26
Grid 3 Results – Time to CompleteGrid 3 presented 8 brands to rate liking using a 3 category scale.Smartphones took longer to complete overall, the traditional gridtook the least amount of time to complete.
27
Grid 3 Results – Item MeansMean ratings of brand liking were not affected due to ResponseFormat or Device.
28
29
Research Question 2:Can we make grid response formatsmore mobile friendly?
Mobile Friendly Response FormatsIn addition to testing alternative designs forgrids, we were interested in making grids moremobile friendly by shortening the responsescales. To do this, we used 4 traditional gridsand randomly assigned either: Mobile-friendly grids – having a smaller
number of responses Mobile-unfriendly grids – having a larger
number of responsesAll responses in the grids were fully anchored(each response had a semantic label).
30
Grid Manipulation – Mobile Unfriendly
Grid Manipulation – Mobile Friendly
33
Mobile Friendly Results
Grid A Results – Time to CompleteGrid A had 5 items on tech attitudes, with an agreementscale, either 7 bipolar categories (unfriendly) or 4unipolar categories (friendly).
34
Grid A Results – Item MeansMeans for the Friendly and Unfriendly scales weredifferent in ways normally found for bipolar and unipolarscales, but were in the same order.
35
Grid A Results – ValiditySomewhat higher validity for friendly scales - correlationsof tech items with total number of tech products ownedwere higher with friendly scale than unfriendly scale.
36
Grid B Results – Time to CompleteGrid B had 5 items evaluating their survey experience,using an agreement scale. The friendly version had 4unipolar responses, the unfriendly had 5 bipolarresponses.
37
Grid B Results – Item MeansAgreement ratings using 4 unipolar versus 5 bipolarresponse format, results in same order.
38
Grid C Results – Time to CompleteGrid C asked participants to rate the importance of various TVfeatures, with either a 5 category unipolar scale (unfriendly) or a 3category unipolar scale (friendly).
39
Grid C Results – Item MeansImportance ratings using 3 versus 5 category unipolarresponse format, results not different, and in same order
40
Grid D Results – Time to CompleteGrid D asked participants to rate the importance of features oftoothpaste in the purchase decision with either a 5 category unipolarscale (unfriendly) or a 3 category unipolar scale (friendly).
41
Grid D Results – Item MeansImportance ratings using 3 versus 5 category unipolarresponse format, results in same order.
42
ConclusionsDo Alternative Grid Presentation Formats Work?In 3 different grids, varying by topic and measurementconcept (importance, liking) we randomly assigned 4 gridformats. As has been found in other areas of research with
measures like sliders, fancy and interactivemeasures often fall short of the efficiency of simplermeasurement techniques. Most of the new gridformats took longer to complete, though noneshowed significantly different means for items fromthe Traditional Grid.
The Traditional Grid appears to function well acrossall devices – though we used only 3 category scalesfor all grids in this comparison.
43
ConclusionsCan Grid Response Formats be Made Mobile Friendly? For 4 grids, we compared longer, more mobile-
unfriendly scales (with more responses) with shorter,more mobile-friendly scales (with fewer responses).Mobile-friendly scales…
• Did NOT substantially alter findings, havingequivalent differentiation capability
• Took less time to complete• In some cases, may be more valid.
Yes, response formats can be made mobile friendly - 3 to 4category unipolar fully labeled scales work well and maybe ideal for measurement in the mobile-oriented future.
44
ConclusionsGenerally, the traditional grid can be improved bydecreasing response categories. Some caveats –1. This was a purposefully mobile study and within a
sample that had access to multiple devices, includingat least 1 mobile device.
2. This study had a higher quality sample (probabilitysample, high cooperation rates). Accidental mobileopt-in participants may respond differently.
3. While these recommendations work nicely for newsurveys under development, changing responseformats that have a history (e.g., multiple waves forwhich there is a historical trend) is difficult, oftenrequiring a bridge study to map the new to the old.
45
