making it fit: how survey technology proviers are responding to the challenges of handling web...
Post on 21-Oct-2014
848 views
DESCRIPTION
Presentation at the ASC International conference, Bristol, 2011. A full written paper is also available from the meaning websiteTRANSCRIPT
Making it fit
How survey technology providers are responding to the challenges of handling web surveys on mobile devices
Tim Macer, meaning ltd, London, UKASC Sixth International Conference, Bristol, 22-23 September 2011
2
Agenda
1. Emerging best practices
2. Requirements identified
3. How are the vendors responding
4. Conclusions and recommendations
Making it fit: web surveys on mobile devices. ASC 2011
1. Emerging best practicesSpecific considerations
Operating systems
App or browser?
3
Making it fit: web surveys on mobile devices. ASC 2011
Multi-mode parallel surveys on web/mobile
Design the survey around single-choice, multiple-choice and open-ended questions alone.
Adopt more concise wording
Limit the survey length to fifteen questions.
Zahariev et al (2010)
4
Making it fit: web surveys on mobile devices. ASC 2011
Specific technical requirements identified…
Identifying which specific smartphones are supported, as only specific makes or models may be supported in some packages.
Verifying that error messages and other components are not dependent on the use of JavaScript.
Having a specific ‘skin’ or page template to format the survey for delivery on a mobile device, which maximised the limited space available for questions.
Zahariev et al (2010)
5
Making it fit: web surveys on mobile devices. ASC 2011
Recommended optimiziations for mobile surveys
Use a concise, well-worded SMS invitation with an integrated link to survey.
To ensure any costs a participant may incur are reimbursed
Avoid complex matrix questions: replace with questions presented one by one.
Encourage active engagement, e.g. taking photographs.
Pick time to send invites in order to optimize response (e.g. between 17.00 and 18.00).
Pferdekaemper (2010)
6
Making it fit: web surveys on mobile devices. ASC 2011
Further recommendations for mobile
Reduce and simplify – avoid horizontal scrolling,
reduce clutter from logos, branding,
consider using drop-down lists rather than radio buttons
Luck (2011) “How to make mobile surveys work for everyone”
Adjust the survey to fit limited the screen of each device Understand the dimension of the device
Capture the browser agent string
Callegaro (2010) “Do You Know Which Device Your Respondent Has Used to Take Your Online Survey?”
7
Making it fit: web surveys on mobile devices. ASC 2011
Current trends in Mobile operating systems in use
OS 2010 2014 trend
Android (Google) 17.7 29.6
iOS (Apple) 15.4 14.9
Research In Motion (BlackBerry) 17.5 11.7
Symbian 40.1 30.2
Windows Phone 4.7 3.9
Other Operating Systems 4.7 9.6
8
Mobile Communications devices: sales to end users by OS Gartner (2010)
Making it fit: web surveys on mobile devices. ASC 2011
What defines an app or a browser survey?
browser
A web survey which has been
formatted so that it will work
effectively in the browser of a
range of mobile devices within
the constraints of these
devices.
app
A survey delivered through a
dedicated application or app
which the participant must first
download onto his/her device
and then accesses via an icon
on the device’s main menu.
9
Making it fit: web surveys on mobile devices. ASC 2011
Browser pros and cons
Uses HTML: achieve high coverage; not restricted to a subset of devices
Participant can engage immediately without having to download anything
Mix formats within one survey instrument: laptop/desktop, tablets, handhelds
✘ Lowest common denominator approach to question types
✘ Not all devices support flash
✘ Integrating media, e.g. uploading photos, can be too difficult for participants to do
✘ Needs a stable internet connection each time “next” is pressed
10
Making it fit: web surveys on mobile devices. ASC 2011
App pros and cons
Does not need a permanent stable internet connection at all times
Can exploit all the programmable capabilities of the device, e.g. diaries with reminders, integrated capture and upload of pictures and video
✘ Download requires high level of trust and engagement from participant
✘ Apps rarely support the entire range of devices out there – reach less of the population
11
Making it fit: web surveys on mobile devices. ASC 2011
2. Identifying the key requirementsEssential needs for browser tools
Essential needs for apps
The survey
12
Making it fit: web surveys on mobile devices. ASC 2011
Essential requirements examined: browser-based tools
1. Automatically detect the device resolution and reformat survey to fit
2. Optimize the presentation of the survey for small-format devices
3. Gracefully unwind grids into a series of individual questions
4. Record device type used in the data
5. Integrated survey invitations and reminders by email and SMS
6. Allow respondent to save and resume later in ‘desktop’ or mobile mode
7. No dependency on mouse controls (shift+click etc)
8. No dependency on using Flash controls
9. Can create specific routing and validation dependent on device type
13
Making it fit: web surveys on mobile devices. ASC 2011
Essential requirements: app-based tools
1. Simple, easy download for respondent
2. Web server-based authoring and deployment environment
3. Capture location data, with respondent permission
4. Capture and upload photos and video
5. Diary surveys: repeated capture of same questions
6. Diary reminder alerts with do not disturb option
7. Online/offline capability
8. Bar code scanning with respondent confirmation
14
Making it fit: web surveys on mobile devices. ASC 2011
…plus, requirements common to both modes
10. Wide range of devices supported
11. Device-specific preview in authoring tool or via simulators
15
Making it fit: web surveys on mobile devices. ASC 2011
Contacted 10 different
technology producers
active in the field
Contacts made in
April 2011 with a simple
electronic questionnaire
7 firms with browser-based solutions
6 firms with apps
5 with both
Eight companies
participated
One refused
One failed to respond
The survey
Making it fit: web surveys on mobile devices. ASC 2011
3. How are the vendors respondingPlatforms supported
Features provided
17
Making it fit: web surveys on mobile devices. ASC 2011
Who is providing what?
Supplier Browser App
Askia ✓ ✓CfMC WebSurvent ✓Confirmit ✓Globalpark EFS ✓ ✓iPinion Surveys ✓ ✓Kinesis ✓ ✓Surveyswipe & Surveypocket ✓ ✓Techneos ✓
18
Making it fit: web surveys on mobile devices. ASC 2011
Browser tools – devices supported
Device iPinion KinesisSurvey- swipe
CfMC AskiaCon-firmit
Global-park
Android ✓ ✓ ✓ ✓ ✓ ✓ ✓
BlackBerry ✓ ✓ ✓ ✓ ✓ ✓ ✓
Symbian ✓ ✓ ✓ ✓ ✓
iOS (Apple) ✓ ✓ ✓ ✓ ✓ ✓ ✓
Windows mobile ✓ ✓ ✓ ✓ ✓ ✓ ✓
WML ✓ ✓
XHTML ✓ ✓ ✓ ✓ ✓
19
✓ Capability currently offered Planned for a Future release
Making it fit: web surveys on mobile devices. ASC 2011
Mobile Apps – devices supported
iPinion KinesisSurvey-pocket
TechneosSODA
Askia mobile
Global-park
Android ✓ ✓ ✓ ✓
BlackBerry ✓ ✓ ✓
Symbian ✓
iOS (Apple) ✓ ✓ ✓ ✓ ✓ ✓Windows mobile
✓ ✓
20
✓ Capability currently offered Planned for a Future release
Making it fit: web surveys on mobile devices. ASC 2011
Web browser solutions
Analyzing the capabilities offered in the market
21
Making it fit: web surveys on mobile devices. ASC 2011
Browser tools – capabilities offered by all vendors
Capability
Mixed mode – full screen and mobile ✓
Detect and reformat to fit device ✓
Optimize for mobile ✓
No dependency on mouse ✓
No dependency on Flash ✓
22
Making it fit: web surveys on mobile devices. ASC 2011
Browser tools – capabilities where support varies
Capability iPinion KinesisSurvey- swipe
CfMC AskiaCon-firmit
Global-park
Unwind grids ✓ ✓ ✓ ✓ ✓ ✓Store device data
• At survey level ✓ ✓ ✓ ✓ ✓ ✓• At question level ✓ ✓ ✓ ✓Invitations, reminders by Email and SMS ✓ ✓ ✓ ✓ ✓ ✓
Preview via simulator ✓ ✓Save and resume ✓ ✓ ✓ ✓ ✓Routing by device ✓ ✓ ✓ ✓
23
Making it fit: web surveys on mobile devices. ASC 2011
Essential requirements fulfilled: browser tools
All offered device auto-detection and optimization for mobile, plus unwinding of grids
Only 2/7 products met all the key capabilities
6/7 supported email and SMS invitations/reminders
5/7 supported save and resume
Capturing of the agent string Most will record device data at a survey level;
Only 4/7 record the device at question level or allow conditional routing on the device type
24
Making it fit: web surveys on mobile devices. ASC 2011
Mobile app solutions
Analyzing the capabilities offered in the market
25
Making it fit: web surveys on mobile devices. ASC 2011
Mobile apps – capabilities offered by all vendors
Capability
Capture location data, with respondent
permission✓
Capture and upload photos and video ✓
Support for Android, BlackBerry and
Apple iOS✓
26
Making it fit: web surveys on mobile devices. ASC 2011
Mobile Apps – download method supported
iPinion KinesisSurvey-pocket
Techneos SODA
Askia mobile
Global-park
Simple 1-click from a web link
✓ ✓ ✓ ✓ ✓
From an integrated panel portal
✓ ✓ ✓ ✓ ✓
From app site or App Store
✓ ✓ ✓ ✓ ✓
27
Making it fit: web surveys on mobile devices. ASC 2011
Mobile Apps – capabilities where support varies
iPinion KinesisSurvey-pocket
Techneos SODA
Askia mobile
Global-park
Web browser authoring and deployment 1 ✓ ✓ ✓ ✓
Device-specific preview ✓ 2 ✓
Diary survey mode ✓ ✓ ✓ ✓
Diary alerts (regulated) ✓ ✓ ✓
Online/offline ✓ ✓ ✓ ✓
QR/Bar code scanning ✓ ✓ 1Authoring via either Confirmit or iBM SPSS Dimensions design module2via 3rd party simulator
28
Making it fit: web surveys on mobile devices. ASC 2011
Essential requirements fulfilled: Apps
6/6 will capture location data, photos or video, all under respondent control
6/6 offered a range of one-click download options
5/6 would operate without a constant internet connection
4/6 offer diary support and the other 2 are developing it Diary reminders are only fully supported (with the option for
participants to disable them) by 4 of the 5.
QR/bar codes supported by 4 but with 2 more developing
29
Making it fit: web surveys on mobile devices. ASC 2011
Discussion
Conclusions
Recommendations
30
Making it fit: web surveys on mobile devices. ASC 2011
Observations
Some vendors have specialized in serving the browser or the app-based solution, but many cater for both
All the browser solutions we examined will integrate seamlessly with standard online surveys (auto-detecting the device)
Most but not all vendors support a wide range of devices iOS, Android and Blackberry universally supported
Symbian only offered by 5/7 browser and 5/6 app products
XHTML supported by just 5/7 browser tools and the old (and very restrictive) WML/WAP by only 2.
31
Making it fit: web surveys on mobile devices. ASC 2011
Observations
Models of best practice are still developing
Previewing and simulating how the survey will appear on different small-format devices is really important Only 2/7 browser tools and 2/6 apps did this,
However, two more in each category plan to support this soon
Software developers are not necessarily taking advantage of the capabilities that mobile devices offer
32
Making it fit: web surveys on mobile devices. ASC 2011
Mobile research is still at an early stage of development
Support still rudimentary
Only two of the solutions met all of the key capabilities
Wide variation in what is offered by different vendors
platforms supported, key features etc
But developing fast
Most of the vendors recorded key capabilities that they were planning to introduce soon
Mobile research is still at an early stage of development
Support still rudimentary
Only two of the solutions met all of the key capabilities
Wide variation in what is offered by different vendors
platforms supported, key features etc
But developing fast
Most of the vendors recorded key capabilities that they were planning to introduce soon
Conclusions 1 of 2
Making it fit: web surveys on mobile devices. ASC 2011
Browser-based tools may offer better coverage but lack the more advanced mobile-specific capabilities
All browser-based solutions examined are competent at rending surveys on full-screen and mobile devices
Developers seem not to be aware of the importance of testing for survey designers
The more advanced and distinctive features of Mobile Research are not well supported in mobile apps yet
Conclusions 2 of 2
Making it fit: web surveys on mobile devices. ASC 2011
Further work
Future work needs to focus on the effect of mobile devices on conventional online research
Survey takers increasingly using mobile devices to take online Typically around 4% and growing
For them, the survey experience frequently degenerates into chaos
Web survey providers in general not addressing mobile surveys – desktop/laptop participation is assumed
35