gamifying. not all fun and games - breaking blue · 2020-01-08 · gamification is weaker,...

Are We There Yet? Where Technological Innovation is Leading Research Proceedings of the Association for Survey Computing, Volume 7. Edited by T. Macer et al. Compilation © 2016 Association for Survey Computing

Proceedings of the Association for Survey Computing, Volume 7 1

Gamifying. Not all fun and games

Phil Stubington, Charlotte Crichton

Abstract

There has been a lot of attention on engaging online participants to address declining response rates and

the risks of poor quality data. Participant availability via panels has plateaued, while completion rates

fall and many surveys appear poorly designed. This raises real concerns about the representativeness of

participants who are recruited from online panels or client email lists and the quality of their data. This

paper reviews evidence from experiments we have run using gamification, some examples of the latest

techniques for gamification and considers the practical challenges of implementing gamified surveys

for agencies, clients and panel providers.

Keywords

Gamification; engagement; online research; response rates

1. Introduction

The challenge facing market research

Maximising the participant experience and thus response rates and data quality are not new topics and

engaging participants has always been vital to market research. Furthermore, there is a strong body of

evidence that suggests this task is becoming harder not easier over time. The clearest indicator for this

is that response rates are falling in most markets. According to analysis published by Pew Research, the

response rate of a typical telephone survey in the United States was 36 percent in 1997 and was just 9

percent in 2015. In the United Kingdom, a paper by Ipsos MORI as long ago as 2008 reported that the

response rate for the National Readership Survey had fallen from 73.4 percent in 1974 to 51.6 percent

(although the situation does now appear to have stabilised with only a further two percent decline since

then). Similar analysis of the UK Labour Force Survey (Barnes W, Bright G & Hewat C (2008)) has

shown a 21 percent decline in response rates over a fifteen-year period.

The move to on-line research should have altered the paradigm for response rates since (at least for the

panel providers such as Lightspeed GMI, Toluna, Research Now and SSI) participants are effectively

pre-screened concerning their willingness to cooperate by the act of signing up to panel membership.

However, online research still has to compete with the numerous other interesting things that the internet

has to offer; on-line samples often vary hugely in the manner in which they are recruited and there is a

wide variety of different approaches to contacting participants of varying efficacy. Furthermore, on-line

survey invitations have to convince the participant to cooperate without the intervention of an

interviewer who can answer questions concerning privacy, data protection or the purpose of the survey.


2 Are We There Yet? Where Technological Innovation is Leading Research

Data concerning on-line survey response rates is inconclusive, for many reasons including (but not

limited to) commercial sensitivity amongst the major panel providers, issues of calculating a response

rate for some types of recruitment such as river sampling1 and the wide variety of sample sources. In a

literature review of on-line response rates, Schonlau M, Fricker R & Elliott M (2002) commented,

“studies on the use of the Web as a response mode vary widely in terms of the nature of their target

populations, how participants are recruited, and whether any attempts at statistical adjustment are made

in the studies’ analyses.” The paper pointed out that even amongst broadly similar surveys conducted

by the U.S. Census Bureau, response rates ranged from 27 percent to 75 percent. A 2008 paper by

Baruch and Holtom reported an equally wide range of response rates in the UK (the average response

rate for studies that utilised data collected from organisations was 35.7 percent with a standard deviation

of 18.8 percent).

An admittedly unscientific web search by the authors of this paper suggests that response rates between

15 percent and 25 percent are increasingly regarded as the norm for on-line surveys, which would

suggest that on-line surveys have joined their telephone and face-to-face counterparts as being

challenged by poor response rates.

The industry’s response

Because of this trend, and the need to ensure high quality data more generally, a number of the major

panel providers and various research agencies have invested significant time and effort in testing the

impact of improving questioning techniques and/or enhancing the visual appearance of surveys.

The work of Jon Puleston of Lightspeed GMI is probably best known in this regard, but many conference

papers have now been presented on this topic, which between them provide a considerable body of (not

wholly consistent) evidence.

In industry parlance, these approaches have tended to be grouped under the umbrella heading of

‘gamification’. Mavletova (2014) summarised the main elements of a gamified survey as: (1) stating

clear rules and goals for the participants; (2) involving participants with a relevant and entertaining

narrative; (3) maintaining motivation by providing interesting and achievable tasks or quests; and (4)

giving feedback on the progress and rewards for accomplishing tasks and answering questions.

In practice, gamification is often conflated with the use of a variety of techniques to improve the

aesthetic of a survey. However, this need not be the case, since many gamification techniques can be

used for text questions (Cape (2016)). Equally, it is possible to improve the visual appearance of a

survey, for example using images to replace answer lists, without drawing upon game techniques. For

the remainder of this paper, for convenience, we use the term gamification in the widest sense to cover

both question wordings and the visual aesthetic of the surveys.

1 Where potential participants are recruited through pop-ups and promotions on various web sites with the survey

normally undertaken immediately


Are We There Yet? Where Technological Innovation is Leading Research 3

2. Our evidence to support gamifying surveys

Why we decided to run our own experiment

Gamification as a means to improve online surveys has been talked about for almost a decade and

numerous experiments looking at the impact of gamification on the participant experience have been

conducted.

As early as 2008, Puleston & Sleep reported the following benefits:

1. Less straight-lining: up to 80 percent lower levels in some experiments,

2. Lower neutral scoring: average 25 percent lower.

3. Lower dropout (if questions are designed ergonomically): able to reduce from 5 percent to 1 percent

in test experiments.

Gamification has received considerable exposure at industry conferences and a significant number of

papers have been presented concerning its effectiveness. Puleston (2012) described it as “the most

powerful and effective means we have ever come across to encourage participants to put more thought

and effort into taking part.”

Nevertheless, as researchers, we should always be aware of publication bias and at least one paper by

Koenig-Lewis, Marquet & Palmer (2013) has cast doubt on a number of the claimed benefits of

gamification including improved response rates and elements of the participant experience. Even its

leading advocate has commented, “There is also the difficulty of squaring off the objectives of a piece

of research with the objectives of a game. Often, when we have thought about putting some of these

ideas into practice, we have found the two can lead you in different directions” Puleston (2012).

Additionally, survey context is important and examination of the published case studies and conference

papers led to us to conclude that there are certain types of survey where the evidence base for

gamification is weaker, specifically:

B2B surveys (or mixed B2B and consumer surveys)

Global surveys involving developing markets

Surveys involving the need to provide accurate behavioural data rather than brand perceptions

(which by their nature are less tangible).

Since these surveys constituted a significant proportion of our online research, we felt that the evidence

base was still insufficient to recommend this approach to many of our largest clients.

Our experiment

Therefore, working with Lightspeed GMI, we commissioned an experiment based on an existing project

which includes a mixture of consumer and B2B interviews, is global in nature, and whose main objective

is to provide detailed behavioural information. In addition, we believed this survey provided a particular

challenge as the topic (purchase and usage of printer ink or toner cartridges) is of little intrinsic interest

to consumers and businesses. Additionally, to collect data on the widest selection of question types, the

normal survey questionnaire was expanded to include a typical customer satisfaction battery and a suite

of agree/disagree statements of the type normally used to allocate participants to segments.



Proponents of gamification advocate that more engaging surveys not only generate better data, they also

provide a better experience for the participant (Puleston et al). However, other experiments (Pashupati

& Weber Riley) have shown no statistically significant improvements in the participant experience. This

is clearly a key topic of interest for the panel providers and one we felt needed to be explored in our

experiment, since as well as increasing response rates for individual surveys, they have a considerable

interest in reducing panel attrition. Therefore, a number of questions were included in our experiment

to gauge participants’ reactions to the survey experience.

For the experiment, we created three versions of the survey. These were a baseline survey, an optimised

(visualised) survey and a gamified survey. The baseline survey reflected the current tracking

questionnaire in terms of both wording and presentation. The visualised survey included visualisation

and simplified questionnaire wordings in addition to features such as timers on question answers. Here

is an example of a question presented in the baseline and formatted versions:

The gamified version of the survey was the same as the visualised survey, but also included some quiz-

type questions interspersed at suitable places throughout the questionnaire. It should be noted that this

was very much ‘light touch’ gamification since the quiz type questions were discrete and did not interact

with the main question flow in a manner necessary to achieve a fully gamified questionnaire with a

coherent narrative structure.



In addition, all three survey designs included one standard open-ended question, one open-ended

question with an alternative wording recommended by a number of papers on gamification, and one

open-ended question with a timer. Including all three open-ended question formats in all three versions

of the survey allowed us to look their impact independent of the other aspects of the survey design.

We ran the experiment in three markets – the US, the UK and India - and conducted 100 interviews

using each version of the survey in each market (for a total sample of 900 interviews). In each of the

cells, the sample was evenly divided between consumers and businesses.

The remainder of this section discusses the key findings from this exercise, along with more recent

evidence where we have incorporated elements from the experiment into the original client

commissioned project, which was the basis of the survey design.

The results

One of the key questions in the survey is about which model of printer people own, since the client

requires the results to be segmented by printer platform in order to be useful. Typically, there is quite a

high level of non-response at this question and, if participants don’t know their printer model then they

have to be screened out of the survey. With the aim of improving participant engagement and thus the

number of participants selecting a printer at this question, in both the visualised and gamified versions

of the test we modified the question and turned this into a timed task. We told participants they had two

minutes to check which printer they owned and showed a timer on the screen counting down.

Participants did not have to use all the allocated time checking their printer and could click next at any

time before the countdown had finished to select their printer on the next page.

The result was a significant increase in response rates shown in the chart below. Similar results were

achieved on a second question towards the end of the survey where we asked people to go and find their

ink or toner cartridge packaging to check the brand. Again, with a time challenge we got notably more

cooperation.



The revised question format has since been imported into the main survey and dramatic improvements

in the completion rate provide compelling testimony to the effectiveness of improved questionnaire

design:

Proportion of participants identifying their printer model

Main survey 2015 Improved survey 2016

Germany 83 percent 95 percent

UK 82 percent 94 percent

United States 82 percent 94 percent

Russia 82 percent 87 percent

In the core section of the main survey, we needed to know the different types of cartridges that have

been bought over the past six months and the quantity of each purchased. Again, we used a more visual

approach to ask this question, showing both the usual text about the type of cartridge, an image of each

type of cartridge and the use of colour to denote whether we were asking about black or colour cartridges.

In the gamified version, we enhanced this question further by using a counter instead of a text box to

capture how many cartridges had been bought.

84%

83%

79%

89%

89%

88%

88%

93%

87%

Could name inkjet printer model

Could name laser printer model

Checked brand of ink or toner oncarton packaging

Baseline Visualised Gamified



In a manner similar to the screener question about the printer owned, this produced a significant increase

in the response rate.

One type of research that is particularly resistant to being redesigned is the tracking or continuous

survey, because of concerns that the data collected won’t be comparable with data collected from

previous waves. These fears are not unjustified, as Puleston (2012) noted, “the answer to this question

[what impact does it have on the data] is not inconsiderable. Often the results can be measurably

different.” The survey that formed the basis of this experiment is a tracking survey.

Therefore, we also needed to be confident not merely that we were getting a better response rate, but

that the results had not changed due to any redesign. Within the experiment, we were able to compare

data on purchase levels of cartridges in the previous six months between the different versions, and as

the chart below shows, they were very consistent across all three versions. Since the experiment has

been completed, we have incorporated the same techniques in the main survey and year-on-year

comparisons confirm that the main impact is an increase in response rates, with the tracking data itself

remaining within the margins of error of the previous wave.



The survey also included some rating questions of the type normally used in customer satisfaction

surveys. To gamify these we changed the questions from grid-type questions to a sliding scale with a

face that changed expression as the slider was dragged along the scale. By changing from a clearly

anchored scale to a more reactive scale, we got a wider distribution of responses across the scale as

demonstrated by a higher standard deviation. We believe the higher standard deviation indicates greater

consideration given to the response and in our view data that are more accurate. Similar tests were run

on the ‘agree’ – ‘disagree’ scales typically used for allocating participants to segments and the same

results were achieved. This is especially useful, since large numbers of participants, offering similar

ratings on these statements makes it extremely difficult for researchers to produce clear segments,

especially using techniques such as k-means cluster analysis.

The participant experience

At the end of the survey, we asked some feedback questions to participants about their experience of

taking part. We then compared responses to these questions across the three different surveys and we

found that participants were more satisfied with the experience of completing the visualised and

gamified versions. In addition to being more satisfied, participants also found the visualised and

gamified surveys easier to understand and valued the variety of question types, especially with the

gamified version.



There were some differences in the impact of visualisation and gamification between the three markets

covered; in particular, there was less impact from visualisation and gamification in India than in the

UK and the US. We believe that this is due to the participants in India having done fewer surveys and

therefore having less fatigue from unengaging surveys. We also saw slightly more of an impact,

particularly of gamification, in the UK rather than the US. This is a bit harder to explain given that

both sets of participants were equally experienced. Perhaps here in the UK we have a more light

hearted culture, or perhaps there was something a bit more British in style about the survey wording

and quiz questions.

Finally, we asked participants to rate the length of the questionnaire and compared it with the actual

time they spent completing it. This is an interesting topic, since on one hand the industry is being

pushed to shorten questionnaires, especially for those completing on a mobile device; whilst on the

other we are trying to introduce techniques to make participants’ take a more considered approach to

their answers. In our case, participants spent longer completing the gamified version, yet their ratings

for how acceptable the questionnaire length was were actually slightly better.

Conclusions

We know that many research teams do not believe that their surveys are suitable for gamification and

that this problem is particularly endemic for B2B surveys and trackers. What this exercise showed was

that even simply presenting the questions better can produce significant increases in response rates and

enhanced participant satisfaction. It is fair to say that from this particular experiment the impact of

gamification was less uniform. For some metrics (e.g. participant satisfaction), the gamified version

was clearly better than the visualised only version, whereas for others there was no measurable

difference. However, gamification can be taken a lot further than we did in this case and we might

describe our approach as gamification ‘light’. A fully gamified survey can be made more integral to

the survey theme, with gaming rewards being linked to full and thoughtful answers and there is plenty

of evidence to show that this can be highly impactful.

When it comes to trackers, the continuity issue cannot be dismissed completely. There is always the

possibility of data discontinuity when any aspect of tracking survey methodology is changed. Outside

of this experiment, we have seen the impact of changes in question wording on responses particularly

to attitudinal questions and even significant impacts from changing sample source between panel

providers. However, that is not sufficient reason to continue with bad practice and the evidence from

our experiment showed that the changes to tracking data could be manageable (indeed they certainly

appear to be less than the changes seen when many studies migrated from either telephone or face-to-

face to on-line). However, as with those changes, we would always recommend a period whereby both

options run in parallel and, if needed, subsequent calibration of the results.



3. Latest techniques for gamifying surveys

Our understanding of what works and what doesn’t work is improving at both a theoretical level and the

evidence base from experimental surveys is increasing. Time does not allow for a comprehensive review

of all the literature in this section, even less the more recent unpublished work from the panel providers,

instead our intention here is to highlight some relatively simple approaches, which we or others have

found successful.

Survey introductions and response rates

Much of the literature surrounding gamification cites improved completion rates, yet the issue of

response rates is equally important. To an extent, these are less susceptible to some of the gamification

techniques, but attention still needs to be paid to both the design and wording of survey invites otherwise

all our efforts will be wasted!

Adams and Hunt’s (2013) write-up of large scale field tests by the Financial Conduct Authority (FCA)

on different forms of written communication with customers provides some very interesting evidence

that changes to the layout, content and in some cases even the signatory can have a dramatic effect on

response rates. Cape (2016), drawing upon self-determination theory, suggests that web survey

invitations should reflect the following guidelines:

Autonomy – you are free to do this or not as you choose

Competence – you are good at this

Relatedness – people like you do this

Value – what you are doing has meaning

We recently had the chance to explore this whilst running a study on behalf of the FCA and our results

showed that even limited changes to the introduction script could produce significant improvements in

cooperation rates.

Version 1 Version 2

We are conducting a random survey of

organisations that are registered with the FCA

and would like to ask you some questions

relevant to consumer credit. Depending on your

answers the survey is likely to take about 20

minutes.

We are speaking with organisations that are

registered with the FCA. The main discussion

will be about forthcoming changes that will affect

your business. It is designed to improve the

FCA’s communications with you and is likely to

take about 15-20 minutes.

Cooperation rate = approx. 50 percent Cooperation rate = approx. 65 percent

Guessing games and quizzes

In addition to the use of game type questions to collect data, stand-alone guessing games and quizzes

can also be used to break up surveys and re-engage the participant with the survey they are taking. They

can include rewards for right and wrong answers or time pressure or a finite number of guesses to

encourage increased attention. A common theme can then be carried on through the survey with short

guessing games or quizzes at intervals throughout the questionnaire to prevent participant fatigue during

longer surveys.



As described in the previous section, we used a number of such questions in our experiment and they

appear to have resulted in higher satisfaction ratings from the participants.

Subject matter quizzes are also starting to be used as a screener component and may prove effective for

ensuring that only people who truly qualify for that survey go on to complete it. This idea seems

especially promising for B2B surveys being drawn from consumer panels since quiz questions can be

used to check whether participants’ knowledge of the subject matter is consistent with their claimed job

function.

However, it is important to use guessing games and quizzes with caution. Puleston (2012) comments

“we found these more elaborate approaches had mixed appeal and could have a somewhat corrupting

impact on the data. Also, participants got confused between fulfilling the game mechanics and the tasks.”

To minimise the chances of this occurring we would recommend the following guidelines:

Guessing games and quizzes need to complement the survey and not distract from it to be an

effective tool in encouraging participation and engagement

They need to be in keeping with the survey and visually reflect the look and feel of the survey

The time needed to complete the game needs to be proportionate to the over survey length and

purpose

If participants are rushing the survey to get to the next guessing game or quiz then these are not

effective additions to the survey and the survey would be better without them.

Guessing games and quizzes can also be used as part of the incentive structure and/or to encourage

participants to engage over a longer period (for example in a diary exercise). Telling participants that if

they check back at a later time they can see how they did in the guessing game can be an effective way

of breaking up a long survey and minimising participant fatigue.

Survey narrative and quests

Normally we write questionnaires following a structure, starting with broad information about the topic

before narrowing to more specific aspects, for example, following the purchase or decision-making

journey. However, this structure is not always communicated to the participants nor integrated into the

design of individual questions.



The idea here is that through applying a more overt narrative to a survey, participants can follow this

through the survey completing tasks (questions) which reflect aspects of this. Taking this a stage further,

quests revolve around wording of multiple questions to make them more interesting for participants.

Anecdotally this is one of the harder aspects of gamification to implement, for example, we were unable

to integrate it into our survey experiment. Nevertheless, creating a scenario and using this to position

questions can make the survey process more interesting and engaging for participants.

Puleston and Sleep (2011) found that turning a question about how much music artists were liked into a

task where the participants were asked to build a playlist of how much each artist should be played

resulted in more artists being evaluated. The standard question resulted in 83 artists being evaluated

whilst the questified version resulted in 148 artists being evaluated.

Applying rules to what participants can do

Rules can be applied to what participants do to encourage more considered answers to questions. This

is a relatively simple way of capturing more considered answers to survey questions. Puleston (2012)

lists a number of ways rules can be applied to questions:

Adding a specific scenario to a question

Adding a timer to the question

Restrictive rules where what the participant can do is constrained either by restricting the

number of words or placing a time limit on how long they can take to answer the question

Whittling down rules where the participant can only select a limited number of things

In our experiment, we found that these techniques appear to be especially successful in increasing the

proportion of participants providing a response to questions which require them to provide factual data

such as the model of printer owned.

Several research teams have also undertaken tests showing that gamification can produce a significant

increase in the level of detail given in open-ended questions. For example, Cape (2016) shows a doubling

of the number of characters in an open-ended reply, tested in two different languages. One of the most

effective approaches we have found is placing an open-ended question at the start of a quantitative

survey to allow the participants to tell their story. The results from a study undertaken amongst those

living with cancer illustrate this dramatically:

Cancer and its treatment can affect all aspects of someone’s life. Before we begin the survey

questions, we would be really grateful if you could tell us in your own words how cancer may have

affected the day-to-day life of the person you care for in terms of their ability to carry out practical

tasks or activities, and how it may have affected their relationships and emotional well-being. Please

also tell us about the impact caring for them had on your life.

Participation rate = approx. 85 percent wrote at least one full sentence

Average word count = 63 (across the entire sample)



4. Gamification – the reality

The situation

When we commenced our experiment, the big panel companies we work with told us that nine out of

ten surveys they were sending out were not optimised for engagement, with the survey experience being

worlds apart from people’s more typical experience online. Discussions with the panel providers in

preparing this paper suggest that this position may not have improved materially since. Based on our

discussions with clients, interviews with researchers and feedback from the panel providers we believe

there are a number of practical considerations that are significantly impeding the adoption of gamified

surveys. These considerations span everyone involved in the process – from the client, to the research

agency, to the panel providers. Resolving some of them goes right to heart of how our businesses are

organised.

One of the biggest issues when designing and implementing gamified surveys is understanding the

theory behind gamification and what is possible in terms of graphic design and survey programming.

Researchers also need a grounding in the implications of introducing gamification on response bias and

how tracking against legacy data might be impacted. A major problem in this regard is many of the

features of gamification are inherently bespoke to an individual survey. Puleston (2012) comments,

“there is no escaping the fact that it is a creative solution. There are no out-of-the-box techniques other

than perhaps point-based scoring mechanisms.” This means that quick fixes such as templating and

questionnaire libraries are not good enough solutions.

How the process of survey design makes gamification difficult

This is particularly an issue within research agencies where the process for designing a survey can make

it difficult for gamification and new techniques to be implemented. Smaller agencies may lack the staff

resource to keep up with developments, whilst larger agencies may struggle to communicate them to

large and often geographically dispersed teams. We also believe the situation is worse if the agency is

relying on external programmers from panel companies or a web survey specialist to provide the

expertise. Unlike improving the visual appearance of a web survey, which can and often is incorporated

at the programming stage without amending the questionnaire flow or wording, at least some elements

of gamification (for example the introduction of a narrative structure into the questionnaire flow) need

to be decided at the commencement of the questionnaire design process. This is where the problem often

starts!

The process for designing a survey is conventionally that a relatively junior researcher (for example at

SRE or Project Manager level) will write the first draft of the questionnaire, which is then reviewed, by

a senior researcher or director before it is shared with the client. Hopefully the junior researcher will

have been thoroughly briefed on the objectives of the study and in most agencies will either formally or

informally have access to a library of previous questionnaires or individual model questions. However,

anecdotal evidence suggests that this briefing will often not include a decision about whether or not to

gamify the survey and the implications thereof. As a result, even if the senior researcher is more familiar

with gamification, reviews the questionnaire and therefore has the opportunity to implement a more

engaging questionnaire, this is often too late in the process, and in some cases would require starting

again almost from scratch.



The immediate issue here is ensuring the entire pool of researchers is up to date with the latest techniques

and therefore the first draft of the questionnaire is gamified. The problem of knowledge about latest

techniques and gamification being held only by some individuals within an agency can be partially

solved by holding sessions to ensure that the knowledge is disseminated and creating documents to

support this. However, the challenge is compounded by the fact that many of the skills needed to

implement a gamified survey are outside the traditional research skill set. Puleston (2012) comments,

“designing survey games is a creative art form and akin to the skill required to design good advertising

- not something that everyone is capable of doing.” This might mean that agencies need to change their

quantitative research staffing structures in a similar way to how many leading qualitative teams now

include video and design specialists.

Convincing the client

Assuming the agency has managed to successfully design a gamified questionnaire, the next challenge

that is frequently experienced is reticence and uncertainty from clients. From our discussions with panel

providers, they report that migrating tracking studies with a legacy data set often dating back many years

is especially problematic and this certainly matches our experience. Tracking studies often provide

valuable management metrics to clients and senior management teams are frequently incentivised on

this basis, which creates an inherent conservatism amongst client research teams. In common with the

process inside the research agency, we believe it is vital to get the clients on-board at the beginning.

Attempting to gamify a survey, which has already been signed off by the client, will make the task

significantly harder, if not all but impossible.

It is important to understand what is driving this reluctance so that this can be overcome and we can

move the industry forward in survey design. There are a couple of straightforward actions, which can

be done to lessen the uncertainty. Firstly, running experiments such as those discussed in this paper and

sharing the results with clients demonstrates the benefits of gamifying surveys. For new studies, it will

be sensible to conduct a pilot phase and at the end of the survey gather feedback from participants on

both the experience of completing the survey and whether there were any areas where it was unclear

what was being asked. For existing studies, probably the most realistic approach is a parallel run where

the original or ‘traditional’ survey is run at the same time (but with different participants) as the gamified

survey.



The industry needs to work together

Constantly present in relation to gamification are the perceived issues of timings and costs. Online

research has given us the expectation that surveys can be run both quickly and cheaply and whilst

seemingly almost every industry conference includes at least one paper on gamification, it’s usually

followed by one on agile research or advocating that we need to undertake research faster! The reality

is also that many of the experiments on gamification (including ours) have been undertaken as stand-

alone projects to a much more relaxed timescale than a normal project would allow. Once in the field,

gamification does not increase the timings and costs of a study in the way that telephone and face-to-

face research do, but it can increase the time taken to get into the field through a more protected survey

design process and this needs to be managed by the industry as a whole.

Ironically, for our industry, we have also been very bad at sharing data on the participant experience,

which might convince more agencies and clients to migrate to using gamified approaches or at least

improved visual survey design. To the best of our knowledge, the major panel providers all collect data

about the participant experience yet the process of sharing this with the clients is patchy at best. If the

panel providers (and specialist web survey companies) were to provide benchmarked data on the

participant experience this might accelerate the process. For best effect, the industry should work

together to agree a standard suite of participant feedback questions so that clients can pool data collected

from multiple panel providers and in house. This would also allow research agencies to review their

own surveys and understand what is working and what is not.

In conclusion, we would argue that the debate in the industry needs to move on from discussing whether

gamification is worthwhile, to managing the practical implications of it. This initiative will need to cover

all parts of the industry from clients, through to research agencies and the panel providers. Industry

bodies such as the MRS and academic institutions also have a vital role to play ensuring that their

training courses reflect current best practice. Ultimately, this has to be a combined effort, since in

deciding whether to participate in a survey the participant is influenced at least as heavily by their

experiences as they are by whatever the new survey promises.



References

Adams P & Hunt S (2013) Encouraging consumers to claim redress: evidence from a field trial.

Financial Conduct Authority.

Barnes W, Bright G & Hewat C (2008) Making sense of Labour Force Survey response rates. Office

for National Statistics.

Baruch Y & Holtom B (2008) Survey response rate levels and trends in organizational research.

Published in Human Relations.

Cape P (2016) Gamifying questions using text alone. Paper presented at IIeX Europe March 2016.

Landy L (2008) Response Rates Finding Direction Bite Sized Thought Piece. Ipsos MORI.

Koenig-Lewis N, Marquet M & Palmer A (2013) The effects of gamification on market research

engagement and response. Swansea University School of Business.

Mavletova A (2014) A gamification effect in longitudinal web surveys among children and adolescents.

Published in the International Journal of Market Research Vol 57.

National Readership Survey (2014) NRS Response Rates.

Pashupati K & Weber-Raley L (2015) Gamification in Survey Research: Do The Results Support The

Evangelists? Paper presented at CASRO Digital Research Conference.

Pew Research Centre (2012) Assessing the Representativeness of Public Opinion Surveys. Pew

Research Centre.

Puleston J (2012) Online research, game on! Paper presented at NCRM conference, Web Surveys for the

General Population: How, Why and When?

Puleston J & Sleep D (2011) The game experiments: researching how gaming techniques can be used

to improve the quality of feedback from online research. Paper presented at ESOMAR Congress, 18–

21 September, Amsterdam.

Schonlau M, Fricker R & Elliott M (2002) Conducting research surveys via e-mail and the web. Rand

Corporation.

Wells C (2015) To Game or not to Game? An investigation of the impact of survey visualisation and

gamification. Published in the International Journal of Market Research Vol 58.



About the authors

Phil Stubington, Director, Breaking Blue

Phil is an accomplished researcher, having spent 30 years working in the sector, and is a member of the

Board at Breaking Blue. Phil’s particular expertise is in developing solutions to markets clients

previously considered “un-researchable”, with a focus on tracking studies concerning usage, market

sizing and market share. He also sits on our Quantitative Development team, which has a remit to

improve our research offer through adoption of better research design and techniques, as well as more

effective ways of presenting results. He is a full member of the Market Research Society and a member

of the Royal Statistical Society. He can be contacted at Breaking Blue, Priory House, Battersea Park

Road, London SW8 4BG; tel. 020 7627 7720; e-mail [email protected]

Charlotte Crichton, Research Manager, Breaking Blue

Charlotte is a skilled research manager, taking projects from commencement and questionnaire design,

through fieldwork to final reporting and recommendations. She is skilled in the handling and analysis

of large datasets, frequently working across many different markets simultaneously to co-ordinate

fieldwork and data processing. Although she regularly manages complex multi-methodology projects,

Charlotte specialises in online research. Charlotte manages the project on which our gamification study

was based and was one of the team involved in the experiment itself. She is a member of the Market

Research Society and the Social Research Association. She can be contacted at Breaking Blue, Priory

House, Battersea Park Road, London SW8 4BG; tel. 020 7627 7750; e-mail

[email protected]

gamifying. not all fun and games - breaking blue · 2020-01-08 · gamification is weaker,...

Documents