MINING THE BRAIN FOR INFORMATION:

NEW RESEARCH TECHNIQUES

By JON PULESTON

VP Innovation

HOW TO BETTER MINE RESPONDENTS

FOR INFORMATION STARTS WITH

LEARNING HOW TO EFFECTIVELY

COMMUNICATE WITH THEM

THINK ABOUT SURVEYS LIKE

ADVERTISING AGENCIES THINK ABOUT

ADVERTISING

NEED TO THINK OF SURVEYS AS A

PIECES OF ENTERTAINMENT

WHICH SURVEY DO YOU WANT TO COMPLETE?

OR THIS ONE?

RESPONDENTS ARE CONSUMERS

MOTIVATING THEM TO THINKHOW WE RAISE PEOPLE’S INTEREST AND GET THEM THINKING

HERE IS HOW A TYPICAL SURVEY STARTS...

WE JUMP STRAIGHT INTO THE NITTY-GRITTY

WHAT’S YOUR SURVEY’S CHAT-UP LINE?

“Which of the following

confectionary products have you

purchased for yourself or a family

member in the last 2 weeks

darling?”

Which brands of toothpaste are

you aware of?

IF YOU HAD TO DO SOME RESEARCH ON

TOOTHPATE, HOW WOULD YOU START THE

SURVEY?

Which of these celebrities has

got the nicest teeth?

…OR LIKE THIS?

COPYWRITING QUESTIONS

RESPONDENT REACTION?

Please describe your favourite meal

Imagine if you were on death row and

had to plan your last meal. What would

you order?

HOW YOU ASK A QUESTION

CAN HAVE A BIG IMPACT

A question

Steak au pouivre

Pesto pasta

Fish and chips

Garlic chicken

A game

Scotch broth soup as a starter served with garlic bread.

Medium grilled gammon steak with a lightly fried egg on

top with chips and side salad. A glass of red wine. A sticky

toffee pudding, followed by cheese and biscuits.

Bacon chips and tinned tomatoes and an egg butternut

squash soup(homemade) fillet steak well done (not burned)

new potatoes fresh garden peas fresh fruit salad and cream

all served with a good white wine

Calamari for starters Curried goat with rice & jerk chicken

with plantain and fried dumplings for main meal and hot

sticky toffee treacle pudding with hot custard for dessert

with a triple amaretto and lemonade to drink

Classic roast dinner with beef cooked medium rare, pink in

the middle and a little bit of blood, the roast potatoes crispy

on the outside soft on the inside, Yorkshire puddings, peas

and gravy cooked in shallow oil and chips

THE RESULT: RICHER MORE THOUGHTFUL FEEDBACK

How much do you agree or disagree

with these statements?

How much like you are these people…

LIKERT QUESTION EXAMPLE…

THE APPLICATION OF RULES Abstract rules are the core constituent of most games

Golf A long walk with a heavy bag

Football Run around for hour and a half

STRIP AWAY RULES FROM GAMES

AND WHAT DO YOU HAVE?

Example...

Describe yourself?

Describe yourself in EXACTLY 7 words

2.4Descriptors

85%Answered

4.5 Descriptors

98%Answered

Pick EXACTLY 3….

In LESS THAN 7 words

Name ONE THING you would change about this product

If you could only buy one of these brands for the REST OF YOUR LIFE

UNDERSTANDING THE VALUE OF RULES IN

QUESTION

If you were buying xxxx which of these things would you DEFINITELY

do prior to purchase

In your PERFECT world what would you do…

ART DIRECTING

GETTING TO GRIPS WITH THE EFFECTIVE USE

OF IMAGERY IN SURVEYS

THIS SORT OF APPROACH IS NOT GOOD ENOUGH

IMAGES CAN ADD MEASURABLE VALUE

WHEN USED PROPERLY...

WHAT THEY CAN DO IS ENCOURAGE

RESPONDENTS TO STOP AND THINK!

Resulting in richer feedback

35% more

drinking incidents

recorded across the

day

Pain sliders

Pain sliders

Source: ESOMAR Asia April 2012

Can gamification techniques cross cultures

Pain sliders

EXAMPLE OF THE ERGONOMIC DESIGN OF

QUESTIONS

UNDERSTANDING THE ROLE OF ICONOGRAPHY

LESS LOAD ON THE WORKING MEMORY

Issues with this question:

1. Boring and repetitive so people’s brains switch off

MAKING TASKS MORE INTELLECTUALLY SATISFYING

12% of people in the UK claim be unaware of the Olympics?

India UK

Commonwealth Games 3.3% 16.2%

Cricket World Cup 3.3% 26.7%

Football World Cup 5.4% 13.3%

Indian Premier League (ILP) 5.4% 62.9%

Wimbledon Tennis Championships 5.4% 9.5%

Formula One (F1) Motor Racing 6.5% 11.4%

Oscars Ceremony/Academy Awards 6.5% 18.1%

Olympic Games (Summer) 12.0% 12.4%

English Premier League (Football) 13.0% 13.3%

Olympic Winter Games 15.2% 13.3%

Rugby World Cup 17.4% 21.0%

World Athletics Championships 21.7% 21.9%

MTV Video Music Awards 23.9% 33.3%

UEFA Champions League 29.4% 15.2%

Youth Olympic Games 30.4% 52.4%

Barclays Premier League 32.6% 20.0%

NBA Championships 33.7% 30.5%

The Masters (Golf) 35.9% 19.1%

Paralympic Games 38.0% 11.4%

NFL Superbowl 52.2% 33.3%

Extreme Sports X Games 55.4% 56.2%

NHL Stanley Cup 66.3% 70.5%

A MORE PURPOSEFUL & REWARDING QUESTION

TO ANSWER

INTELLECTUAL SATISFACTION

UK India

Football World Cup 0.0% 0.0%

Olympic Games (Summer) 0.0% 3.3%

Formula One (F1) Motor Racing 0.0% 3.3%

MTV Video Music Awards 0.0% 5.4%

Oscars Ceremony/Academy Awards 0.0% 1.1%

UEFA Champions League 0.0% 8.7%

Barclays Premier League 0.9% 9.8%

Commonwealth Games 0.9% 1.1%

Olympic Winter Games 0.9% 5.4%

Paralympic Games 0.9% 6.5%

Rugby World Cup 0.9% 3.3%

The Masters (Golf) 0.9% 4.4%

Wimbledon Tennis Championships 0.9% 3.3%

World Athletics Championships 0.9% 3.3%

Cricket World Cup 1.7% 0.0%

English Premier League (Football) 1.7% 5.4%

NBA Championships 2.6% 10.9%

Extreme Sports X Games 3.5% 9.8%

NFL Superbowl 3.5% 14.1%

Indian Premier League (ILP) 7.0% 1.1%

Youth Olympic Games 7.8% 2.2%

NHL Stanley Cup 12.2% 16.3%

MAKING QUESTIONS MORE

REWARDING

RESPONDENT HAD TO PREDICT THE BRAND AGAINST

THE CLOCK & WIN POINTS FOR GETTING IT RIGHT

AWARENESS TASK: ANSWER TIME PER QUESTION

Standard Survey Guessing Game with time pressure

MORE ACCURATE BRAND RECOGNITION

HOW TO GET RESPONDENTS TO THINK

IN DIFFERENT WAYS

fasterHarder intuitively creativelyObjectively

MOST QUESTIONS IN SURVEYS MINE

ANALYTICAL THINKING WHICH ONLY A

FRACTION OF OUR DECISION ARE BASED

UPON

CONCIOUS DECISIONS

UNCONCIOUS DECISIONS

SURVEYS TEND TO BE DOMINATED BY

SYSTEM 2/ANALYTICAL JUDGEMENTS

Which of these pet food products

would you buy?

The one with the

best ingredients

The one with the nice

picture of a dog

Research says this Sales say this

CAN BE AS SIMPLE AS IMPOSING TIME LIMITS TO CHANGE THINKING PROCESS

BRAND CHOICE

TECHNOLOGY IS DRIVING THE RAPID

GROWTH OF IMPLICIT MEASUREMENT

TECHNQUES

Brand logo

1/28TH second

Impressions of person subliminally influenced by brand logo.

Test against a blank image or different brand or pre and post advertising

2 second 4 second

IMPLICIT ASSOCIATION TESTS

-30% -20% -10% 0% 10% 20% 30%

Exciting

Likable

Only interested in making money

Someone I could relate to

Someone I could count on

Care about me

Annoying

Company I would enjoy

I don’t need in my life

Con artist

Very ordinary

Impact of brand on perceptions of person

* Based on comparison of net scores from top 2 & bottom 2 choices on 7 point scale v control

Disagree Agree

SOURCE: COG RESEARCH

MEASURING IMPLICIT EMOTIONAL

RELATIONSHIPS WITH BRANDS

FunWarmReliableOld FashionedCheerful

FastSlow

Agree

Disagree

Cool

Old fashioned

Reliable

CheerfulFun

Warm

FastSlow

Agree

Disagree

Cool

Old fashioned

Reliable

Cheerful

Fun

Warm

USING FACIAL RECOGNITION TECHNOLOGY WE CAN NOW MEASURE HOW PEOPLE FEEL WITHOUT ASKING QUESTIONS

WWW.GMI-MR.COM | CONFIDENTIAL | 60

OPEN YOUR MINDTO THE POWER OF

THE

OPEN

ENDED

QUESTION

A BI-PRODUCT OF BIG DATA ARE RAPID

ADVANCEMENTS IN NATURAL LANGUAGE

PROCESSING TOOLS & TECHNIQUE

THIS TECHNOLOGY IS OPENING UP NEW OPPORTUNITIES

FOR MARKET RESEARCHERS

DISTRIBUTION OF STARS

50% OF FILMS SCORE 4 STARS

rushskyfall

olympus has fallen

harry potter

now you see meelysium

despicable me 2kick ass 2

alpha papairon man 3

the dark knightrunner runnerthe way way

backfreshman father

brother bearshe was to good

for himlawless

the muppetsles miserablesjack reacherabout time

great gatsby

looperst trinians

8 degrees belowthe butler

mamathe heatmeet the robinsonsthe rush

this is the endthe world's end

transformersred obsession

incrediblesmirror boy

red2mortal

instrumentsboratmud

scooby doo 2american piei'm so excited

narniabattleship

next three daysthe boat that

rockedno pain no gain meet the milners

the croodshansel and gretel

the flying scotsman

crossing over

the last mileoblivion

the notebooki give it a yearthe sweeney

avatarthe woods

only the rainthink like a man act like a lady

visitor qtower heistweekend

twilightepic

behind the candelabra

papillionend of watch

plan 9 from outer

spacepuss in boots

red statethe conjuring

we're the millers

planesone direction this

is usthe awakening

grown ups 2man of steel

smurfsmonsters universitytaken 2

anuvahoodhang em high

after earthpitch perfect

the lone ranger

WORD SCORING SYSTEMSa system based upon respondents ratings of words

super

great

gem

thrilling

fascinating

inspiring

exciting

very good

beautiful

clever

entertaining

engrossing

enjoyable

enchanting

innovative

classic

amusing

hilarious

moving

interesting

adventurous

touching

good

fun

cool

incredible

pretty good

heart-warming

very nice

emotional

liked

provoking

quite good

decent

quirky

different

fine

nice

bizarre

average

typical

ordinary

confusing

light weight

childish

juvenile

dated

overrated

dull

not great

nothing

poor

disliked

boring

mindless

disappointing

stupid

exceptional

masterpiece

spectacular

fantastic

outstanding

superb

excellent

amazing

greatest

brilliant

awesome

wonderful

fabulous

epic

powerful

favourite

loved

marvellous

best

absurd

bad

very poor

nonsense

pointless

useless

lousy

sucked

drivel

pathetic

dreadful

garbage

junk

awful

rubbish

terrible

crap

the worst

disgrace

4 points

3 points

3 points

2 points

2 points

1 points

-2.5 points

-2.5 points

-0.5 points

-3 points

“extremely ****”

“really ****”

“quite ****”

“incredibly ****”

multiply by 1.75

multiply by 1.5

multiply by 0.5

multiply by 2

“not ****”

multiply by -0.5

APPLIED TO FILMS: GREATER DIFFERENCIATION OF SENTIMENT

REFLECTING EXACTLY WHAT THE RESPONDENTS THINK

“good sci-fi story with excellent special effects”

WHAT STOOD

OUT ABOUT IT

WHAT TYPE OF

FILM IT WAS

“it was hilarious. i loved the minions and couldn't

stop smiling”

“great film, most enjoyable whilst playing

my emotions”

“some excellent action, a reasonable plot

although a little transparent”

A SINGLE OPEN-ENDED REVEALS MORE

THAN JUST SENTIMENT & RATINGS

FILMS:

CODING & CLUSTERING COMMENTS:

HUMOUR

“funny”

“hilarious”

“humorous”

“made me

laugh”

“comical”

“light-

hearted”

ACTION

“action”

“action-

packed”

“lively”

“explosive”

“exciting”

“high-

octane”

DRAMA

“emotional”

“romantic”

“heart-

warming”

“touching”

“inspiring”

“love”

“dramatic”

TENSION

“tense”

“thrilling”

“jump”

“suspenseful”

“scary”

“dark”

“intense”

“anxious”

THEMATIC DIFFERENCES

EMOTIVE COGNATIVEVS

“emotional”

“heart-

warming”

“touching”

“happy”

“love”

“joy”

“interesting”

“educational”

“thought-

provoking”

“engaging”

“moralising”

“values”

ENGAGEMENT OF HEART OR MIND

Twilight Breaking Dawn – Part 2 The Hunger Games

OUR LANGAUGE REVEALS OUR

LESS CONCIOUS OPINIONS

It’s the cheapest

Its cheap

Its cheaper than…

Its affordable

Economical

Good value

Better value

Value for money

The best value

Worth paying more for

It’s the best quality

10 DIFFERENT WAYS TO DESCRIBE THE PRICE

OF A PRODUCT…

Buying purely on price

Quality trade off

Buying on quality

PREDICTIVE

THINKING

TECHNIQUE

WILL YOU BUY THIS?

YES = <0.3 correlation with sales*

* Please note this is an anecdotal figure

WILL YOU CLEAN UP?YES = 50%

PREDICT HOW MANY WILL CLEAN UP?YES = 15%

HOW MANY CLEANED UP?

WE ARE NOT GOOD AT PREDICTING OUR OWN BEHAVIOUR

13%

How much would you pay?

£10, £11, £12, £13, £14, £15, £16, £17, £18, £19, £20+

THE VALUE OF THE PROJECTIVE VIEWPOINTS

Predict what price you think this will be sold at?

£10, £11, £12, £13, £14, £15, £16, £17, £18, £19, £20+

<50% less than actual price

Within 1% of actual price

WHO WILL YOU VOTE FOR?

1,000 respondents

PREDICT WHO WILL WIN?

20 people gambling

IOWA ELECTION MARKET

IN 451 out of 596 OCCASIONS

ELECTION FORECASTING

PREDICTION ACCURACY* = Quality of

information x Motivation of predictors to

make a correct prediction x Ability of the

people to objectively process that

information.

Note it’s not dependent directly on

sample size. One person with

access to all the right information

can make a correct call e.g. Nate

Silver!

THE MONTECARLO METHOD

The “mote carlo” method

instead of sampling lots of

people and averaging

out their answers (all of

who may have the same

network error) it samples

lots of sources of

information and averages

out their answers.

-100

0

100

200

300

400

500

600

Thumbs up

Eye catching

Curiosity

Interest

Clarity

Positive perception

2 sec thumbs up

"This is exactly the type of advertisingwe should be doing!"

Design

Winning ad

Considered verdict

2 sec verdict

A MONTECARLO APPROACH TO AD TESTING:

Every measure tells a slightly different story

PREDITVE MARKET TRADING

-200%

-100%

0%

100%

200%

300%

400%

500%

600%

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

Sum Net Investment Traditional rating

GREATER DIFFERENTIATION OF OPINION

CREATIVE THINKING

USING GAMES IN SURVEYS TO GET PEOPLE

THINKING MORE CREATIVELY

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

With Stimulus Without

Very imaginative

Quite

Not very

QUALITY OF IDEAS

RANDOM STIMULATION

1. Idea for an idea...

6 to 18 ideas

2. Thinking rooms

0.7 to 6.9 ice cream names

3. Thinking through the minds of...

1 or 2 thoughts increased to 10 to 20

SUMMARY

• Learning how to mine our brains for

information is all about learning how to get

people think in different ways

• I would encourage you to treat respondents

in a survey like consumers

• Break away from some of the more

traditional question approaches

• Open you mind to the power of open ends

• Embrace the Monte Carlo approach

• Make things FUN!

Q & A

TWITTER @JONPULESTONwww.question-science.blogspot.co.uk