game-changing audience research techniques

Game Changing User Research Research as cheap & easy as a cup of coffee

PRESENTED BY ALISSA RUEHL & NATHAN RENIHAN

TWEET!

#BBCON

#UserResearch

@ALISSARU

WHY?

People in the world

Visitors to your site

Visitors to your donation

form

Donors

People in the world

Visitors to your site

Visitors to your donation

form

Donors

People in the world

Visitors to your site

Visitors to your donation form

Donors

People in the world

10% more

Visitors to your site

10% more

Visitors to your donation form

10% more

Donors

People in the world

Visitors to your site

10% more

Visitors to your donation form

10% more

Donors

People in the world

Visitors to your site

Visitors to your donation

form

10% more

Donors

As an insider

your perspective is very different.

Bite sized

research

PROOF

• We’ve taken our own medicine

• Onsite research:

- Total time: 20 minutes

- Total cost: $0

vs

OUR USER RESEARCH

4 11

WHAT?

What’s the mystery you’re trying to solve?

• Why are people abandoning the donation form?

• Why don’t more people give recurring gifts?

• What are the main things people are looking for on

my site?

• Do people notice the promotion we’re doing?

• How should we organize our new website (or

newsletter, etc)

POTENTIAL CASES

WHO?

People involved with your organization

The general public

PARTICIPANTS

• Good targets

- People somewhat involved with your organization

• Event participants

• Donors

- The general public

• Avoid

- Employees

- Board

• Generally avoid*

- Major volunteers

- Friends & family

HOW?

•Starbucks testing

•Card sorts

•Surveys

3 QUICK & EASY METHODS

STARBUCKS TESTING

When you want to know if average people run into

obstacles when trying to complete a task

CARD SORTS

When you want to know how people think information

should be organized

SURVEYS

When you want to answer specific questions about

people currently involved with your organization

STARBUCKS TESTING

When you want to know if average people run into

obstacles when trying to complete a task

RULES FOR TESTING

TASK BASED

VERBAL FOLLOW UP

IT WORKS ABOUT LIKE THIS

ALWAYS BE PREPARED

1. Make sure you pick something that regular people

use, you will get regular people at Starbucks.

2. Test internally for practice

3. Test internally to make sure it is truly a 5 minute

test

4. Take 1 person with you. 1 person stays with laptop,

1 person solicits.

BEFORE YOU GO

1. Make sure Starbucks is ok with it. Talk to the

manager on duty first, have a pitch ready for that

conversation so you can easily explain what you

want to do.

2. Bring your charger, mouse, test internet connection

(if needed), etc.

3. Grab a table - go early or avoid rush times.

BEFORE YOU START

1. Talk to the people that are sitting around, you are buying

their NEXT cup of coffee, avoid the people in line.

2. Put your best foot forward, pick someone that will get a

good response.

3. Work on your pitch, it has to be faster than an elevator

pitch. Think of your pitch like a tweet: short, sweet and

to the point.

4. Make it obvious that you are a non-profit, lead with that,

wear the gear, beat people over the head with it; people

are more likely to help.

5. It helps to stress that this will take 5 minutes or less.

WHEN YOU ARE READY

1. Don’t record names

2. Do get a wide cross section of users (age,

gender, race, etc)

3. Test the savvy level of the user, make sure they

aren’t extremely savvy or extremely novice.

4. Give them their task

5. WATCH

6. Try not to help

7. Ask follow up

WHEN YOU HAVE A USER

• If you find a massive gotcha, STOP. No need to keep

testing. Go back and fix it and try again another day.

• Once you have 3-5 users you probably have some really

good feedback, no reason to keep going. Maybe switch

to a new thing to test.

• This whole process might take 1.5-4 hrs

• You will most likely learn things that will blow your mind.

Things you never would have thought of.

WHEN THE TEST IS OVER

QUESTIONS ON STARBUCKS TESTING

CARD SORTS

When you want to know how people think information

should be organized

People in the world

Visitors to your site

Visitors to your donation

form

Donors

07/10/2013 44

07/10/2013 45

07/10/2013 46

07/10/2013 47

vs

OUR USER RESEARCH

12 3

• Write down each 2nd level menu item on

its own note card

- In this example

• Write down

– News Archive

– Wayne’s blog

– Press releases

– Media contacts

• Don’t write

– NEWS

CARD SORTS

1. Get a participant and some table space

2. Give them

- your note cards

- some blank cards or post-it notes

• (No more than you have space in your top level

navigation)

3. Let them sort the pre-written note cards into

groups

4. Ask them to label each group (using the

blank note cards)

5. Record the results

6. Repeat at least 10 times

CARD SORTS

07/10/2013 51

• Tips

- Be sure to stay quiet while the participant is working

- Ask them to think aloud and listen to their commentary

- Take a photo when each person is done to capture their

work

• Variations

- Pre-write the headers (closed card sort)

- Conduct the card sort online

• http://www.optimalworkshop.com/optimalsort.htm

CARD SORTS

QUESTIONS ON CARD SORTS

SURVEYS

When you want to answer specific questions about

people currently involved with your organization

POLL: WHO HAS CREATED OR SENT A SURVEY?

POLL: WHO HAS ABANDONED A SURVEY THEY’VE

RECEIVED?

• 11 out of 15 people surveyed had never done any

user testing

- Surveys were the way most people do user

research

• 15 of 15 people surveyed had abandoned a survey

half way through because it was too long or too

confusing

OUR USER RESEARCH

• Keep it short

- Usually have 5 – 15 questions

- Should take less than 5 minutes to complete

• Always have a general comment field

- Let them speak to you, not just answer questions

• Have someone proofread your survey

- Make sure none of the questions are confusing

• Be neutral

- You want information, not vanity metrics

• You don’t need to survey everyone

- How many responses do you need to answer your questions?

• You can use statistics, or

• General questions, homogeneous audience ~100 responses

- Assume a response rate of 10 – 20% for people somewhat involved with

your organization

SURVEY ETIQUETTE

• Think of each question as testing a hypothesis. It

can help to word them that way

- Less useful: We should ask how many times per year they visit our

museum

- Better: I suspect that most people visit either only once per year or

more than 4 times. This will affect how we price and market our

membership programs. If we find a lot of people who visit 3-4 times

a year, we might decide to adjust our programs

SURVEY ANALYSIS TIPS

• Avoid “neutral” options

- Very likely

- Somewhat likely

- Somewhat unlikely

- Very unlikely

• (But you can use “not applicable”, if applicable)

• Experiment with forced rankings

• If you’re doing a large survey, consider how many open-ended

questions you want to analyze

SURVEY ANALYSIS TIPS

QUESTIONS ON SURVEYS

OTHER THINGS

WEB ANALYTICS

When you want to know how people are getting to your

site & what they are doing on your site

07/10/2013 71

07/10/2013 72

Focus Groups

07/10/2013 73

Formal Usability Studies

07/10/2013 74

Blackbaud Discovery

THIS PRESENTATION BROUGHT TO YOU BY

BLACKBAUD USER EXPERIENCE

• We want to learn from you so we can design

experiences that delight

• Participate in Blackbaud Discovery!

- Help us with our research

- Have your voice heard for the products you use

• Sign up to participate at

www.blackbaud.com/discovery

• Stop by the Discovery booth in the expo hall

FOR MORE INFORMATION

LINKS

• Alissa’s blog posts on npengage

- http://www.npengage.com/author/alissa/

- http://www.npengage.com/online-fundraising/testing-your-donation-

form-part-1-why/

- http://www.npengage.com/online-fundraising/testing-your-donation-

form-part-2-designing/

• Other posts

- http://www.npengage.com/web-design/nonprofit-web-design-process-part-

surveys-and-interviews/

LINKS

• Analytics

- http://www.google.com/analytics/

- http://www.npengage.com/content-marketing/google-analytics-1/

• Starbucks testing

- http://www.mattbaline.net/3-lessons-from-guerrilla-mobile-usability-at-

starbucks/

- http://joshledgard.com/i-wish-id-thought-of-that-operation-starbucks/

- http://blog.kooaba.ch/2012/02/developing-shortcut-starbucks-usability-

testing/

• Card sorting

- http://rosenfeldmedia.com/books/card-sorting/

– http://www.optimalworkshop.com/optimalsort.htm

• General

- http://uxdesign.smashingmagazine.com/2009/05/15/optimizing-conversion-

rates-its-all-about-usability/

- http://www.nngroup.com/articles/non-profit-websites-donations/

- http://www.sensible.com/rsme.html