effective digital diary studies / uxpa webinar may 2016
TRANSCRIPT
Effective Digital Diary StudiesSara Cambridge [email protected] Webinar: May 18, 2016
What do diary studies offer that interviews don’t?
3
Interviews don’t capture the range of ways people do things or the rich, in-the-moment details that people quickly forget.
Fitness Tracker Case Study40 people who use fitness trackers were recruited to participate in several digital diary studies over 8 months.
The first diary study had them report on 5 things they do with their tracker. With every entry, they submitted a short video and answered a few questions.
44
5
Erica, 33
“It’s 10:53pm, and I’m only at 7388 steps….we’re about to go take the dog for a walk. Got to make sure we get to our 10,000 steps before the end of the night.”
Show us how you use your fitness tracker
Digital diary studies excel at: • Gathering contextual information cost-effectively • Capture repeated behaviors in context • Providing instant feedback on products • Uncovering latent needs in general problem space
What This Presentation Will CoverWhat a Diary Study is
Advantages of Diary Studies
Platforms for Conducting Digital Diary Studies
How to Conduct a Digital Diary Study
Highlights for Ensuring an Effective Digital Diary Study
7
What is a Diary Study?
A Diary Study is a Cultural ProbeCultural probes use a collection of tools, artifacts and tasks intended to provoke the user to think about their environment in new ways….With minimal intrusion, researchers can glean insights into participants’ environments that can help to identify problems and uncover new opportunities.
9
Adam Starkman & Jennifer Chow http://designresearchtechniques.com/casestudies/cultural-probes/
Key Elements of Diary StudiesTheme-based: the subject and questions are suggested by the researcher.
Self-reported: the actual reporting is done by the participant, using whatever platform/prompts were provided.
In-the-moment: the participant is asked to do the reporting immediately after engaging in the activity, in order to capture as much detail as possible.
Longitudinal: the goal is to capture a range of uses, rather than a singular experience. This can include how the user’s experience evolves over time.
10
Traditional Diary StudyMozilla used paper diary studies in 2012 to learn how participants were using bookmarks in Firefox.
They asked 10 people to record their experiences with bookmarks over 3 days.
11
Brian Groudan, Mozilla research, 2012 / https://blog.mozilla.org/ux/page/24/
Traditional Diary StudyPeople came in for an in-person interview afterwards, where they discussed their entries and sorted them by theme.
Guiding design principles were culled from the analysis that were the basis for design better bookmarking tools.
12
Why Use Diary Studies?
What’s so great about ‘in-the-moment’ feedback?
Can’t you get the same information in an interview a few days later?
14
15
Most see it like a video camera….
How Does Memory Work?
but really it’s like a slow leak
The Psychology of Habits
16The Power of Habit, Duhigg
Behavior triggerSequence of actions
occurs repeatedly
Reward / Finish
Our brains turn the sequence into one automatic routine;
“chunked”
The Psychology of Habits
17The Power of Habit, Duhigg
We can’t access the details of habitual actions until we start them
Diary studies ask people to report when details are still accessible
This also allows for collecting workarounds people are not aware of
Behavior trigger
Details of actions become
accessible
Reward / Finish
Habit sequence executed
What is a Digital Diary Study?
19
Getting people to actually do the reporting is a major obstacle for traditional diary studies.
http://kotaku.com/hong-kong-residents-supposedly-cant-live-without-their-476758731
20
Smartphones are ideal for digital diary studies since people use them in short spurts throughout the day.
http://kotaku.com/hong-kong-residents-supposedly-cant-live-without-their-476758731
Average use: 85 times a day*
*Study of 23 people aged 18-33: http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0139004
Tools for Executing Digital Diary Studies
Platform (web/app)
SMS/Text
Blog
Video
21
Digital Diary Study Platforms
22
Name Platform Cost
20/20 Research Web/app diary study high
24Tru Web/app diary study high
CrowdTap Web/app diary study high
UX 360 Web diary study high
Dscout App diary study medium
Dedoose Web diary study low
Typeform online survey low
23
Diary Study Platform Advantages: • Offers analysis; faster to insights • Lots of data collection options • Many have panels for recruiting • Guidance, support, education • Team members can participate • Many participants = quant data
Disadvantages: • Cost • New tool • For more tech-savvy participants • Can’t adjust once launched
$595 for recruiting
$295 for each study
24
Most diary studies are a mix of multiple choice and open ended questions.
Example Dscout Diary Study:New Product LaunchGoal: Get feedback on a new digital entertainment product
Details: 5 families, 4 months, in-person interviews throughout
Platform: Dscout app, asking them to report every time they used the service, answering a series of questions.
Key Finding: The service was most robust for mobile viewing, but the families did the bulk of their viewing on TVs.
Surprising Finding: The study uncovered significant technical errors that were unknown to our client.
25
26
SMS/Text
Advantages:
• Free
• Good for less tech savvy participants
• Can adjust on the fly
• More likely to get immediate response
Disadvantages:
• All analysis is manual
• Limited data input
Example SMS Diary Study: Tracking Diabetes PatientsGoal: Learn how often Type 2 Diabetes patients actually check their glucose level, as well as how often they have their glucose monitor with them
Details: 6 participants, 4 weeks, in-person interviews at the beginning and the end
Daily Text: How are you feeling today? Have you checked your blood glucose today? Where is your meter now?
Key Finding: People check their glucose & have meter less often than was assumed. Several aspects were identified that dictate how often people check their glucose
Study Insight: Asking how they were feeling made them feel cared for, rather than reminded/judged
27
28
Advantages:
• Free
• Good for less tech savvy participants
• Can adjust on the fly
• Likely to get more details than if using SMS
Disadvantages:
• All analysis is manual
• Emails are easy to overlook
Example Email Diary Study:New Product LaunchGoal: Understand how and why people use different channels when managing their investment account
Details: 18 participants, 2 weeks, in-person interviews at the beginning and the end
Daily Emails: Questions to understand any activity taken regarding investment accounts (activity taken, channel used, how felt about it, etc), plus occasional tasks for probing deeper on specific aspects
Surprising insight: People sometimes conflate how often they think about their money with how often they check their accounts
Biggest Challenge: Not having enough time to fully analyze data
29
30
http://www.uxmatters.com/mt/archives/2012/01/video-diaries-a-method-for-understanding-new-usage-
patterns.php
Video
Advantages:
• Can collect rich content
• Edited video = rich storytelling
• Most people send cameras back
Disadvantages:
• All analysis is manual
• Transcribing video
• Cost of multiple cameras
• Ease of use is critical
31
http://www.effectiveui.com/blog/2012/03/29/using-digital-diary-studies-to-understand-the-customer-experience/
Blog
Advantages:
• Free
• Can adjust on the fly
• Likely to get more input than SMS
Disadvantages:
• All analysis is manual
• Researcher must set up accounts ahead of time
How to Conduct a Digital Diary Study
The 3 Phases of a Diary Studies
33
The goal: a concise, focused study with the right participants.
The goal: high participation and high quality data.
The goal: to turn data into relevant insights.
• Clarify your research question
• Refine and pretest your study
• Minimize the work for your participants
• Recruit the right participants
Clarify Your Goals & Research Question Before Building the StudyDefine the goals for the study. What do you want to learn?
Turn that into a research question that will guide your efforts as you create your diary study.
36
Fitness Tracker Study Research Question:
How are activity trackers used by people attempting to make concrete
improvements to their fitness habits?
37
What Type of Diary Study Will You Build?Ideation: Collect ideas/suggestions. “Show me your ideal…”
Inventory: Understand key things participants use, own or do. “Show me all of your...”
Moments: Capture experiences as they happen in real time. “Show me every time you…”
Process: Understand key steps of a process in real time. “Show me your process for…”
Reflection: Explore likes, dislikes, memories and motivators. “Tell me how you feel about….”
38
http://dscout.helpscoutdocs.com/article/121-tutorial-2-building-your-mission
39
Consider how often you ask them to report. It should be enough to collect useful data, but not enough to cause fatigue.
Help Your Participants Succeed By Reducing the Reporting Burden
40
Pairing themed multiple-choice questions with open-ended ones creates a logical flow for people and answers both “what” and “why”.
41
Another benefit of multiple-choice is you can quickly see trends as the data comes in.
42
Open-ended questions provide the richest data but are more work for both for you and the participant. Use them sparingly.
I think it should send reminders on when to be more active when I'm lagging in my goal such as vibrating.
I do wish that other activities could be tracked with the Fitbit without having to input it myself.
Maybe if I could get some sort of notification on my phone if a specific friend or group of friends had passed me.
43
Pretest by sitting down with someone as they go through every step. Look for awkward or unclear phrasing.
45
Diary studies generate a lot of data. It is good to start small.
10 x 10 x 10 =questions participants entries
data points
1000
46
If recruiting through Dscout, you’ll only pay incentives for participants that finish. Expect to pay $5-$20 per entry.
We paid $25 for a total of 6 entries.
If you’re working with a specialized or busy population, you may need to pay as much as $20/entry.
47
We looked for participants who have a fitness tracker and were regularly using it as part of a fitness or weight loss goal.
Find the Right Participants
• Review & make notes from your dataas it’s coming in
• Be persistent with participants
• Ask them questions for clarity
• Do post-study interviews
Watch the data as it comes in. This is the fun part.
Are your questions holding up? If not, can you fix it?
Either way, learn for next time.
50
51
Why did you get your first fitness tracker? Larry, 31
“I got a fitness tracker because was about 300 lbs, on a diet, exercising, wasn't losing weight. I knew there was something missing. I wanted to learn how many calories I was burning so could know how little or much I should be eating. That was the missing link. I have lost 150 lbs, and the only way did it was because I started using Jawbone Up.”
52
Take notes of interesting patterns and questions to ask participants as you go through it.
54
Minimize your drop off rate. Stay in touch with your participants. Be friendly and persistent.
55
Ask them questions if you want more clarity about their answers. Be appreciative and interested in what they have to say.
56
Do follow-up interviews with the most engaged participants. You already have rapport and can dig deeper into their reporting and actual behaviors.
• Use the built-in analysis tools
• Start by answering your initial question(s)
• Identify patterns and/or exceptions
• Share your findings with real examples
58
The good news is that by simply reviewing and making notes on the data, you have already started on the analysis.
59
Diary Study platforms offer tools to analyze the multiple choice and rating questions, reducing the work the researcher has to do.
60
If analyzing manually, data should be entered into a spreadsheet so it can be coded, refined and sorted.
What Further Analysis Should be Done?That depends on many variables, such as:
• What your initial question/s were
• How surprised you are by the data that has come in
• How it fits into other research methods
• Your final deliverables
61
Digital Diary Study Key Takeaways
Main Benefits of Using Diary Studies• To collect rich stories about how and why people do things
• Get intimate glimpses of how a product fits into people’s lives
• Allows for understanding evolving use of a product
• Uncover latent needs for new product development
• Cost-effective way to collect contextual insights
63
Checklist for a Great Diary StudiesPhase 1, Prepare:
• Have a clear, focused research question to guide your efforts
• Have an appropriate mix of closed/open-ended questions
• Pre-testing it to ensure clarity and understanding
• Start small & recruit the right participants
• Reduce the burden on them
Phase 2, Execute:
• Watch the data and follow up with participants during the study
• Do follow-up interviews with the best participants
Phase 3, Analyze:
• Answer your initial question first
• Give yourself enough time to analyze the data
64
Tips for Building Your Own Diary StudyBefore you can use it, you may have to sell it. Here are a few things to keep in mind:
• It’s an extremely flexible tool; make it work for your needs
• Start small, let it prove it’s value to your organization and yourself (no-cost platform, 5-6 participants)
• Evangelize the insights and stories you collect to build more support.
• You become an expert by doing. So dive in!
65
For a 4-min video about diary studies: www.gotoresearch.com/speaking/sara/
Questions about diary studies? Ask!
@ maybanks