Modern UX research in action:10 research stories

MO

DER

N U

X RE

SEA

RCH

IN A

CTI

ON

: 10

RESE

ARC

H S

TORI

ES

2

IntroThe customer journey is often more complex than simply visiting a website.

When most people think of remote usability studies, they think of testing the user experience on websitesand apps with participants sitting at home with their device or in front of a computer. And while it’s important to provide an awesome online experience, the customer journey is often more complex than simply visiting a website for 15 minutes and making a purchase or exploring a new app for the first time.

These days, there’s a lot more you can do to uncover insights about how your target customers interact with your brand. With modern UX research tools,

At UserTesting, we’ve set out to push the limits of our own platform, craft some unusual studies, and gather insights beyond traditional website usability testing.

it’s possible to test customer experiences across all channels, from websites and mobile apps to in-person experiences and even physical products. For example, UserTesting’s mobile recorder allows you to record experiences beyond the device, so you can observe the participants’ surroundings and watch them interact with items in the real world. You can also study customer experiences in specific locations, like retail stores, restaurants, and public transit.

Plus, by getting creative with different methodologies, you can observe how people interact with your productover time, how different audience segments perceive your brand differently, and how you measure up to thecompetition in the eyes of your target market.

MO

DER

N U

X RE

SEA

RCH

IN A

CTI

ON

: 10

RESE

ARC

H S

TORI

ES

3

Yelp’s mobile appdestination studyWHAT WE STUDIEDYelp’s mobile app is a bit of an enigma. It’s beloved by many, and it has tens of millions of downloads in the App Store and the Play Store... as well as thousands of 1-star reviews. However, reading through the one star reviews, it’s clear that many are actually reviews for businesses that users must have accidentally submitted via Yelp’s in-app “Leave a review about this app” messaging. Oops!

The reviews of the Yelp app in the App Store don’t tell the full story about the user experience on the app itself. So the UserTesting team decided to conduct a study to explore how people use the app out in the real world.

NOW, CONSIDER THESE QUESTIONS:Rather than doing a traditional usability study, we decided to launch a destination study using UserTesting’s mobile recorder to get some folks out on the street in search of a restaurant to grab a bite to eat.

With their smartphone in hand, we asked our participants to tell us a bit about how they typically use Yelp, and then we had them use the app to find a lunch spot nearby.

MO

DER

N U

X RE

SEA

RCH

IN A

CTI

ON

: 10

RESE

ARC

H S

TORI

ES

4

WHAT WE LEARNEDParticipants liked Yelp because it offered all the important information they would need to choose a restaurant or other local business: hours, ratings, phone numbers, a map feature, etc.

The users felt like the pictures and reviews in Yelp helped to set realistic expectations of what they would actually encounter when they arrived at the restaurant in person.

Because there is so much information available in Yelp, our participants pointed out that having filters is a necessity—but the filters can get a little overwhelming and confusing.

WRAP-UPBecause we ran this study with users walking around on the street, it felt like we were actually walking alongside our users as they took action. We got realistic feedback on how users interact with the app’s features in a distracting setting like a busy street while they’re hungry. We also got to see users interacting with the map and directions in real time.

MO

DER

N U

X RE

SEA

RCH

IN A

CTI

ON

: 10

RESE

ARC

H S

TORI

ES

5

Fitbit device and applongitudinal studyWHAT WE STUDIEDWearable technology for health and fitness has been on the rise for awhile, and Fitbit’s products are some of the most popular. We wondered:

Does a band around your wrist really motivate you to exercise? How drastic is the impact on behavior changes? What causes people to keep wearing this device?

HOW WE DID THE RESEARCHWe ran a longitudinal study with five users and measured how (if at all) user behavior changed while wearing the Fitbit.During the first part of the study, we asked users to demonstrate how they interact with their Fitbit and the accompanying app. In the second part of the study, we checked in a week later to find out how often they had used their Fitbit in the last week and what, if anything, had changed in their behavior or habits since wearing the Fitbit.

“It makes me want to get up and move around when I have it on.”

MO

DER

N U

X RE

SEA

RCH

IN A

CTI

ON

: 10

RESE

ARC

H S

TORI

ES

6

WHAT WE LEARNEDAfter one week of using the Fitbit, all five users stated they had noticed a change in their behavior and habits since using the Fitbit.

These changes included users stating they felt more inclined to stay active, especially in terms of how much they walked, and they felt motivated to keep track of what they ate throughout the day.

However, there was still room for improvement. Users mentioned some of the app’s features could have been explained more thoroughly, and several users wanted a tutorial to help them set the device up andbecome familiar with the functions.

WRAP-UPFor the users in our study, a simple wristband really did impact how much they decided to exercise each day. The constant feedback and accountability users got from Fitbit helped them stick to their goals longerand made them more aware of their eating and exercising habits. It’s not just the device that leads people to have healthier life; it’s the fact that Fitbit alerts people to lead a healthier life and gives them the awareness to change.

MO

DER

N U

X RE

SEA

RCH

IN A

CTI

ON

: 10

RESE

ARC

H S

TORI

ES

7

Nest LearningThermostat unboxingWHAT WE STUDIEDWatching customers unbox, assemble, and install your physical product can be as insightful as following what people are doing on your website. We ran an unboxing study with users who recently purchased the Nest Learning Thermostat to see how the product compared with their expectations.

HOW WE DID THE RESEARCHUnlike ordinary usability studies that involve digital properties like websites and apps, for this study, we were only interested in the real-world experience with the product, so we launched a beyond-thedevice study. Participants recorded their unboxing experience via their smartphone camera and UserTesting’s mobile recorder. They spoke their thoughts aloud as they opened the box, assembled the thermostat, and installed it in their home.

“There’s a level on it too, right on the device, right? That’s awesome.”

MO

DER

N U

X RE

SEA

RCH

IN A

CTI

ON

: 10

RESE

ARC

H S

TORI

ES

8

WHAT WE LEARNEDOur participants found several unexpected delights as they installed their devices. One customer was happy the device had a back plate because it wouldn’t damage the paint on the wall. Another was pleased that it came with a built-in level, making installation a snap.

It’s valuable to see how customers discover your product’s features in real-time to catch their initial impressions, rather than waiting for them to write a review or complete a follow-up survey.

WRAP-UPHaving users record the whole process of unboxing lets you get a peek into the world of first impressions and “fresh eyes.” It’ll help you better understand what delights and frustrates users, and you’ll figure out how to improve different aspects of your product.

MO

DER

N U

X RE

SEA

RCH

IN A

CTI

ON

: 10

RESE

ARC

H S

TORI

ES

9

Apple Watch exploratory studyWHAT WE STUDIEDThe Apple Watch was one of the most anticipated product releases of 2015. We wondered if it would live up to the hype. Would it make users’ lives easier? Which parts of the user experience would be awesome, and which could use some improvement?

HOW WE DID THE RESEARCHWe ran a qualitative, remote usability study with seven early adopters. Our participants ranged in age from 18 to 54 and were located in the US and the UK.

“Controlling my iPhone (and what songs are playing) through mywrist is really cool and useful.”

WHAT WE LEARNEDMost participants agreed that the Apple Watch takes a little while to get used to, since the interface is so different from other products. Some of the users said they wished the Apple Watch had certain features, like the ability to adjust the screen brightness and timeout, that it actually does have; they just weren’t immediately apparent.

Three of our participants didn’t receive notifications on the watch when they got a text message, and two weren’t sure whether setting “Do Not Disturb” on their iPhone would affect their Apple Watch, too. (It turns out that it will if you select the “Mirror iPhone” setting.)

MO

DER

N U

X RE

SEA

RCH

IN A

CTI

ON

: 10

RESE

ARC

H S

TORI

ES

10

WHAT WE LEARNEDStill, participants agreed that the Apple Watch did a great job with “little things that make life easier.” Sending a text was easy, thanks to goodvoice-to-text recognition, emojis, and pre-programmed common replies like “Hey” and “On my way.” One user liked the ability to control themusic playing from his iPhone through his Apple Watch, especially if his iPhone is in his pocket or plugged into speakers across the room. Several users noted they didn’t have to look at their phone as often, so they wouldn’t be distracted by other apps every time they wanted to read a notification.

WRAP-UPIn spite of the initial learning curve—and the fact that many third-party apps weren’t up to snuff yet—all seven of our participants said theywould use the Apple watch on a daily basis, and they’d be very likely to recommend it to a friend. And, of the five participants who had usedother smartwatches before, all five preferred the Apple Watch.

MO

DER

N U

X RE

SEA

RCH

IN A

CTI

ON

: 10

RESE

ARC

H S

TORI

ES

11

Snapchat demographiccomparison

WHAT WE STUDIEDIf you’re over 25, you may find Snapchat to be a bit of a mystery. It has a reputation as the app that teenagers use to share self-destructing pictures in a place where parents won’t find them. Its interface seems to be designed for a crowd that has grown up using mobile devices and apps rather than the desktop web. Plus, its website, billboards, and marketing materials tend to lack a clear explanation of why you should use it.

Here at UserTesting, this made us wonder how users of different age groups interact with the app for the first time. Do they get it? Do they like it? Do they feel like it was made for them?

“I don’t know what the number 1 means... what the ghost means... what the rectangular smiley face means.”

HOW WE DID THE RESEARCHWe ran a remote usability study with 18 first-time Snapchat users: six who were between the ages of 18 and 24, six who were 25-40, and six who were over 40. We wanted to find out how the different age groups explored and got the hang of the app—and whether or not they felt like the app was made for them.

MO

DER

N U

X RE

SEA

RCH

IN A

CTI

ON

: 10

RESE

ARC

H S

TORI

ES

12

WHAT WE LEARNEDWhen our users opened the app for the first time, they arrived straight on the camera screen, and they saw their own face via the front-facing camera. (One groaned, “The first thing I see is myself with a double chin.”) A few wished there had been a tutorial to help them get started. It turns out there is a video tutorial, but only three participants discovered it, and one didn’t understand how to play it.

Older participants acted more irritated than their younger counterparts by the lack of instructions and labels on the app’s icons. By contrast, the youngest age group was more confident poking around the app to discover each feature and the meanings behind the icons.

Even though the under-25 age group had the easiest time navigating Snapchat, a few of them still weren’t convinced that they needed it, noting that they could already share pictures using other apps. Older users were less likely to want to continue using the app, and several said it seemed like it was for “kids these days” instead of users their own age.

WRAP-UPOf the users who did enjoy Snapchat and wanted to keep using it,the primary reason was because they found it fun. None of the usersdescribed the app using words like “useful” or “helpful.” But Snapchat’smillions of users prove that sometimes, having a point… isn’t the point.It’s about providing a fun and addictive service that resonates with justthe right audience.

MO

DER

N U

X RE

SEA

RCH

IN A

CTI

ON

: 10

RESE

ARC

H S

TORI

ES

13

Google Wallet destination studyWHAT WE STUDIEDIn a world of smartphones, smartwatches, and the Internet of Things, credit cards may soon be a thing of the past. Mobile payment apps like Google Wallet store users’ debit, credit, and gift card information, allowing them to make purchases online and in stores that use NFC—without the need for a plastic card.

We decided to explore how first-time users interact with Google Wallet. How do they feel about keeping their payment information in their phones? What’s it like to make a purchase in a store using only the Google Wallet app?

HOW WE DID THE RESEARCHTo answer these questions, we ran a destination study with five users. We asked them to install and explore the app on their phones, and we gave them each a $10 Walgreens gift card to load into the app. Then we asked them to go to their nearest Walgreens and buy something under $10 using the app.

“I don’t know if you’re able to scan the bar code directly.You may have to type it in.”

MO

DER

N U

X RE

SEA

RCH

IN A

CTI

ON

: 10

RESE

ARC

H S

TORI

ES

14

WHAT WE LEARNEDIn general, users were excited about the app. It was easy to set up, the interface was intuitive, and our participants immediately understood the navigation.

While Google Wallet requires users to select and enter a PIN to access the app, a couple of our participants wanted more reassurance that their information would be safe.

Making a purchase in a brick-and-mortar store was the most confusing part of the experience. Two participants said they wished they’d had clearer instructions while making an in-store payment for the first time. Three users noted that the store employees weren’t familiar with Google Wallet, so they weren’t able to offer any additional help to the users.

WRAP-UPMobile payment apps like Google Wallet face the challenge of balancing convenience and security. While some users are quick to adopt technology that makes their lives easier and eliminates the need for repetitive tasks, others prefer to wait until more people they know have tried it and found it to be safe. Most folks aren’t throwing away their plastic cards just yet. But if Google Wallet and its competitors continue to focus on providing asecure, easy-to-understand, and delightful user experience, we may start to use mobile payment apps more often than credit cards in the very near future.

MO

DER

N U

X RE

SEA

RCH

IN A

CTI

ON

: 10

RESE

ARC

H S

TORI

ES

15

Virgin America competitor studyWHAT WE STUDIEDMany travel companies have a strikingly similar user experience. When users go to book a flight online, they tend to expect a website that looks more or less the same as all of the other travel companies out there.

Virgin America has turned those expectations upside down in the hopes of creating a distinctive, memorable, and hopefully excellent customer experience. Everything from their ads to their in-flight safety messaging goes against the traditional industry norms.

But is it effective? Do users understand how to choose a flight? Do they find the workflow helpful or confusing? We wanted to find out if Virgin America’s design choices create a delightful user experience—or just disorient new customers.

HOW WE DID THE RESEARCHWe ran a direct comparison study with six first-time Virgin America users. We asked each of them to select and book a flight from Chicago to San Francisco on VirginAmerica.com, and then select and book a similar flight using a different website of their choice. We measured how easy or difficult they found the flight booking process on each website, and we tracked the time it took them to select a flight on each website. Finally, we asked them to verbally compare their experiences on the two websites.

“I’m guessing I have to click on the Main Cabin. Therewe go; now we’re moving.”

MO

DER

N U

X RE

SEA

RCH

IN A

CTI

ON

: 10

RESE

ARC

H S

TORI

ES

16

WHAT WE LEARNEDUsers noticed right away that the experience on Virgin America was different. While some features were confusing at first (such as selecting departure dates on the calendar and picking a flight out of a list of options), the users reported that the process felt streamlined and enjoyable. All six participants were able to successfully select and book the flight. It took users a longer time on average to select a flight on Virgin America than on the other websites, but users still thought it was easy to use.

WRAP-UPConventional wisdom tells companies to find out their users’ expectations and then match them. But many customers are more open-minded about online experiences than companies expect. Brilliant visual design, straightforward copy, and an intuitive interface can go a long way in helping users adapt to a new way of doing things. And, as we saw with our study of Virgin America, users are willing to excuse some hiccups in their workflow if the design appeals to their emotions and makes them feel delighted.

MO

DER

N U

X RE

SEA

RCH

IN A

CTI

ON

: 10

RESE

ARC

H S

TORI

ES

17

Netflix redesign usability studyWHAT WE STUDIEDWith over 62 million users worldwide and 10 billion hours of movies and shows streamed in Q1 2015, it’s clear that Netflix has been doing something right. That’s why their website redesign in Q2 2015 could be considered a risky move. By changing something so popular, Netflix ran the risk of confusing and alienating existing users who loved the old design. It was critical for Netflix to keep their existing customers happy while also attracting new ones.

HOW WE DID THE RESEARCHWe recruited five Netflix subscribers to participate in a remote usability study. We screened for people who primarily watch Netflix on their desktop computer.

WHAT WE LEARNEDNetflix’s new website was designed to be more consistent with its mobile, console, and smart TV apps. This creates a more unified omnichannel experience for customers who use Netflix on different devices. Most of our participants said the new design felt newer, sleeker, and more up to date.

“I’m guessing I have to click on the Main Cabin. Therewe go; now we’re moving.”

MO

DER

N U

X RE

SEA

RCH

IN A

CTI

ON

: 10

RESE

ARC

H S

TORI

ES

18

WHAT WE LEARNEDIn the new design, users can hover over an image to expand a slideshow of select clips, along with more information on the movie. While users liked the idea of being able to learn more about a movie at a glance, the interface may take some getting used to. Four participants accidentally started playing the movie when they were attempting to find more info. Others lost their place when they accidentally hoveredover a different movie while viewing details of one.All five of our participants pointed out that they were glad the redesign still offered a great browsing experience, and they found it very easy to find movies that matched their personal taste preferences. Still, they agreed that there was one area that had room for improvement: the “My List” feature could be a little overwhelming, and several users had trouble locating it.

WRAP-UPWith more and more people interacting with companies on multiple devices, it’s more important than ever to create a consistent experience across all channels. While no design is perfect, users appreciate knowing what to expect from a company. The “modern, sleek” feel that users noticed in Netflix’s new site may be doing just that.

MO

DER

N U

X RE

SEA

RCH

IN A

CTI

ON

: 10

RESE

ARC

H S

TORI

ES

19

Starbucks app destination studyWHAT WE STUDIEDThe Starbucks mobile app introduced an exciting new feature in Fall of 2015: the ability to order and pay for drinks ahead of time, skipping the line in the store. While this seems like a huge convenience for many coffee drinkers, we wondered: Will Starbucks customers understand how to use the feature right away? How smoothly will the process of ordering, paying, and picking up go in the real world? What will people like and dislike about the experience?

HOW WE DID THE RESEARCHWe recruited six Starbucks mobile app users from around the United States for a two-part study. The first part of the study was a remote user interview. We asked participants how they typically use the Starbucks app, when and where they use it, and what value they find in it. The second part of the study was a destination study in which participants ordered a drink via the app and picked it up in a Starbucks location.

“I don’t see my drink anywhere. Maybe I should talk to the cashier?”

MO

DER

N U

X RE

SEA

RCH

IN A

CTI

ON

: 10

RESE

ARC

H S

TORI

ES

20

WHAT WE LEARNEDIn the interviews, we found that the two main reasons why participants used the app were status and convenience. Participants said they felt like VIPs when they used the app to gain rewards and load up their Starbucks card. They also felt like it was convenient to be able to keep their Starbucks card on their phone, where it was always handy. In the destination study, we found that while it was easy to locate the nearest Starbucks, actually placing the order presented more challenges. Some users weren’t sure whether they needed to pick a location or a drink first. Two encountered errors stating their order could not be placed, but no explanation of why. Picking up the drink was the most confusing part of the process: some participants understood where to pick up the drink, others didn’t, one participant’s drink wasn’t ready when she arrived, and one had to start the process all over by getting in line to order her drink.

WRAP-UPOverall, participants had positive experiences, although no order and pick-up was seamless Participants were excited about the idea to use the app to enjoy their favorite drink, even if it took a little longer than expected. Even though the ordering process wasn’t perfect, all the participants got what they wanted in the end—their favorite Starbucks fix! When technology didn’t quite work as planned, great service and loyalty to the brand stepped in to compensate.

MO

DER

N U

X RE

SEA

RCH

IN A

CTI

ON

: 10

RESE

ARC

H S

TORI

ES

21

iPhone 3D Touch exploratory studyWHAT WE STUDIEDWith the introduction of 3D Touch for iPhone 6s and 6s Plus, the iPhone screen acts as a pressure sensitive surface, making way for an additional “dimension” of interaction beyond swiping, pinching, and tapping. Users can now apply different amounts of pressure to initiate different types of interaction on their phone.

HOW WE DID THE RESEARCHWe invited six iPhone 6s users and two iPhone 6s Plus users to explore the feature in a remote usability study. We asked them to try out the Quick Actions in the home screen, the “Peek” and “Pop” gestures in a dummy email inbox, and the keyboard-turned-trackpad in Notes. All participants had already played around with their new phones, so they all had some degree of familiarity with 3D Touch or “Force Touch” prior to the study.

WHAT WE LEARNEDMost participants admitted they were still getting used to the feature and felt they’d need to break their old habits from prior iPhones before they could take full advantage of the 3D Touch. The majority of participants encountered some trouble in applying the right amount of pressure. Two participants wanted to see some type of tutorial for 3D Touch to help them better understand how to correctly apply the different amounts of pressure.

“If I press on it, there’s a little arrow up top there. If I swipe it,I can do these different tasks.”

MO

DER

N U

X RE

SEA

RCH

IN A

CTI

ON

: 10

RESE

ARC

H S

TORI

ES

22

WHAT WE LEARNED Even without mastery of 3D Touch, participants acknowledged that this new system allowed for smoother transitions from one interaction toanother, and it saved them time by cutting out multiple steps in an action. One participant likened the 3D Touch to an advanced right click, while several other participants admired the additional “layer”or “depth” of navigation that it allowed.

WRAP-UPMore than anything else, the novelty of this technology stuck with participants as they worked to get used to the 3D Touch. When we asked participants to choose adjectives to describe the technology, a majority chose words like, “inventive,” “impressive,” “creative,” and “fun.” Not only did these adjectives reflect an overall positive experience, but it emphasized how fresh the experience was for them.

HEADQUARTERS144 Townsend St.San Francisco, CA 94107__

WEBUserTesting.com

SALES1-888-877-1882

UserTesting is an on-demand human insight platform that quickly gives companies a first-person understanding of how their target audience behaves throughout any experience and why. To learn more, visit www.usertesting.com.

About UserTesting

© 2018 UserTesting, Inc. All rights reserved