signals research plan

Dashboards

February 2016

Copilot UserResearch

ABOUT THIS RESEARCH

The Copilot team wants to help Copilot users be more efficient by surfacing signals to “what needs my attention right now?”

These signals might include copy needing an edit or a story that is going viral or a story pitch that needs approval.

Through user-centered design methodologies, the Copilot team will gain a more empathetic understanding of “signals” so we can then design better experiences for our users.

How might we help Copilot users create, edit and discover the content that matters most and limit the reliance on search?

COPILOT USER RESEARCH 3 of 24

TL: DR


WHATThe Copilt Nimitz team is conducting research to better understand:

How might we help Copilot users create, edit and discover the content that matters most and limit the reliance on search?

HOWFour user-centered design methods will be used:

Card Sorting

Empathy Prototype

Test Prototype

Wine and Dine

WHENThe research will be conducted between February 15 and March 25.

THE RESEARCH TEAM


Laura Cochran

Caroline Casanova

Nadia Rivadeneira

COPILOT DESIGN PROCESS


Understand

Define

DivergeDecide

Prototype&

Validate

CONTENTS

Questions and AssumptionsMethodsResearch Design

1016

7


QUESTIONS & ASSUMPTIONS

QuestionsWhat is “my stuff?” What is the 80% of “stuff”

editors need to find faster?

What does top performing mean? How does it

help editors make decisions?

What signals tell editors how things are moving

along?

Is how many actions performed on a piece of

content in the last 24 hours a signal of “progress?”

Certain content types were left out of the

dashboard modules. Are these content types

meaningful to display? Is it obvious what isn’t

included in the feeds?

Is there value in having two separate dashboard

When is it most meaningful to look at the current

dashboard? What data is relevant to display on

the dashboards?


How do editors rank the importance of published,

scheduled, modified, uploaded and top

performing?

Do editors want to choose modules to display on

the dashboards?

Is adjusting timeframe important for top

performing?

What actions does an editor want to take on

content?

What is missing from the dashboards?

Is the photo upload obvious? Is it useful?

Is it clear that the quick create is a way to create

an article?

What is surprising about the way the dashboards

are being used?

WHAT ARE WE VALIDATING?

AssumptionsEditors will use the trending dashboard during morning planning to make decisions about what to

write.

Editors only care about what’s trending on Google, BING, Alexa & Chartbeat.

Editors know what is trending already or have sources to get this information.

The content dashboard will help editors find what they need faster.

Editors need to create articles and upload photos faster.

Content type and the content title is most useful to display all the time.

Editors need to get to the edit page for content faster.

The dashboard will limit the reliance on search.

10 items displayed is meaningful.

Recently scheduled content is useful.

Status is the best way to group content.

The dashboards should tell editors 80 percent of what they need to know.

What am I waiting on? Who is waiting on me?

Who do I need to harrass?

What is ready to be programmed?

How are things moving along?

How is my content performing?

What is trending?

METHODS

There is a wide range of methods available for user experience assessments. Every project will not use the full set of methods, but nearly every project will benefit from a range of methods. These methods are determined based on the types of information you hope to learn.

Attitudinal vs. BehavorialQualitative vs. QuantitativeContext of Use

For this project, we’ve determined card sorting, paper prototypes, a test prototype, as well as, a wine and dine would be the best way to better understand our user’s attitudes and the context in which they might use a content dashboard.


CARD SORTING

HOW

One-on-ones let you hear the participant thinking aloud while sorting the cards and observe their

reactions and thought processes. You can also conduct card sorting through Optimal Workshop.

WHYHow does this system help people make decisions? What is the right workflow and language?

Card sorting is a method that helps you better understand your user’s mental model and what their

expectations and needs are. There are two types of card sorting. An Open Card Sort has open-

ended labeling combined with a grouping exercise, which allows you to gather inspiration on the

right language to use for your information hierarchy. A Closed Card Sort allows you to test your

labels to see if they are intuitive.

PROTOTYPE FOR EMPATHY

WHYHow do people expect to use the system?

When we think about prototyping, we often think about evaluating and validating solutions.

Prototypes can also be used to gain empathy, exposing different information than card sorting,

interviewing or observations might. Empathy prototyping helps you gain understanding of people’s

mindsets around certain issues, while a solution prototype helps you gain understanding around

your solution. Here, we are creating an environment to gather deeper insight before solutionizing.

HOWEmpathy prototyping builds upon foundational information you’ve already gathered through one-

on-one interviews, secondary research or another means that helped you understand the design

space. First, the team should think about what topics in the design space they still want to learn

more about. Listing questions and assumptions can help you surface these topics. Then, you can

brainstorm ways you might investigate the subjects you still have questions about. Following the

brainstorm, you can create prototypes for empathy to test with users.

Share the prototypes with users asking questions focused on expectations and perceptions.

PROTOTYPE TO TEST

WHYAre user needs met? Is the user experience optimal? Did we miss any major flaws in the system?

Once you have narrowed down your solutions, you are ready to create a prototype to share the

experience or experiences with your users. Prototyping to test lets your users react to the solutions

and probe whether or not your solution decisions are optimal. It is an opportunity to test the

perceptions of your users and their needs.

HOWAt this stage, the prototype can still be low-res visually but the content and flows you want to test

should be at a point where you anticipate a user can successfully interact with your solution and

accomplish tasks you want to test.

Don’t spend too long on one prototype. You do not want to get too emotionally attached to one

solution. Focus on what you are hoping to test with the user -- what behavior do you expect -- and

create a prototype that lets you investigate the user’s perception of this expectation.

Create a task-based discussion guide focused on interactions and completion of tasks.

WINE AND DINE

WHYSurveys or focus groups do not help create a vision for the future. People, when asked what they

want, are focused on current realitities, not what could be. Therefore, the information gathered is

incremental at best. “If I had asked people what they wanted, they would have said faster horses,”

Henry Ford once said in reference to the automotive industry.

A Wine and Dine is a more eloquent way to better understand people’s unmet needs, aspirations

and behaviors through conversation over food and drinks.

HOWWine and Dines work best when participants are comfortable. You want participants to speak freely.

Keeping this in mind, finding a group of friends or colleagues works well. Ideally, in this group, you

can capture all of your extreme users.

Bigger is not always better. For this setting, six people is ideal. However, you need to adjust based

on how vocal you know the participants will be. You should balance the ability to give everyone an

opportunity to speak with the need for diversity in the group, or representation of your extreme

users.

RESEARCH DESIGN

The methods are part of the research design, the process to learn more about the people and their situation. You are designing research to learn: what user’s aspirations are; how they wish to be perceived; what their specific goals are; what the other relevant parts of their life are.

The plan includes the methods, team, schedule, budget and any other logistics important to the research design.


STAFFING

RECRUITINGNadia Rivadeneira

Laura Carroll

CARD SORTINGLaura Cochran

EMPATHY PROTOTYPINGCaroline Casanova

Gustavo Medina

Joe Bilt

Kelly Bucci

Laura Carroll

Laura Cochran

Mike Fennen

Nadia Rivadeneira

Yaya Wang

TEST PROTOTYPINGJoe Bilt

WINE AND DINELaura Cochran (moderator)

Laura Carroll (logistics)

Nadia Rivadeneira (note taker)

Caroline Casanova (note taker)

Joe Bilt (observer)

Ed Cudahy (observer)

Jon Tow (observer)

Mike Fennen (observer)

Gustavo Mediana (observer)

John Shehata (observer)

Copilot Team (observer)


SCHEDULE

18 of 24COPILOT USER RESEARCH

RecruitingGetting OrganizedCard SortingPaper PrototypingTest PrototypingWine and DineSynthesis

Week 2Week 2Week 3Week 3N/AWeek 4Week 4

BUDGET

19 of 24COPILOT USER RESEARCH

Supplies1-on-1 InterviewsWine and DineTotal

$50$70$250$370

CARD SORTING

PREPSurvey with 5 to 7 people asking what they categorize as “my stuff” and “top performing” then

ranking importance or each. Build cards based on answers.

TESTAsk 5 to 7 different people to put cards in “my stuff,” “top performing,” “both” or “other” using

Optimal Workshop tool.

PROTOTYPE FOR EMPATHY

PREPCreate paper prototypes to gain insight into two areas. The first set will show “signals” of progress to

gather insight into what progress is and how it helps editors prioritize. The second set of prototypes

will show different types of metadata and display of this metadata on the dashboards. Schedule

wine and dine and 1-on-1 interviews. Create discussion guide.

TESTShow paper prototypes at wine and dine and 1-on1 interviews. Gather feedback with the discussion

guide.

PROTOTYPE TO TEST

PREPMake sure a dashboard for scheduled brand is in production or in staging. Create task-based

discussion guide to explore usability issues and major flaws in the current system. Schedule 1-on-1

interviews and wine and dine.

TESTConduct 1-on-1 interviews with 5 to 7 people. Share test prototypes at wine and dine and gather

feedback. Follow discussion guide.

WINE AND DINE

PREPCoordinate with Kate to pick a date, determine catering options and make sure technology setup is

complete. Create invitations for two Wine and Dine events, one with GQ and one with Bon Appétit.

Invite 5 to 7 people to attend the event, keeping the extreme users in mind. Create discussion guide

for the event and conduct a mock Wine and Dine to prepare for the event. Determine sketching

exercises to gain insight into how editors priortize and what their current tools help them do.

TESTConduct sketching exercise. Share paper and test prototypes at wine and dine and gather feedback.

Gather general feedback from participants and moderate discussion.