ethnography in software design - an anthropologist's perspective

Ethnography in Software Design An Anthropologist’s Point of View ACE! Conference 2015, Krakow Poland

: https://www.flickr.com/photos/101187156@N03/14366224997/

https://www.flickr.com/photos/101187156@N03/14366224997/

Kelly Moran Lead Design Researcher

@Kel_Moran

Education @ CSUN

Early Influences

Uzbekistan

Masters Program @ UNT

Study Abroad @ Saudi Arabia

Worked for:

Worked with:

“Uncover user needs, Design great solutions,

and build out solutions to launch.”

Dist

ill M

eani

ng fr

om O

bser

vatio

n

Build

the

Back

log

CONTACT BUILDING & EVOLVINGFOCUSED INNOVATION

Only a subset of these activities will be appropriate for any given project.

Qualitative Research

Quantitative Research

Affinity DiagrammingConstruct themes from qualitative data.

Analysis & Synthesis Opportunities

Ideation & Iteration

Validation & Evaluation

Approach Planning Design Research Experience Strategy & Strategic Ideation PlanningDefinition

Iden

tify

Chal

leng

es in

Con

text

Set t

he S

tage

REVEALING REALITY

Foundational Analysis

Heuristic EvaluationIdentify inital breakdowns and opportunities

Digital Marketing SWOT AnalysisIdentify strengths, weaknesses, opportunities & theats.

Stakeholder InterviewsUnderstand staheholders’ business goals & strategy.

Technical Organization Capability AanalysisAnalyze existing skills and toolsets.

Scenario-based System WalkthroughsDemo of existing solution.

Metrics EvaluationEstablish quantitative baseline of engagement & conversion data.

Content InventoryCatalog the content of the site or application.

Competitive AnalysisEvaluate competitors and comparables on specific axes.

User Experience

Data

Technology

Enterprise Architecture Capability AssesmentReview existing enterprise technology infrastructure.

Solution(s) Architecture AssessmentReview existing application(s) architecture.

RFP RequestDeliver proposal to defined scope document and existing requirements.

Project ApproachAsses a possible project and plan high level approach.

Align & Assess WorkshopAssess readiness across core capabilities.

Backlog GroomingReprioritize backlog, add new stories.

Zero Feature ReleaseDemonstrate CI, automated testing, core solution setup.

High Level Technical ArchitectureDescribe high level architecture, including packaged components.

Development Infrastructure ConfigurationContinuous integration setup.

Technical Package Identification & EvaluationPerform product evaluations for package solution components.

Architecture Spikes & Proofs of ConceptProve candidate architectures via top bottom spikes.

Existing Research ReviewMarket research, website feedback, corporate strategy, etc.

Brin

g th

e So

lutio

n in

to F

ocus

Research PlanDesign activities to meet research goals.

Contextual InquiriesObserve & document user in context & environment.

User Journals & DiariesUser document their experiences over time.

Participatory DesignCo-creation explorations with users.

Card SortExplore users’ mental models for content and labeling.

SurveysSolicit structured feedback from users.

User Workflow ModelingVisually document workflows & work systems.

Ideation WorkshopsImmerse stakeholders in data and brainstorm opportunities.

Persona DevelopmentCreate customer types to document observed behaviors and values.

Consolidated Workflow DiagramAggregate individual user workflows into one diagram.

Current Journey MapVisualize the user’s perspective of the current experience.

Quantitative Data VisualizationPresent quantitative data visually.

Opportunities Generation & EvaluationOpportunities & prototype choice.

Opportunities MatrixPrioritize in three dimensions, including user experience impact.

Marketing OpportunitiesStrategic planning of owned, earned & paid online tactics.

Engagement PlanDevelop the strategic and tactical plan to achieve the client’s goal.

Experience-Driven RoadmapPlan how great UX can be achieved through the design.

Design PrincipalsArticulate design principals to guide the design and development process.

Requirements & User Stories DefinitionWrite user stories based on detailed user scenarios.

Application & Navigation FrameworkValidated navigation and framework.

Wireframed Key WorkflowsValidated wireframes of key workflows.

Visual Design LanguageVisual design language defined.

Information ArchitectureMap the product from the users’ point of view.

StoryboardsIllustrate graphical representations of scenarios.

Qualitative & Quantitative Data SynthesisAnalyze validation data..

User ScenariosWrite detailed narratives for user experience flows.

Future Journey MapVisualize the user’s future, improved experience.

Concept ValidationValidate design prototypes through user feedback.

Application & Navigation Framework ConceptsCreate models for the navigation & framework of the application or site.

Workflow ConceptsDraw high-level wireframes for key workflows.

Visual ExplorationExplore different visual treatments and styles of the application or site.

KANO Feature PrioritizationPrioritize features with users through KANO analysis.

we make software make sense.TM

Heuristic Evaluation Findings and themes from the research synthesis Personas Consolidated Workflow Diagram Journey Map Stakeholder workshop: Findings presentation and Opportunities brainstorm Experience Principles Opportunities generation and consolidation Documentation/Data Review Identify participants, begin coordinating recruiting Scenario-based Product Walkthroughs Stakeholder Interviews SME Consultations Heuristic Evaluation Revealing Reality Continue contextual inquiry (CI) participant scheduling CI protocol development Workspace Observations

“Design Research directs the strategy & design for

our software projects”

WHAT DOES ANTHROPOLOGY

LOOK LIKE?

http://michaelvhurley.com/2013/01/04/my-worthless-degree/

Anthropologist?

CSI Vegas?

“Anthropology demands the open-mindedness with which one

must look and listen, record in astonishment, and wonder that which one would not have been

able to guess" – Margaret Mead

WHAT IS ETHNOGRAPHY?

10 things to know

1

Ethnography is the descriptive study of people and cultures

@Kel_Moran

2

Ethnographic Research is…

An approach not a specific method

@Kel_Moran

3

Ethnographic Research…

Favors qualitative over quantitative - the ethnographer

seeks to provide “thick” description

@Kel_Moran

;-}

;-/

;-)

4


Conducted in context, typically over an extended period of time, and holistic, seeking the wider

picture@Kel_Moran

To understand the pen, you must understand the paper.

5


Systematically (purposefully) conducted, but is responsive to

emerging trends and themes (flexible).

@Kel_Moran

6


Utilizes key informants who can act as guides and help provide

access to the community

(Also offer “reality checks”)@Kel_Moran

7


Seeks out the insider (emic) perspective and layers in outsider

(etic) insights and interpretations

@Kel_Moran

8


Generative - done to discover new information, not to test

existing hypotheses

@Kel_Moran

“Never theorize before you have data. Invariably you end up

twisting facts to suit theories instead of theories to suit facts.”

- Sherlock Holmes (Arthur Conan Doyle)

9


Seeks to tease out the implicit, not typically stated, features of a group

“Making the familiar strange and the strange familiar”

@Kel_Moran

The Nacirema

The NaciremaAmerican


Is inclusive, typically including a participatory component

10

@Kel_Moran

BONUS!

Bronislaw Malinowski was a Polish born Anthropologist who pioneered extended fieldwork.

@Kel_Moran

WHAT IS DESIGN ETHNOGRAPHY?

3 things to know

1Design Ethnography…

Is a way “…to increase the success probability of a new product or service or, more appropriately, to reduce the

probability of failure specifically due to a lack of understanding the basic

behaviors and frameworks of consumers.”

Salvator, Tony; Genevieve Bell; and Ken Anderson (1999) “Design Ethnography,” Design Management Journal (pp. 35-41). p.37 @Kel_Moran

2

Avoids an over-reliance on self-reported data

(what they say is not always what they do)

@Kel_Moran

3

Qualitative research, done in the context (environment) of the intended users, seeking to discover and understand their problems from their viewpoint,

with the designer’s viewpoint used to ideate potential solutions.

@Kel_Moran

INTEGRATINGIN-CONTEXT RESEARCH .

Discovery Phase • Designer pairs with a Researcher • Researcher leads with a background in the social

sciences • Designer assists

Researcher Designer

Design Phase • Roles switch, and the Researcher assists the Designer

ResearcherDesigner

After Detailed Design • Designer typically is embedded into the Development

Phase • Researcher comes back on board for user testing

DesignerDeveloper x N

Researcher

Discovery Design Build

Provides continuity and keeps the user’s voice present

User Focused Innovation

CASE STUDY

ENTERPRISE SOFTWARE ACCOUNT RECONCILIATIONS

The Client and the Product

Vendor of accounting software • Customer feedback of “too many clicks” and “hard to use” • Sales were lagging

Used in organizations with large, multi-functional accounting departments • Needed to be customizable • Should fit within a suite of other enterprise products

Research Basics

Who • 19 users at 6 user sites, plus 3 users inside the client company

• = 22 total observed users • Both primary reconcilers (doers) and reviewers (managers)

Where • 7 locations across 4 states • 2-4 users at each location

From 22 users: Around 800 unique insights and observations

were recorded

IN-CONTEXT OBSERVATION

“We have everything set up for you in a conference room.”

What do you see?

Contextual Learnings

• 5 out of 6 of the client-user groups observed used two monitors

• adding machines (calculators) still in use • typically in either a cube or an open workspace

• = lots of noise and movement • users “get into a zone” to focus on their work

• paper everywhere • one user had multiple post-it notes with the same number

for customer support displayed near the phone • User observed walking to a locked room (with a broken ankle)

to look up reference numbers

CONTEXTUAL LESSONS FROM USERS

Contextual Learnings

• With a lot on their minds, Excel is king • High cognitive load • High use of Excel quick keys • Importing data of multiple types into Excel as images • Highlighting and/or circling important data

http://icons8.com

http://icons8.com

“It’s better than having it rejected back.”

Contextual Learnings (continued)

• Attachments and cover sheets • Manager preference for a single attachment • Printing out documents from multiple sources to scan

them into one attachment • Making a Custom “cover sheet” in Excel to sum up the

work

“I look at [the product] as basically just a holding station.”


• “Roles” as defined in the software did not match the work-based roles of the users

• In several locations a “reconciler” also held the role of “administrator,” but juggling these in the software was cumbersome

• The administrator role could become overwhelming

Assumed User Role with Linear Workflow

Observed Interrupted User/Admin Workflow

StartAdministrative Work

Start EndUser Work

EndUser WorkUser Work

Interruption

“I have to change gear…to move to that other role.”

“Soon I won’t have any accounting work. It will all be

[administrative] work.”


• The problem goes beyond the system • Lack of technical support • The “real work” was done outside the system before we

arrived.

EXTENDED RESEARCH FOLLOW-UP RETROSPECTIVES

Understanding What We Didn’t See

• Visited 4 new groups • Plus an internal

check with the client’s accounting department

• Increased our overall reach

• Used a custom-built research activity

• Helped us understand how the users view their work stream, and where the software fits in

Now what do we do with all that data?

Synthesizing the Data - Major Insights1. Learning, training, & support 2. Don’t make it hard on us 3. We know who we are and

what we need 4. We need an agile, smart

workflow and tracking system 5. Don’t add to our cognitive

load 6. Be our partner 7. We have a lot going on

besides what you do for us 8. Collaboration, teamwork,

communication, & working with others

Insights Lead to Concepts

TRANSITIONING INTO DESIGN

Concepts Lead to Design• Organizing the system’s

workstream to match the user’s view.

• Renaming the “account home page” the “cover sheet” and emphasizing its similarities to the Excel-made coversheets.

• Bringing in more Excel patterns

• Users viewed the system as supplementary, and part of a larger process.

• Cover sheets were created and standardized by each accounting group.

• Observed high use of Excel.

Concepts Lead to Design (continued)

• Making both uploading and viewing attachments easier and more consistent with the rest of the UI.

• Overall more consistent UI

• Putting the information the users needed most to keep in mind in a prominent, and persistent, place on every screen.

• Uploading and viewing attachments was a common problem; and it didn’t match the rest of the experience.

• Inconsistent UI was jarring to navigate.

• Users had to scan the page to find the most relevant information on each screen.

VALIDATION TESTING

Three Pages Tested - Two Versions Each (revised and re-tested for a total of three testing rounds)

• Reconciler Dashboard • List Page - results of search

• Refined filtering • Details page - account home; i.e. cover sheet

User-Led Validation and Changes

• Most users don’t need the 6-currency view panel • Changed the default to 3 currencies

• Accountants need to be sure their entries save • Ties back to the generative research • Added a “save” confirmation button instead of auto-saving

• “Build-a-filter” concept performed better than the “expose all filters” option

• Attaching files to the cover page was validated as preferable to using a separate attachments page

BRINGING ETHNOGRAPHY TO YOUR PROJECTS .

Start Small - Start Anywhere!

Does not need to cover 7 regions across a continent (seeing a few users makes a difference)

Get into the environment (try Skype if needed)

Don’t go in with solutions in mind (don’t be too sure you know the problems already either)

@Kel_Moran

How to ObserveThe physical environment - open? cramped? hot? cold?

Lighting, noise - pleasant? distracting? manipulated by the user?

People - who interacts with whom?

Artifacts - equipment, paper notes, binders…

Document it all - notes, yes, but also photographs and audio/video if permitted.

@Kel_Moran

How to Ask QuestionsBe respectful, but don’t be shy - they have a job to do, and so do you

You are not an expert in their work/play - even if you think you are

Rephrase what they say and ask if you got it right - let them correct you even if you know you did

Avoid leading questions - ask them to describe instead

Take note of their ideas and ask “What problem does this solve?” - they’re not the experts at finding solutions, but they tried their best

@Kel_Moran

Honor the Idea of Reciprocity

Your users (research participants) are giving you something, so be sure to give something back.

@Kel_Moran