Doing Ethnography

Key principles

• Everything is strange– Question why?– Stop and reflect

• Members’ point of view– Developer themselves – why do that?

• No a priori expectations– Non judgmental– Data analysis and changing plans continuous

Before the study

• Recruiting participants– Access– Building trust

• Ethical issues– Relationship between observer and observed– Recognised ethical code– Participants’ right to withdraw at any point– Informed consent - ?– Data protection and anonymity

• Complementary methods– Observation– Interviews

Observation

A matter of legitimacy • One of the most important data collection tools of an

ethnographer. Originally, it was almost a requirement for ethnographic research. It provides legitimacy:– “To say the Yanamami do this and the Borroro do that, and to

be taken seriously, you have to have been there, seen them, and if not done it and brought back the T-shirt, at least captured and recorded their lives (these days on videotape). Fieldwork became de rigeur for ethnography and every practising anthropologist worth his or her salt had “his” (or “her”) tribe. This, then, is a cardinal thing to note about ethnography. Its practice is a particular form of legitimation. Ethnographers “know” in ways others don’t and can’t. And what they know derives in part from personal experience.” (Genevieve Bell, May 2004)

• As another data collection method it is helpful to triangulate the data.

Types

• According to the researcher engagement:

– Participant: the researcher acts as the informants:

writes code, attends meetings, discusses solutions.

Example: Sharp et. al. (2004)

– Non-Participant: the researcher only observes the

informants. He is “fly in the wall”. Example: De Souza

et. al. (2004);

• In any case, the researcher must document as much

information as possible in his/her field notes;

Field Notes

• A description of events, people, interactions, tool usage, things listened, heard, experiences, impressions;

• Be as detailed as possible, i.e., write down the higher number of details possible;

• Separate observations and quotes from the informants from impressions and comments from the researcher;

• A private document that can only be shared within the research team;

What to collect? Or the level of detail to collect?

• The researcher needs to decide the level of detail that he needs his observations (Randall and Rouncefield 2004, p. 64). – Gestures, eye gazes, movements as done by (Heath

and Luff, 1992) in their study of subway control rooms;

– Time spent in each activity performed by knowledge workers (Gonzales and Mark, 2004) and software developers (Goncalves, de Souza et al, 2009)

• This level of detail depends on the research questions. Every researcher needs to define what he/she needs.

Practical Tips• Write your fieldnotes, if possible, during the data

collection [in the field], or at the end of the day;• In the beginning of your study, write down as much

as possible to the level of detail you need:– Meetings, interactions, tool usage, phone calls,

process adherence, etc.• Everything or anything can be relevant at this point.

Often, at this point you do not know what is or is not important to your research

• Later, as your research progresses, field notes can be refined to focus on the “interesting” aspects only.

• It is important that the ethnographer is introduced to the informants by a respected member of the group.

• In the beginning, there is a natural “resistance” to the ethnographer, but this will cease as time progresses;

• Often, ethnographers have “key informants”, i.e., experienced informants that introduce the ethnographer, facilitate his access, answer questions, provide explanations, and so on.– Finding and cultivating them is important!

Structuring frameworks to guide observation

• - The person. Who? - The place. Where?- The thing. What?

• The Goetz and LeCompte (1984) framework:- Who is present? - What is their role? - What is happening? - When does the activity occur?- Where is it happening? - Why is it happening? - How is the activity organized?

When should I stop observing?

• No obvious ending. • Schedules often dictate when your study ends. • Otherwise, stop when you stop learning new things.

– when you start to see similar patterns of behavior being repeated, or

– when you have listened to all the main stakeholder groups and understand their perspectives.

What data to collect? (1)The following illustrative list (Crabtree, 2003, p. 53):• Activity or job descriptions.• Rules and procedures (etc.) said to govern particular

activities.• Descriptions of activities observed.• Recordings of the talk taking place between parties

involved in observed activities.• Informal interviews with participants explaining the detail

of observed activities.• Diagrams of the physical layout, including the position of

artefacts.

What data to collect? (2)

• Photographs of artifacts (documents, diagrams, forms, computers, etc.) used in the course of observed activities.

• Videos of artifacts as used in the course of observed activities.

• Descriptions of artifacts used in the course of observed activities.

• Workflow diagrams showing the sequential order of tasks involved in observed activities.

• Process maps showing connections between activities.

Data recording

• Notes, audio, video, photographs

• Notes plus photographs• Audio plus photographs• Video

Comparison of the three main approaches to data recording (1)

Criterion Notes plus camera Audio plus camera Video

Equipment Paper, pencil, and still camera.

Handheld recorder with a good microphone. Headset useful for easy transcription.

Video camera – handheld or fixed. Editing, mixing, and analysis equipment needed.

Flexibility of use

Very flexible. Unobtrusive.

Flexible. Relatively unobtrusive.

Needs positioning carefully to capture relevant activity. Obtrusive.

Completeness of data

Only get what note taker thinks is important and can record in the time available.

Visual data is missing. Notes, photographs, and sketches can augment recording but need to be coordinated with audio recording for analysis.

Collects thorough and detailed data, especially if more than one camera is used, but video material needs to be coordinated for analysis.

Comparison of the three main approaches to data recording (2)

Criterion Notes plus camera Audio plus camera Video

Disturbance to users

Very low. Low, but microphone needs to be positioned.

Medium. Camera can be difficult to ignore. Preliminary field studies and piloting with participants can help to lessen the impact.

Reliability of data

May be low. Relies on humans making a good record and knowing what to record.

High but external noise, e.g. fans in computers, can muffle what is said.

Captures detail of relevant activities, provided camera is positioned appropriately.

Analysis Transcription straightforward. Rich descriptions. Transcribing data can be onerous or a useful first step in data analysis.

Critical discussions can be identified. Transcription needed for detailed analysis. Can revisit permanent original record.

Critical incidents can be identified and tagged. Software tools for detailed analysis. Can revisit permanent original record.

Interviews

Interviews

• An approach for data collection used by different research methods, including ethnographic, qualitative and even quantitative;

• There are different types of interviews and each one of these types is more often associated with a particular research method

Types of interviews

• In general, interviews can be classified along two continua (DeWalt and DeWalt, 2002):– The degree of control by the researcher and

informants; and– The degree to which the stimuli (questions)

presented to each informant are uniform.

Degree of control by researcher

– In one extreme, the researcher has no control and he is a pure observer in an conversation. The informant has control about what he or she wants to talk about; and

– In the other extreme, the researcher creates a questionnaire or survey with pre-defined questions with no accommodation for the concerns or understanding of an individual respondent.

Degree to which questions to each informant are uniform

– In one extreme the survey is defined beforehand and is presented to all informants with no room for his/her specific concerns; and

– In the other end of the continuum, each conversation is unique, there is no attempt to ask the same questions in the same way to each informant.

Types of Questions

• The distinction among the types of interviews is closely linked to the type of questions that are asked:– Closed questions: well-defined questions

with a limited number of answers. The informant has to choose among this set of answers;

– Open questions: ill-defined questions that require an opinion, an explanation, etc.

Example of Closed Questions

• Please indicate your level of agreement for the following statement: “My colleagues provide timely information about their changes in the source code that affect my work”. a. Strongly disagreeb. Disagreec. Neutrald. Agreee. Strongly agree

Example of Open Questions

• Can you explain me what do you do as a software architect?

• Please, could you sort for me the features in [the current prototype] that you think would make your own work more effective for the group?

Ask simple questions first

• Start the interview with the simple questions.

• In general, questions that require the informant to trust you should not be used in the beginning of the interview. Build rapport first.

• Even, if there already is rapport between you and him, ask simple questions first;

Ask short questions

• Instead of asking long and confuse questions, ask short questions: “tell me about your day, what did you do today?”

• “Grand tour” questions are questions that ask an informant to describe his reality, to provide an overview of his work– Can you explain me what do you do as a

software architect?

• Based on the informant’s answer, you can follow up on different aspects of his work.

Avoid Generalizations

• When the informant generalize something (“I am usually late in my coding activities”), ask him to be specific (“Give me an example.” “When was the last time you were late in your code activities”).– Specific cases often provide important

insights about one’s work;– You can also explore why he believes in the

generalization

Naive Questions (play dumb!)

• Over time the researcher becomes more and more familiar with the field. Despite that, he should avoid start making assumptions about what he knows about the field, he should not stop asking simple, even naive questions;– Avoid questions that confirm what you think you

know! • Everyone here uses Eclipse, right?

– Instead ask the informant to explain something again to you, so that you can confirm what you think you know.

• Which software development tools are used here?

A large part of doing good field work is that ability to make yourself vulnerable and slightly stupid. That willingness to suspend early assumptions about the way the world works, and to let other people's ways of making meaning be the ones that make sense for you. (Genevieve Bell, May 2004)

Naive Questions (3)

• If you have a CS degree or similar and is doing fieldwork in Software Engineering, the informants might make assumptions about what you already know (compilers, programming languages, etc).

• So, depending on your interest, you might even pretend to not know some of these aspects, i.e., play dumb!

Active Listening

• The researcher is primarily a listener who gains information in the terms used by the informants as the informant answers the questions.– This means, the researcher interrupts the

informant as little as possible, and only when he thinks there is an interesting aspect that is not fully developed by the informant

Active Listening (2)

• The researcher is quiet, but is active. – He is aware of what is being discussed;– He is making [mental or jot] notes about what

is said, who said it and whether that makes sense in the context of the project;

– He is demonstrating his interest to what is being said and his respect to the ideas of the informant;

Sensitive Silence

• In some cultures, just 1 or 2 seconds of silence is enough to encourage informants to continue talking without the need of the researcher to intervene;– This silent can be used as a way to let an

informant expand on a particular topic;

• Note that being quiet during an interview should not be underestimated, this in fact, is one of the more difficult skills to learn as one learns interviewing [DeWalt and DeWalt, 2002]

The Uh-hu Prompt

• “Uh-hu”, “hmmm-hmmm”, “ok”, “yes” , “really” are all prompts to active listening as a way of saying to the informant:– “I am following you”, or “This is interesting”, or

“Please go on”

Repetition Feedback

• This basically is to repeat the last word or comment by the informant to stimulate him to continue the conversation and to demonstrate that the researcher is following the conversation, is interested– Be careful to not sound like a parrot ;-)

Summary Feedback

• According to this technique, the researcher simply repeats what has been said by the informant to confirm his understanding and to encourage the informant to continue and make corrections;

Tell me more!

• “Tell me more!”. Use questions and comments that encourage the informant to continue talking about what happened, when happened, who was involved, etc:– Tell me more!– What else?– What happened?– What did you do?

• The idea here is to encourage the informant to keep talking about the same topic, instead of encourage him to change topics;

Practical Aspects

The informant’s time is precious. 1. Use wisely

– Do no plan interviews longer than one hour. People will be too tired after 45 minutes;

2. Be careful:– Always have extra time for your interviews, i.e., save

45 minutes for a 30-minute interview;– Choose a quiet place without interruptions;– Ask permission to tape the interview. If that is

acceptable, take extra batteries and tapes with you. Otherwise, take notes, jot down reminders without interrupting the informant.

• If you cannot write notes during the interviews, do it less than 24 hours afterwards. After that, details get lost;

• Write a summary of the interview:– What was interesting?– What was not interesting?– What did you learn different?– What was unexpected?– What would you do differently in the next interview?

• These summaries will allow you to find relevant information fast and reflect about the next interview.

Quality assurance• Set goals

– Know what your focus is

• Pilot study – Learn from your mistakes

• Member checking– Replay your observations – you may have it wrong

(e.g. red bar)

• Triangulation– By data collection method– By analysis approach

Practical ExerciseThis exercise will give attendees a chance to try out observation for themselves

Writing ethnography

An ethnography is a “post hoc” representation or account of what has been seen, heard and found “in the field”. Writing the ethnography is not just “writing up” the field notes. It involves their interpretation and analysis.(Dourish 1994)

Data analysis• Writing ethnography

– Narrative

• Identifying Themes– Confirming and disconfirming – Many voices– Do I have to use grounded theory?

• Grounded Theory– Building a theory – what does it look like?

• Theoretical lenses– Distributed cognition, activity theory, …?

45

Grounded Theory• Authored by Glaser and Strauss in 1967;• It does not require a prior theory about the data,

that is, a set of hypotheses to be tested. – Instead, the goal of grounded theory is

precisely to generate theory grounded exclusively on the existing data.

• It aims to develop a theory or explanation about what is going on in the field, or more specifically, what is available in the data collected.

[Glaser & Strauss, 1967] and [Strauss & Corbin, 1997]

Grounded Theory: Overview• Grounded theory is based on Coding, which is

the analysis of the data;• Field notes and transcriptions of interviews are

coded to identify concepts and categories: – A concept names a phenomenon. It abstracts an

event, object, or action/interaction that is significant to the researcher [Strauss and Corbin, 1998; p103].

– Categories are grouping concepts put together under a more abstract high order concept [Strauss and Corbin, 1998; p113].

46

47

Grounded Theory: coding

• Open coding– data is micro-analyzed (line-by-line) to identify

categories

• Axial coding– categories were broken into subcategories. Whereas

categories stand for phenomena, subcategories answer questions about the phenomenon, such as when, where, why, who, how, and with what consequences;

– Identifies relationships between categories

48

Grounded Theory: coding

• Selective coding– the most important categories are selected to be core

categories, that is, the categories that will be used to describe the emerging theory

49

Grounded Theory: Open Coding

Distributed cognition• Used for analysing collaborative work

– Ed Hutchins ship navigation and aeroplane flying– Ackerman and Halverson call centres– Adapted for use in HCI

• The people, environment & artefacts are regarded as one cognitive system

• Focuses on information propagation & transformation

• Can be used to answer ‘what if’ questions & breakdowns

Practical ExerciseThis exercise will ask attendees to return to the data used in the morning and consider coding from a different perspective

Group DiscussionThis exercise will be a chance to consider what is/may be different about applying ethnography in software engineering contexts