chapter 7 data gathering. outline four key issues data recording interviews questionnaires...

Chapter 7

Data gathering

Outline

Four key issues Data recording Interviews Questionnaires Observation Choosing and combining techniques

Four key issues

Goal setting Relationship between data collector and data

provider Triangulation Pilot studies

Setting goals

Examples: To understand how technology fits into

normal family life To identify which of two icons representing a

‘send email’ action is easier to use To find out whether the redesign you are

planning for a hand-held meter reader is along the right lines

Setting goals

Identify specific goals for the particular study

Goals will influence the nature of the data gathering sessions, the techniques, and also the analysis

Goals should be clear and concise

Relationship with participants

Making sure that this relationship is clear and professional

Ask participants to sign an informed consent form

Incentives for participating in the data gathering sessions might be needed in circumstances that there is no clear and direct advantage to the participants

Triangulation

Using more than one data gathering / data analysis technique

Examples: Using observation to understand the context of

task performance Interviews to target specific user groups Questionnaires to reach a wider population

Provides different perspectives and corroboration of findings across techniques

Pilot studies

A small trial run of the main study Plans for a data gathering session should be

tested by doing a pilot study before launching into the main study Test the equipment and instructions Questions can be tested for clarity Experimental procedure can be confirmed as

viable Anyone involved in a pilot study cannot be

involved in the main study

Data recording

Which techniques are used will depend on the context, time available, and the sensitivity of the situation

The choice of data recording techniques will have an impact on how intrusive the data gathering will be

Data recording

Three common techniques are Notes plus still camera Audio plus still camera

Audio is less intrusive than video Video

Capture both visual and audio data but can be intrusive

Interviews

Types of interviews Open-ended or unstructured Structured Semi-structured Group interviews

Unstructured interviews

Exploratory, to explore the range of opinions No particular expectation about the format or

content of answers Example: What are the advantages of using a

PDA? Both interviewer and interviewee can steer

the interview A balance between relevant answers and

following unanticipated new lines of enquiry

Unstructured interviews

Generate rich data Interviewees may mention issues that the

interviewer has not considered Data analysis is time-consuming Impossible to replicate the process

Structured interviews

Interviewer asks predetermined questions Useful when the goals are clearly understood

and specific questions can be identified Questions are closed, meaning they require

an answer from a predetermined set of alternatives

For each participant, question wording should be exactly the same and questions are asked in the same order

Structured interviews

Examples: Which of the following websites do you visit most

frequently: amazon.com, google.com, msn.com? How often do you visit this website: everyday,

once a week, once a month, less often than once a month?

Have you ever purchased anything online? If so, how often do you purchase items online:

every day, once a week, once a month, less often than once a month?

Semi-structured interviews

Use both closed and open questions For consistency, a basic script is used for

guidance so that the same topics are covered with each interviewee

The interviewer starts with preplanned questions and then probes the interviewee to say more until no new relevant information is forthcoming

Semi-structured interviews

Example: Which music websites do you visit most

frequently? (hottestmusic.com) Why? (site layout) Tell me more about the site layout Anything else that you like about the site? Thanks. Are there any other reasons for visiting

this site so often that you haven’t mentioned?

Focus groups

Interview people in groups Normally, 3 to 10 people are involved, and

the discussion is led by a trained facilitator In requirements activities, to identify conflicts

in terminology or expectations from different sections within one organization

The method assumes that individuals develop opinions within a social context by talking with others

Planning and conducting an interview

Developing questions or topics to be covered Collating any documentation to give to the

interviewee Checking that recording equipment works in

advance and you know how to use it Working out the structure of the interview Organizing a suitable time and place

Developing interview questions

Two types:− ‘closed questions’ have a predetermined

answer format, e.g., ‘yes’ or ‘no’− ‘open questions’ do not have a

predetermined format

Closed questions are easier to analyze

From: www.id-book.com

Developing interview questions

Split compound sentences into separate questions

Avoid using technical terms or complex language

Try to keep questions neutral Example: “Why do you like this style of

interaction? -> assumes the person does like it and will discourage some interviewees from stating their real feelings

Running the interview

Introduction interviewer introduces himself explains why he is doing the interview reassures interviewees regarding any ethical

issues asks if they mind being recorded, if appropriate


Warm-up session starts with easy, non-threatening questions, such as questions about demographic information

Main session in which the questions are presented in a logical sequence, with the more probing ones at the end


Cool-off period consists of a few easy questions to ease tension if it has arisen

Closing session in which the interviewer thanks the interviewee and switches off the recorder or puts her notebook away

Other forms of interview

Telephone interviews

Online interviews asynchronous communication, e.g. email synchronous communication, e.g. IM

Enriching the interview experience

Interviews often take place in a neutral environment, e.g. a meeting room away from the interviewee’s normal desk

Interviews can be enriched by using props such as prototypes or work artifacts that the interviewee or interviewer brings along

Questionnaires

Questions can be closed or open Closed questions are easier to analyze, and

may be done by computer Can be administered to large populations Paper, email and the web used for

dissemination Sampling can be a problem when the size of

a population is unknown as is common online


Questionnaires

Start by asking for basic demographic information

Question order Different versions of the questionnaire for

different populations Clear instructions on how to complete the

questionnaire

Questionnaires

A balance between using white space and the need to keep the questionnaire as compact as possible

Decide on whether phrases will all be positive, all negative or mixed.

Question and response format

Check boxes and ranges

Rating scales Likert scales Semantic differential scales

Open-ended responses

Likert scales

1. Gather a pool of short statements about the subject to be investigated. For example, “This control panel is easy to use”

2. Decide on the scale How many points does the scale need? Should the scale be discrete or continuous? How to represent the scale?

3. Select items for the final questionnaire and reword as necessary to make them clear

Semantic differential scales

Used less frequently than Likert scales Explore a range of bipolar attitudes about a

particular item Each pair of attitudes is represented as a pair

of adjectives For example:

Attractive _ _ _ _ _ _ _ Ugly

Clear _ _ _ _ _ _ _ Confusing

What scales to use – 3, 5, 7 or more?

Use a small number, e.g. 3, when possibilities are very limited, as in yes/no type answers

Use a medium-sized range, e.g. 5, when making judgments that involve like/dislike, agree/disagree statements

Use a longer range, e.g. 7 or 9, when asking respondents to make subtle judgments. For example, when asking about a ‘level of appeal’ of a character in a video game

What order?

Place the positive end of the scale first and the negative end last. This matches the logical way people think about scoring. For example:__ strongly agree

__ slightly agree

__ agree

__ slightly disagree

__ strongly disagree

Encouraging a good response

Make sure purpose of study is clear Promise anonymity Ensure questionnaire is well designed Offer a short version for those who do not have time

to complete a long questionnaire If mailed, include a stamped addressed envelope Follow-up with emails, phone calls, letters Provide an incentive 40% response rate is high, 20% is often acceptable


Online questionnaires

1. Devise the questionnaire as if it is to be delivered on paper first

2. Develop strategies for reaching the target population

3. Produce an error-free interactive electronic version

4. Make the questionnaire accessible from all common browsers and readable from different sized monitors and different network locations

Online questionnaires

5. Make sure information identifying each respondent can be captured and stored confidentially because the same person may submit several completed surveys

6. Thoroughly pilot test the questionnaire

Advantages of online questionnaires

Responses are usually received quickly No copying and postage costs Data can be collected in database for

analysis Time required for data analysis is reduced Errors can be corrected easily


Problems with online questionnaires

Sampling is problematic if population size is unknown

Preventing individuals from responding more than once

Individuals have also been known to change questions in email questionnaires


Observation

Direct observation in the field Direct observation in controlled environments Indirect observation: tracking users’ activities

Direct observation in the field

There is a careful balance between being guided by goals and being open to modifying, shaping, or refocusing the study as you learn about the situation

Being able to keep this balance is a skill that develops with experience

Structuring frameworks for observation in the field

Easy framework The person – who is using the technology at any

particular time? The place – where are they using it? The thing – what are they doing with it?


More detailed framework – pay greater attention to the context of the activity (Colin Robson, 2002) Space – what is the physical space like and how

is it laid out? Actors – what are the names and relevant details

of the people involved? Activities – what are the actors doing and why?


Objects – what physical objects are present, such as furniture?

Acts – what are specific individual actions? Events – is what you observe part of a special

event? Time – what is the sequence of events? Goals – what are the actors trying to accomplish? Feelings – what is the mood of the group and of

individuals?


Specific framework to support field studies in conjunction with designing future technologies (Rogers and Bellotti, 1997)

Two parts: Problematizing existing settings Envisioning future settings


Problematizing existing settings Why is an observation about a work practice or

other activity striking? What are the pros and cons of the existing ways

technologies are used in a setting? How have ‘workarounds’ evolved and how

effective are they? Why do certain old-fashioned practices, using

seemingly antiquated technologies persist, despite more advanced technologies being available in the setting?


Envisioning future settings What would be gained and lost through changing

current ways of working or carrying out an activity by introducing new kinds of technological support?

What might be the knock-on effects for other practices and activities through introducing new technologies?

How might other settings be enhanced and disrupted through deploying the same kinds of future technologies?

Degree of participation

Insider = Participant observer Attempt to become a full member of the group he

is studying Important to keep the two roles clear and

separate Observation notes are objective, while

participation is also maintained Outsider = Passive observer

Not take any part in the study environment at all

Planning and conducting an observation in the field

Choosing a framework Level of participation How to record the data How to gain acceptance in the group being

studied


How to handle sensitive issues such as cultural differences

How to ensure that the study uses different perspectives (people, activities, job roles, etc.) Work as a team

Each person can focus on different people or parts Observation and reflection can be easily interweaved Observations can be compared -> more reliable data Results will reflect different perspectives


Everyone in the group needs to be attended to, not just those who receive you well

Need to refocus the study as you reflect upon what has been seen

Ideas will sharpen into questions that guide further observation


At the end of each day You must check the notes and other records You must write up experiences and observations Writing a diary or private blog is one way of

keeping up Annotate any collected or copied documents,

describing how they are used and at what stage of the activity


As notes are reviewed, personal opinion should be separated from observation of what happened

Anything for further investigation should be clearly marked

Check observations with an informant or members of the group to ensure that you have understood what is happening and that your interpretations are accurate

Ethnography

A large part of most ethnographic studies is direct observation, but interviews, questionnaires, and studying artifacts used in the activities also feature in many studies

It aims to observe a situation without imposing any a priori structure or framework upon it, and to view everything as ‘strange’

Ethnography

The observer adopts a participant observer (i.e. insider) role as much as possible

Ethnographic data: the observer gathers what is available, what is ‘ordinary,’ what it is that people do, say, how they work

Ethnography

Initially, time should be spent getting to know the people in the work place and bonding with them

Explain to them why you are there, what you hope to achieve, and how long you plan to be there

Ethnography

The observer needs to be prepared to switch in and out of people’s work cycles, moving into the shadow if something happens that needs the worker’s immediate attention

A good tactic is to explain to one of the participants what you think is happening and then let him/her correct you

Ethnography

It is important not to appear overly keen or obtrusive

Asking too many questions, taking picture of everything, showing off your knowledge, and getting in people’s way can be very off-putting

Ethnography

Listening and watching while sitting on the sidelines and occasionally asking questions is a much better approach

When you have gained the trust and respect of the participants you can then ask if they mind you setting up a video camera, taking pictures, or using a recorder

Ethnography

List of materials that might be recorded or collected

Activity or job descriptions Rules and procedures said to govern particular

activities Descriptions of activities observed Recordings of the talk taking place between

parties involved in observed activities

Ethnography

Informal interviews with participants explaining the detail of observed activities

Diagrams of the physical layout, including the position of artifacts

Photographs of artifacts (documents, diagrams, forms, computers, etc.) used in the course of observed activities

Ethnography

Videos of artifacts as used in the course of observed activities

Description of artifacts used in the course of observed activities

Workflow diagrams showing the sequential order of tasks involved in observed activities

Process maps showing connections between activities

Applied ethnographic research

Ann Rose et al. developed a procedure to be used by technical design teams with limited ethnographic training

This applied form of ethnography acknowledges the comparatively small amounts of time available for any kind of user study


4 stages:

1) Preparation Understand organization policies and work culture Familiarize yourself with the system and its

history Set initial goals and prepare questions Gain access and permission to observe and

interview


2) Field study Establish a rapport with managers and users Observe and interview users in their workplace

and collect data Follow any leads that emerge from the visits Record your visits


3) Analysis Compile the collected data in numerical, textual,

and multimedia databases Quantify data and compile statistics Reduce and interpret the data Refine the goals and processes used


4) Reporting Consider multiple audiences and goals Prepare a report and present the findings

Direct observation in controlled environment

Prepare a script to guide how the participants will be greeted, be told about the goals of the study, and how long it will last, and have their rights explained

In the laboratory, the emphasis is on the details of what individuals do

In the field, the context is important and the focus is on how people interact with each other, the technology, and their environment

Direct observation in controlled environment

Think-aloud technique A useful way of understanding what is going on in

a person’s head

Requires people to say out loud everything that they are thinking and trying to do, so that their thought processes are externalized

Indirect observation: tracking users’ activities

Techniques: Diaries Interaction logs


Diaries Participants are asked to write a diary of their

activities on a regular basis, e.g. what they did, when they did it, what they found hard or easy, and what their reactions were to the situation


Advantages of diaries Diaries are useful when participants are scattered

and unreachable in person Not take up much resource No special equipment or expertise Suitable for long-term studies Online data collection, enabling the data to go

straight into a database


Disadvantages of diaries Rely on participants being reliable and

remembering to complete diaries Participants’ memories of events are often

exaggerated

Incentives may be needed The process has to be straightforward and

quick


Interaction logs Involves instrumenting the software to record

users’ activity in a log that can be examined later


Advantages it is unobtrusive, but it also raises ethical

concerns about observing participants without their knowledge

Large volume of data can be logged automatically. However, powerful tools are needed to explore and analyze this data quantitatively and qualitatively

Spyware

Choosing and combining techniques

Depends on The focus of the study The participants involved The nature of the technique The resources available

Summary Three main data gathering methods: interviews,

questionnaires, observation Four key issues of data gathering: goals, triangulation,

participant relationship, pilot Interviews may be structured, semi-structured or

unstructured Questionnaires may be on paper, online or telephone Observation may be direct or indirect, in the field or in

controlled setting Techniques can be combined depending on study focus,

participants, nature of technique and available resources


chapter 7 data gathering. outline four key issues data recording interviews questionnaires...

Documents

data collector

audio data

specific questions

main studyplans

main studytest

pilot study

particular studygoals

considereddata analysis