Analysing and interpreting cognitive

interview data: a qualitative approach

Presentation structure

• background• aims and objective of QT• design of cognitive interviews

• review of methods of analysis• NatCen approach• issues for discussion

Background

• aims and objectives of QT• does test question meet measurement objectives• if not, what problems arise• implications of problems for survey

• design of cognitive interviews• probe sheet• interviewing techniques• sampling strategy

Review of methods of analysis

• little written on analysis of cognitive interview data• two main methods cited:

• standardised coding scheme • qualitative analysis

• standardised coding scheme approaches are documented (usually)

• qualitative analysis descriptions are often scant on detail

Standardised coding schemes

• can be used to as stand alone question appraisal tool (e.g. QAS)

• can incorporate some elements of behaviour coding• interviewer has a problem reading the question or recording

the answer

• focus on cognitive Q & A model• comprehension/communication• recall/computation• bias/sensitivity (judgement issues)• response category • plus ‘logical issues’

Issues with standardised schemes

PROS• lend themselves to presenting data quantitatively (x Rs had this problem)• perceived (by some) to be more robust• process is replicable• useful in cross-national/ cross cultural settings, where standardisation important

CONS• need a lot of detailed codes under each main heading (particularly comprehension)• time consuming• loose context: why did R interpret Q in that way?• lend themselves to presenting data quantitatively (x Rs had this problem)

Qualitative approach

“Just naming and classify what is out there is usually not enough. We need to understand the patterns, the recurrences, the whys. As Kaplan (1964) remarks, the bedrock of inquiry is the researcher’s quest for ‘repeatable regularities’.”

Miles & Huberman (1994)

Approaches to qualitative analysis

Ethnographic accounts• detailed ‘thick’ descriptions of cultures or organisations

Life histories • analysed as individual cases or mined for common themes

Content analysis • identifies content and context of documents, often involves counting

(not strictly qual)

Grounded theory• generates analytical categories and the links between them through

an iterative process of collecting and analysing data

Approaches to qualitative analysis

Narrative analysis• examines how a story is told and the intention of the teller

Conversation analysis• examines the structure of (usually) naturally occurring

conversations

Discourse analysis• focuses on how knowledge is produced through the use of

language

Interpretative analysis • attempts to present and re-present the world of those

studied, by identifying and describing substantive themes, and searching for patterns between them

Key stages of the analytical process

Data management• identifying themes• sorting and reducing

data

Generation of findings• describing• classifying• finding linkages and

patterns• identifying explanations

Characteristics of ‘good’ analysis system

• Remains grounded in the data• Transparent data reduction process• Facilitates and displays ordering• Permits within and between case analysis

Seeking wider applications

Developing explanations

Detecting patterns of association

Establishing typologies

Identifying elements & dimensions

Summarising / synthesising data

Sorting data

Tagging data

Identifying initial concepts / themes

Primary data

Datamanagement

Descriptiveanalysis

Explanatoryanalysis

The analytical hierarchy in qualitative research

Data collection

?

Thematic analysis - purposes and principles

Structured display of data by theme (Q) across all cases

Creating categories and classifying data within them*

Demonstrating range and diversity

Using examples to illustrate and amplify

Must be comprehensive

Labelling and categorising must be valid

Carrying out thematic analysis - NatCen approach

Familiarisation with dataIdentification of factors

• highlight, summarise, provisionally label

Categorisation• is this a different manifestation of that• is this a subset of that• is this of the same order as that

Iterative process of refinementStart close to the data - become more abstract and

interpretativeMust be comprehensiveAim is analytical coherence

Looking for explanations

Informed by:• hunches and hypotheses• reflections during fieldwork and analysis• other research or theories

Process involves:• detailed within case analysis• comparison between cases• repeated interrogation of the data

milking datamoving back and forth between casessearching for rival explanations

Expect multiplicityMust be comprehensive

Summary of NatCen approach

Detailed notes made on interviews

Notes reviewed

Chart set up

Notes charted (chart revised)

Charts reviewed

Data interpreted

Findings emerge

Recommendations made

Report written

Example chart

Case details Q1Emp status

Q2Job title

Q3No. hrs wrklast wk

Q4No.employees

DC001, M, 52yrs,Self-employed

AnswerCompResponseOther issues

DC002, F, 24, pttemporaryworker & ftstudentDC003, F, 36, ftemployee

DC004, M 68,retired, self-employed (pt)

Issues to consider

• better documentation of qualitative analysis approach

• integration of code frames within thematic approach

• development of best practice for analysis of cognitive data

• cognitive interview data as one component of testing strategy

• collaborate findings using other data sources (e.g. split ballot experiments)