national rusa institute: “serving 21st century users: opportunities & challenges”

1

Reference Assessment Programs: Evaluating Current and Future

Reference Services

INTERVIEWS & FOCUS GROUPS

National RUSA Institute:

“Serving 21st Century Users: Opportunities & Challenges”

Marie L. Radford, Ph.D.Associate Professor,

School of Information & Library Science, Pratt Institute

October 12-14, 2000

Omni Inner Harbor Hotel

Baltimore, MD

Introductions

• Presenter- Marie L. Radford, Ph.D.

Associate Professor, School of Information & Library Science

Pratt Institute,

200 Willoughby Ave.

Brooklyn, NY 11205-3897

mradford@prodigy.net

(718) 636-3512 (o)

(718) 636-3733 (fax)

• Workshop Participants

AGENDA

• Interviews – Types of interviews– Questioning types– Listening effectively– Advantages & Uses– Disadvantages– Planning & Conducting Interviews

• Focus Groups – Advantages & Uses– Disadvantages– Planning & Conducting FGs

• Analysis• Reporting Results• Limitations

4

INTERVIEWS

• What is an Interview?

“An interview is a conversation involving two or more people, which is guided by a predetermined purpose.”

(Lederman, 1996, p. 5)

5

TYPES of INTERVIEWS

• Person-to-Person with Individuals

• Telephone• Focus Groups

KEY COMPONENTS

Good Questions

Good Listening Skills

Good Interpersonal Skills

7

TYPES OF QUESTIONS

• 4 valuable types for information gathering interviews– OPEN

• “What is it like when you visit the library?”

– DIRECTIVE• “What happened when you asked for

help at the reference desk?”

– REFLECTIVE• “It sounds like you had trouble with

the online catalog?”

– CLOSED• “Have I covered everything you wanted

to say?”

8

NEUTRAL, LEADING, AND LOADED QUESTIONS

• NEUTRAL– “What are your impressions of

the M. L. Radford Memorial Library?”

• LEADING– “You don’t like the librarians at

the M. L. Radford Memorial Library, do you?”

• LOADED– “How many other unreasonable

requests have you made to the reference librarians?”

Questioning Do’s & Don’ts

DO’S Be clear Be focused 1 idea per

question Use specific

questions Give sufficient

time PRETEST!

DON’TS Hypotheticals Wandering

interviewee Too many

choices Ambiguous or

abstract questions

Leading or loaded questions

10

Effective ListeningDo’s & Don’ts

DO’S• Pay attention• Listen actively• Invite talk• Open mind• Supportive

atmosphere• Show

acceptance & understanding

• Paraphrase• Note NV cues

DON’TS

• Rush to fill silences

• Give opinions

• Be defensive

• Forget to listen for feelings

• Hesitate to clarify

• Glaze over

11

In-Person Interviews: Advantages & Uses• Face-2-face interaction

• In-depth info.

• Understand experiences & meanings

• Highlight individual’s voice

• Preliminary information to “triangulate”

• Control of sampling

• Greater range of topics

12

Telephone Interviews

• 3 Types of Sampling1. All who phone are asked to

participate in a short interview immediately

2. Phoning users are asked to participate in a call-back interview

3. Random dialing of community members (samples non-users)

• Disadvantages– Not all users have phones– Nonverbal cues missing– Follow-up may be difficult

13

Interviews: Disadvantages

• Time Factors– Varies by # & depth– Transcription: 1.5 hrs. per hr.– Lots of prep. & administration

• Cost Factors– Higher the #, higher the cost– Training interviewers– Tape purchase & transcription

• Additional Factors– Self-report data– Errors in transcription or note

taking possible

14

Library Applications

• Academic– Reference

Encounter

• Public– Family

Place Project

15

ACADEMICThe Reference Encounter

(Radford, 1999)

• Interviews & observation

• 3 sites– Community College– Undergraduate College– Research University

• Qualitative methodology to capture complexity & depth

• 27 pairs of librarians/users

• Analysis– Critical Incident– Paired Perceptions– Categories

16

PUBLIC

Family Place Project (Middle Country Public Library, Centereach, NY & Libraries For the

Future, NYC) • Interviews & questionnaire

• 5 sites– 3 Urban– 1 Suburban– 1 Rural

• Qualitative methodology to evaluate change process

• Analysis– Critical Incident– Categories

17

Phase I: Planning

• Determine purpose of interviews in overall assessment

• Formulate Q’s & pretest

• Identify: Interviewees & Interviewers (& train ‘em)

• Choose notes or tape recorder

• Decide when & where

• Decide f-2-f or telephone

• Plan schedule of interviews

• Choose method of data analysis

• Identify time & private place

18

Phase II: Conducting Interviews

• Be on time & don’t waste their time• Obtain permission to use info. (report

and/or publication) & if taping• If taping check equipment & have

back-up• Create safe climate, assure

confidentiality• Be prepared, flexible, & stay on task• Listen & know when to probe• Accept that some interviews won’t go

well• Thank them!

19

Focus Group Interviews

• What is a Focus Group?

“A focus group is an in-depth, face-to-face interview of a group of 8 to 12 people representing some target group and centered on a single topic.”

(Zweizig, Johnson, Robbins, & Besant, 1996)

20

Focus Groups: Advantages & Uses

• Advantages of individual interviews plus…

• Take less time

• Creates synergy:– Comments stimulate others

– Unexpected insights

– More complete info.

– Elicit strong & common opinions

– Less inhibiting, less formal

• Access needs of under-served or underrepresented groups

• Control of sampling

21

Focus Groups: Disadvantages

• Skilled moderator required

• Participants may be too quiet or too outspoken

• Analysis, summarization & interpretation of responses may be difficult

• Loads of planning & administration time

• Transcription of tapes is time consuming & costly

22

Library Applications

• ACADEMIC– Assessment of

AV Delivery Service

– Undergraduate College

– Focus Groups of students, faculty, users and non-users of service

– Used to develop survey to evaluate and improve service

• PUBLIC– Preparation for

service excellence training

– Urban library system

– Focus Groups of staff (mixed librarians & non librarians) and users

– Used to develop survey to inform training

23

Phase I: Planning

• Determine role of focus group study in overall assessment

• Identify: – target group(s)– moderator & assistant

• Decide # of groups (3-4 groups per targeted group)

• Develop interview guide (& pretest)• Plan schedule (1.5-2 hours each), when,

& where• Choose notes or tape recorder• Choose method of data analysis• Con’t...

24

More Planning

• RECRUIT VOLUNTEERS– Identify pool of potentials

– Plan for 8-12 per group (over-recruit and remind)

– Offer reward if possible (is not $, free ILLs, photocopying, food)

– Try to get representative members

– Form homogeneous groups:

• Academic (e.g., faculty, undergrads, grad. students)

• Public (e.g., adults, teenagers, non-users)

25

Moderator Characteristics

• Excellent communication skills• Experience in group dynamics• Knows when to probe• Restrained• Ability to:

– Involve all participants– Ask neutral questions– Quickly establish rapport– Summarize areas of

agreement/disagreement– Direct but not control discussion

26

Phase II: Conducting Focus Groups• Be on time, don’t waste time

• Bring/check supplies (flip chart, markers, masking tape, etc.)

• Obtain permission to use info. (report and/or publication) & if taping

• If taping check equipment & have back-up

• Begin by creating safe climate

• More...

27

Conducting Focus Groups Continued

• Use effective listening

• Help quiet people talk, limit talkative people

• Design well constructed guide, including:– Introduction (purpose, ground

rules)– Ice breaker or warm-up set of

questions– Relevant major/probe questions– Summary or closing

• Thank them!

28

Interviews & Focus GroupsPhase III: Analysis

• Listen/transcribe tapes, review notes• Ways to analyze data:

– Code data into pre-determined categories

– Use data to identify categories

– Use data as basis for summary statements “capture the essence”

– Interpret the data- intensive analytic method

• Compare all interviews/focus groups• Compile and summarize• Look for trends or problems to study• Do not overgeneralize from results!

29

Interviews & Focus GroupsPhase IV: Reporting Results

• Parts of Report– Statement of purpose– List of issues purpose– Explanation of how data was collected

and analyzed– Summary of findings: What was said

on each issue– Sample quotes (anonymous)– Interviewer’s or moderator’s

impressions– Recommendations

• Short term (low hanging fruit)

• Long term

30

Limitations:Interviews & Focus Groups

• Difficult to generalize• May not be representative • Analyzes perceptions, not facts• Subjective analysis (reliability requires

more than 1 person)• Raw data could be misleading• Results can be limited by:

– Poor/inexperienced moderator/ interviewer

– Poorly constructed discussion guide/interview questions

• Compensate by combining methods (survey/questionnaires/ observation)

31

Summary & More Information

• Where to get more information

• List of books, articles, in packet.