evaluation techniques of teaching: focus groups and nominal group technique

Evaluation techniques of teaching:

focus groups and the Nominal Group Technique

Tünde Varga-Atkins, eLearning UnitUniversity of Liverpool

15 November 2012SEDA Conference

Aston Business School, Birmingham

OUTLINE

• Introduction•Demonstration of the Nominal Group Technique• Benefits and disadvantages of NGT• A new, combined two-staged approach: Nominal Focus Group• Reflections in own context

OUTCOMES

• Familiar with the NGT and its stages.• Aware of the benefits and potential challenges of NGT.• Contrast focus groups & NGT.• Consider a combined approach of FG&NGT.• Reflect on the evaluation technique in own context.

CONTEXT

• Curriculum development• Student engagement• Our research-informed

experiences as educational developers for last 4-5 years.

• Hopefully a useful evaluation technique.

• Works in other contexts: staff, or any setting requiring group-decision making.

• Method selected to suit purpose!

LISTENING TO THE EXPERIENCE?

Wood-peckers? Wolves? Dinosaurs?….

“with the questionnaire, you never know if [you] give the right question out.”(staff)

“I found it extremely helpful to have not just an idea what is going on, but to hear what is said by students.”(staff)

survey face-to-face groups

LISTENING TO THE EXPERIENCE?

Wood-peckers? Wolves? Dinosaurs?….

Please rate how useful was the wood-pecker’s song in relation to your journey: 10-Very useful. 0-Not useful at all.

Gary Robson - Flickr

survey face-to-face groups

PRIOR EXPERIENCES WITH NGT?

quantitative qualitative

at-a-distance

face-to-face

surveys

focus groupsNominal groups

Delphi technique

interviews

WHAT ISNOMINAL GROUP TECHNIQUE?

•Structured group activity •One given topic•Facilitates groupdecision-making• Immediate results•Quantitative element•Reduces researcher/participant bias Delbecq & Van de Ven (1971)

NGT: STAGES

1) Individual responses

2) Clarification and consolidation

3) Ranking responses

Example question: ‘What would you change in your course?’

Duration = 1-1.5 hours‘Nominally’ group < individual

AND NOW: LET’S HAVE A GO...THE QUESTION IS:

In your current role as educational developer

what is one key challenging issue you are facing?

[purpose: identifying top 3 key ones to tackle together][normally we would ask participants to write 2-3 – but

shortening the task here due to time constraints]

STAGE 1: CHALLENGING ISSUE(S)

.1 2 3

45 6

STAGE 2 CONSOLIDATION: CHALLENGING ISSUES

35

24

.

6

1

Same answers

Same answers

Same answers

STAGE 3 RANKING: YOUR TOP 3 KEY CHALLENGES

Item no.

Item description

3 points 1

2 points 2

1 points 4

[normally top 5 but for brevity only doing 3]

that you want the group/SEDA etc. to tackle…

THE TOP FIVE KEY CHALLENGES AS EDDEV-ER:

1

2

3

4

5

1

2

3

4

5

SUMMARY & BRIEF QUESTIONS

•Nominal Group Technique• 3 stages• 1-2 key questions explored.• Focus on individual work.• Group consensus.• Quantitative outcome.• Scalable: results from more groups can

be integrated.

BENEFITS AND CHALLENGES

• Paperchase exercise

• 3 minutes for each flipchart:

NGT vs focus groups

dis/advantages

of NGT

NGT vs surveys

GUIDE TO NGT & PROJECT REPORT

See References

On slideshare.net , search for Nominal Group Technique

Context NGT is more useful for: NGT is less useful for:

Research purpose

Evaluation and decision-making

Researching general learner experiences

Topic focus When you have one single topic to explore

When you have more topics or a complex topic to explore

Likely research questions

“What changes would you make to your programme/curriculum?”“What would help you improve the quality of feedback on this course?”

“What are your experiences with your programme so far?”“What are your experiences with the quality of feedback on this course?”

Participants

Participants with different power relations within the same group; when consulting various stakeholders groups within same research (e.g. from students through to experts).

If power relations are not an issue in the group.

A COMBINED, TWO-STAGED APPROACH: ‘NOMINAL FOCUS GROUP’

Why?What is it?

Jaye McIsaac, Educational Development, University of LiverpoolVideo at: https://stream.liv.ac.uk/mntbvv9d

https://stream.liv.ac.uk/mntbvv9d

A COMBINED, TWO-STAGED APPROACH: ‘NOMINAL FOCUS GROUP’

Stage 1: focus group

Stage 2: nominal, ranking ‘bit’

STUDENT EVALUATION ON EVALUATION (SURVEY, N=13)

This is the Worlde cloud of the student survey responses on the group process.

“felt you could be honest and discuss openly about

your opinions”

SURVEY: EFFECT OF SESSION & F2F

“All of the students agreed and appeared to be facing very similar issues to myself in terms of feedback”

“Was nice to see if the University cared about the

problems we are all having.”

Visual attribution of responses is for illustrative purposes only

(survey was anonymous)

“Discussing made me remember problems in

previous years.”

VIEWS ON STAGE 1: FOCUS GROUP

“good to hear other students’ opinions to help expand my own”

“can speak much more info than writing down on

a post-it.”



“able to agree/disagree with other people’s

experiences and share your own to enforce or refute their opinion.”

“[gave] more time to think about answers.”

VIEWS ON STAGE 2: ‘NOMINAL’ BIT

“if individuals didn’t contribute much in the open discussion, their views were still taken in to account [in stage 2]”

“outlined the main problems with feedback

and made it clear what is needed to improve.”



“The open discussion helped to get me thinking of my own experience of feedback, whilst writing

my opinion on the post-it note helped get my

opinion across..”

SURVEY: VIEWS ON COMBINATION

“The open discussion allowed many ideas to be put forward, where as the

second part of the session allowed a

summary of all of the views that were

discussed.”



STAFF PERCEPTIONS ON THE TWO-STAGED ‘NOMINAL FOCUS GROUP’

“Yes the nominal group technique, in the end it brought everything together into a sharper point again. … It kind of made it easier for us to identify what the students thought was the most important thing...”

STAFF PERCEPTION ON THE‘NOMINAL FOCUS GROUP’

“dealing with basically a bullet point, … you might get the meaning wrong. You might not understand, really what they meant. Whereas [the Focus Group the citations from students] explained a bit more of what they meant. ”

SUMMARY: FOCUS GROUP ONLY

Volume of issues?Overall feelings?

Issue bias?Participant bias?

Group energisesGroup helps to formulate ideas and feelings‘others feel the same!’, reassurance

SUMMARY: NOMINAL GROUP TECHNIQUE ONLY

If more questions?If experiences?

Students warmed up?

Immediate resultsQuantitative ranking indicates volume of issuesOverall impression

SUMMARY: ADVANTAGES OF 2-STAGED NOMINAL FOCUS GROUP

Immediate resultsQuantitative ranking indicates volume of issuesOverall impression

Group energisesGroup helps to formulate ideas and feelings‘Others feel the same!’

Stage 1: FG Stage 2: NGT

SMALL-GROUP DISCUSSION

• What is the relevance of these techniques (NFG, NGT, FG) in your context?

• What methods and techniques does your institution / department use for evaluation of teaching?

• What works well?• What are the challenges? Opportunities? • Any relevance of these methods? • Or considerations for using these methods?

KEY MESSAGE

We have found the two-staged ‘Nominal Focus

Group’ to be an efficient and useful method for

evaluation of teaching & curriculum development.[It is a technique to add to your repertoire of evaluation methods. It may not suit all contexts, and the full evaluation cycle is the most important including a feedback loop and

action on results! ]

[OUTCOMES] YOU:

• Are now familiar with the NGT and its stages.•Discussed benefits and potential challenges of NGT.• Contrasted focus groups & NGT.• Considered a combined approach of FG&NGT.• Reflected on methods in own context.

THANK YOU

Contact: Tünde Varga-Atkins [email protected]

#tundeva #elearninglpool http://liverpool.academia.edu/T%C3%BCndeVargaAtkins

Today’s resources on http://slideshare.net(search for nominal group technique)

REFERENCES

• Delbecq, A., Van de Ven, Andrew, & Gustafson, D. (1975). Group techniques for program planning : a guide to nominal group and Delphi processes. Glenview Ill.: Scott Foresman.• Further references in: • Varga-Atkins, McIsaac et al (2011) Using the

nominal group technique with clickers to research student experiences of e-learning: a project report [http://slidesha.re/xQlBCg ]• Varga-Atkins, McIsaac et al (2011) The Nominal

Group Technique – a practical guide for facilitators [http://slidesha.re/AmYOgv]

HANDOUTS: STAGE 1 INDIVIDUAL RESPONSE

Participants Facilitator

Participants enter their individual response on a post-it note.

Ensures everyone works on their own and writes clearly and legibly.

Post-its are pinned on a flipchart.

Facilitator helps pin up responses and numbers each response so that they can be referred to later.

Participants read out their own response.

Facilitator , if needed, asks for a brief clarification on the item. The items are NOT discussed in detail in this stage.

HANDOUTS: STAGE 2 CONSOLIDATION

This is the longest stage.


Participants (Ps) find similar/same items.

Facilitator prompts Ps to find similar items. Facilitator asks Ps to work together on merging items if they are the same.

Ps discuss and agree on the merging of similar items (group consensus).

Facilitator adds newly formed/merged items as Ps discuss these. (and making sure items are not themed, but only similar items are

Participants do this until all items have been grouped if relevant.

Facilitator makes sure each item is numbered.

HANDOUTS: STAGE 3 RANKING


Participants are asked to choose their top 3 (normally 5) most important responses to them. The order of importance is important.

Faciltiator hands out ranking sheet and explains the ranking.

Participants rank the items on their ranking sheet. (or on flipchart is also possible.) 3 points go to the most important one, 2 points to the second most important and 1 point to the third most imp.

Participants hand in their ranking sheet.

Facilitator calculates ranking score.

RANKING: WHAT ARE THE 3 MOST IMPORTANT ITEMS TO YOU?

Item no.

Item description

3 points

2 points

1 points