Multiple Choice Questions -

Item writing & item analysis

15th April 2014

Jane Holland MD PhD MRCSI

RCSI Royal College of Surgeons in Ireland Coláiste Ríoga na Máinleá in Éirinn

including slides from Richard Arnett, PhD

MCQ Item writing & item analysis - Outline

Introduction to item-writing

Response formats

Stimulus formats

Technical item flaws

Introduction to item analysis

Jane Holland

http://www.nbme.org/publications/item-writing-manual.html

MCQ response formats

• True / False

The candidate is required to make a determination

as to whether each option provided is true or false

• One Best Answer

The candidate is required to indicate a single

response i.e. the most likely diagnosis

NBME Manual p 13 – 18

Schuwirth: Med Educ. 2004 Sep;38(9):974-9.Jane Holland

True / false vs. Single Best Answer

False True

(Simple true/false items)

(Complex true false items)

Q. True statements about dogs include:

A. They were first domesticated in New Zealand.

B. 35% of households own at least one dog.

C. They have a field of vision up to 200°.

D. They communicate by barking.

Jane Holland

True / false vs. Single Best Answer

Q. Which Irish rugby union player holds the record

for most international caps?

A. D’Arcy

B. Heaslip

C. Kearney

D. O’Connell

E. O’Driscoll

Jane Holland

Q. Which of the following statements about Ireland

is true?

A. Tayto exports exceed € 1 Bn

B. The climate is sub-tropical

C. The main export is zinc

D. The population is 5 million

E. Rainy weather frequently occursX

True / false vs. Single Best Answer

Trade / exports

Weather

DemographicsD

C

B

A

E

Jane Holland

True / false vs. Single Best Answer

Q. What colour is the sky?

A. A hazy shade of winter

B. Blue

C. Grey

D. Orange

E. Red

Jane Holland

Q. Which of the following calls is a rugby union referee most

likely to make in the case of an intercepted knock-on?

A. Advantage

B. Line out

C. Penalty kick

D. Penalty try

E. Scrum

Q. Which of the following was an original recording by

Meatloaf?

A. Bat out of Hell

B. One out of three ain’t bad

C. Paradise by the Kitchen Light

D. Stand by me

E. Total Eclipse of the Heart

Jane Holland

The “cover test”

MCQs – how many options?

Jane Holland

Swanson et al; Academic Medicine. 2005;80(10):S93-S6.

Tarrant & Ware; Nurse Education Today. 2010;30(6):539-43.

Stem /

vignette

Question

(Lead-in)Options

Single Best Answer

Jane Holland

MCQ stimulus formats

Context -free

Tests factual

knowledge

Context-rich

Tests reasoning

or application

Schuwirth: Med Educ. 2004 Sep;38(9):974-9.

Kennedy, 2007Jane Holland

Creating

Evaluating

Analysing

Applying

Understanding

Remembering

NBME Manual p 35 - 49

Q. Which dermatome is at the umbilicus?

A. T6

B. T8

C. T10

D. T12

E. L2

Q. A 65-year-old librarian presents with skin lesions

suggestive of a Herpes zoster infection affecting

the dermatome which includes the skin of the

umbilicus.

Which segmental spinal nerve is associated with

that dermatome?

Jane Holland

Q. A 6 year old boy is admitted with a history of colicky

abdominal pain which has now localised to his right

iliac fossa.

The initial, visceral pain felt with appendicitis is

referred along sympathetic fibres from which level

of the spinal cord?

A. T6

B. T8

C. T10

D. T12

E. L2

NBME Manual p 35 - 49Jane Holland

Stem

Lead-in

Options

Clinical (or contextual) vignette

• Age / gender

• Presenting Problem / history

• Examination findings / test results

Which of the following is the most:

• Likely diagnosis

• Appropriate treatment

Homogenous

• One correct & 4 distractors

• Alphabetical (or logical) order

• All plausible & same length (short!)

NBME Manual p 35 - 49Jane Holland

Q. Which of the following Moodle activities

allows participants to select a single

option from a list?

A. Checklist

B. Choice

C. Forum

D. Survey

E. Workshop

Jane Holland

Q. Gavin is running a conference, at which there will

be a number of parallel workshops with limited

places. Which of the following Moodle activities

could he use to allow participants to preselect

their preferred workshop?

A. Checklist

B. Choice

C. Forum

D. Survey

E. Workshop

Jane Holland

Q. Which of the following Moodle activities

allows participants to select one choice

from a list of options?

Word repeats

Jane Holland

A. Checklist

B. Choice

C. Forum

D. Survey

E. Workshop

Grammatical

clues

Long

correct

answer

Absolutes

Word

repeats

Logical

cues

Convergence

Technical

Item flaws

NBME Manual p 19 - 27Jane Holland

Q. Which of the following Moodle activities

allows participants to select one choice

from a list of options?

Word repeats

Jane Holland

A. Checklist

B. Choice

C. Forum

D. Survey

E. Workshop

Q. It is extremely important for teachers to

maintain a healthy diet. Therefore,

students should always give them an:

A. Apple

B. Banana

C. Bread roll

D. Chocolate bar

E. Mohito

Grammatical clues

Jane Holland

a / an

plural / singular

Q. A 76 year old diabetic is brought into the accident

and emergency department with suspected

diabetic ketoacidosis. After confirming that she is

hyperglycaemic, the first step in management

should be intravenous administration of:

A. Chest X-ray

B. Electrocardiogram

C. Fasting blood glucose level

D. Insulin

E. Placement of an nasogastric tube

Grammatical clues

Jane Holland

Word repeats

Q. A 58 year old man with a history of heavy alcohol

use and previous psychiatric hospitalisation is

confused and agitated. He speaks of experiencing

the world as unreal. This symptom is called:

A. Depersonalisation

B. Derailment

C. Derealisation

D. Focal memory deficit

E. Signal anxiety

Jane Holland

Logical clues

Q. Crime is:

A. Equally distributed among the social classes

B. Overrepresented among the poor

C. Overrepresented among the middle class &

rich

D. Primarily an indication of psychosocial

maladjustment

E. Reaching a plateau of tolerability for the nation

Jane Holland

Also be wary of overlapping numericals...

Convergence Strategy

Q. Local anaesthetics are most effective in the:

A. Anionic form, acting from inside the

nerve membrane

B. Cationic form, acting from inside the

nerve membrane

C. Cationic form, acting from outside the

nerve membrane

D. Uncharged form, acting from inside the

nerve membrane

E. Uncharged form, acting from outside the

nerve membrane

Jane Holland

Long correct answer

Q. Secondary gain is:

A. Synonymous with malingering

B. A frequent problem in OCD

C. A complication of a variety of illnesses and

tends to prolong many of them

D. Never seen in organic brain damage

Jane Holland

Absolutes...

Q. Which of the following shoes are never

worn by men?

A. Boots

B. Runners

C. Sandals

D. Stilettos

E. Wellies

Jane Holland

Absolutes

Holsgrove G, Medical Education 1998; 32: 343 - 350

Jane Holland

0

10

20

30

40

50

60

0% 1% 5%

“Never”

0

10

20

30

40

50

60

80% 95% 99% 100%

“Always”

Jane Holland

Shape Long vignette & stem short options

Cover test

• Ideally can know the correct answer before seeing the

possible options

Use homogenous distracters

• i.e. all arteries, all medications, all diagnoses

• Grammatically consistent

• Logically compatible

• List in logical or alphabetical order

Don’t use vague frequency terms

Never use absolutes!

Do not write a negative question!!

NBME Manual p 35 - 49

Some useful principles

Jane Holland

Jane Holland

Difficulty Discrimination

Corrected

Point

Biserial

Distractor

Choice

Introduction to Item Analysis

Classical Test Theory

Item Response Theory (IRT)

Generalizability Theory (G-Theory)

Richard ArnettDe Champlain: Med Educ. 2010; 44:109-117.

How did the items / questions perform??

1. What proportion of candidates got the

question correct?

2. Which options were chosen most frequently?

3. Which options were chosen by low, medium

and high performers on the test as a whole?

4. How did success on an item correlate with

overall test score?

Jane Holland

Difficulty Discrimination

Corrected

Point

Biserial

Distractor

Choice

De Champlain: Med Educ. 2010; 44:109-117.

Item difficulty (DIF) – or facility

What proportion of candidates got the question correct?

90% of candidates answered correctly

DIF = 0.90

< 0.40 – hard

> 0.80 – easy

0.40 – 0.80 – good question

0.45 – 0.55 are extremely discriminating

Jane Holland

Which options were chosen most frequently?

Distractor choice

Examine the proportion of candidates choosing

each option

ALL options should be chosen by candidates

Replace any options not chosen

Jane Holland

Discriminating Index (or power)

Which options were chosen by low, medium

and high performers on the test as a whole?

Lower 27% Upper 27%Middle

L

N

U

U – L

N= DI

Ideally, DP should be > 0.40

Jane Holland

Jane Holland

Richard Arnett

Additional measures of Discrimination

Point biserial correlation (RPB)

Equivalent to Pearson correlation co-efficient

Is a measure of association between a binary (nominal)

variable and a continuous variable

Corrected point biserial correlation (CRPB)

The item is removed from the test total prior to calculation

Range = -1 to +1

>0.2 good, <0.2 weak

Negative – inverse relationship!!!!

Jane Holland

Jane Holland

Jane Holland

Jane Holland

Moodle Quiz statistics

Item Difficulty...

Jane Holland

In a quiz with random questions...

Position of items within the test may change

Therefore, analyses must distinguish between positions in

the test, and test items

Students may answer different sets of questions

i.e. Answer 30 out of 50 possible questions from the bank

Np

N

Click on the quiz, then access:

Administration

Quiz administration

Results

Statistics

Jane Holland

Jane Holland

Jane Holland

Jane Holland

Good Stats Good Questions

Good stats don’t necessarily mean good

questions and good questions don’t

always get good stats...

...but it is a good place to start

Richard Arnett