introduction to dundee's mmi process

1

The Role of Multiple Mini Interviews (MMI) in selection

Dr Jon Dowell, Admissions Convenor, Dundee Medical School

OverviewRationale behind MMI for selectionA brief historyWhat do MMIs assessHow do MMIs workPerformance in UK contextFeedback from applicants and assessorsThe future?

Primus

Secundus

Tertious

Alexander Monros1720 - 1846

Medicine has ‘previous’

"I dislike him & his lectures so much that I cannot speak with decency about them. He is so dirty in person & actions."

RationalePersonal qualities considered ‘important’ but not reliably

assessed by interview and without much predictive validity. (Goho + Blackman 2006. cognitive r 0.06, clinical r 0.17)

Introduced for medicine at McMaster, Canada (Reiter, Eva et al).Piloted 2002 – 12 x 8 min. Generalisability since +/- 0.7 Predicted OSCE performance ß 0.44 (interview ns)Clerkship ratings ß 0.57 (interview ns)MCC Part 1 (selected components) ß 0.3-0.4 (interview ns)

Increasing detail emergingA cost efficiency comparison between the multiple mini-interview and traditional

admissions interviews. Set up costs, running equates

Rosenfeld JM, Reiter HI, Trinh K, Eva KW. Adv Health Sci Educ Theory Pract. 2008 Mar;13(1):43-58.

The multiple mini-interview: how long is long enough? 10 – 8 – 5 minsDodson M, et al. Med Educ. 2009 Feb;43(2):168-74

Factors affecting the utility of the multiple mini-interview in selecting candidates for graduate-entry medical school. 8Q reliability 0.7

Roberts et al. Med Educ. 2008 Apr;42(4):396-404 applicant 22% variance

Should candidate scores be adjusted for interviewer stringency or leniency in the multiple mini-interview? Rasch Modeling

Roberts C, Rothnie I, Zoanetti N, Crossley J. Med Educ. 2010 Jul;44(7):690-8

Increasingly widespread

Countries• Canada,• Australia• UK• US

Disciplines• UG Medicine• UG Veterinary• UG Dentistry

MMI in Dundee 7 minute x 10 stations

Reliability– 2008/9 n 450 0.7– 2009/10 n 500 0.69– 2010/11 n 600 0.69/.67 (0.88)

– 2011/12 Dental school collaboration

MMI in St Georges 5 minute stations x 8 Stations

Muriel Shannon +Aileen O’Brien

Reliability– 2009/10 n 1078 0.69– 2010/11 n 1333

MMI in Belfast 5 minute x 9 stations

Keith Steele Reliability

– 2010/11 n 500 0.56

MMIs typically aim to assess

– Interpersonal and communication skills (inc empathy)– Teamwork and leadership– Preparation and motivation– Critical thinking, problem solving– Ethical/moral reasoning– ‘Integrity’

How?

PracticalitiesCapacity (n = 600)

20 applicants per ‘Run’ per sessionTwo teams in parallel (80/day)8 days total

Staffing (2 academic leaders + 2 admin)For each run

5 faculty (interview)5 student or Simulated Patients (interactive)2 ‘actors’ (employed junior students)3 ‘role players’ (students or SP)

Preparation = big jobStations - Drafted and piloted with students• Assessor instructions• Candidate instructions• Resources (script, shapes, materials, UCAS forms etc)• Actor training

Training at start of every session 30 min general 20 min station specific

Station/Assessor/Candidate Information

Domain Scores

Overall Judgement

Red Flag & Narrative

Performance of Dundee MMI

Score Distribution

Station Reliability

Overall Alpha 0.69Item-Total Correl

Alpha if Item

Deleted1 0.34 0.672 0.31 0.673 0.17 0.704 0.37 0.665 0.43 0.656 0.47 0.647 0.34 0.678 0.28 0.679 0.38 0.6610 0.39 0.66

Domain Reliability

Number of Stations

Number of Scores

Cronbach’s alpha

Communication 9 12 .767Critical Thinking 6 6 .453Moral Reasoning 4 4 .170Prep +Motivation 3 4 .515Teamwork 2 3 .395Integrity 1 1 -

Gender difference - YesInteractive One - one

Females performed significantly better than males. 2009 Female average 109 /150 vs 105 p = < .01 Effect size: 0.14

Applicant Group difference?

2009 – 2010 2010 - 2011

Apologies but we are intending to eventually publish from this data

so have removed it from circulated slides.

Assessors Survey 2008-9

116 /201 (58%) assessors completed online survey.• 91% thought the process fair.• 88% thought applicant stress moderate or less. • All stations rated highly relevant, except for the

‘Question of Ethics’ where opinions divided.

Candidate Survey 2008-9

• 324 /433 (75%) completed online survey.• 94% felt MMI was ‘fair’.• 90% felt it is a ‘valid way to assess candidates’.• 71% preferred MMI to traditional interview.• ‘Student Counselor’ (role play) most enjoyable

We are happybut where next?

Hawk-Dove Effect

1 2 3 4 5 6 7 8 9 10

STATIONS

A

E

B

D

C

P. H. Harasym, Ph.D.

Dundee 2011 Rasch vertical ruler

MFRM creates a common scale for candidate ability, examiner stringency, item difficulty and ......Comparing raw and ‘fair’ scores for 350 offers 34 (9.7%) different candidates selected.

Conclusions and Next StepsMMI appear very promising:

• Set up ‘costs’ considerable• Running costs comparable (if students used)• Student involvement positive• Applicants are positive• Predictive validity emerging and encouraging.

Outstanding issues:• Rotating content.• Application of IRT (DIFF, Fair scores etc)• Oh yes – and the bloopers are great.

Thank you

Discussion?