![Page 1: IMPLEMENTING MULTIPLE MINI-INTERVIEW (MMI) AT YOUR ...EDUCATIONAL OBJECTIVES Cite two reasons for implementing Multiple Mini-Interview (MMI) in residency Cite applicant interviewing](https://reader030.vdocuments.site/reader030/viewer/2022040918/5e92e695c114f0789b1ff7df/html5/thumbnails/1.jpg)
IMPLEMENTING MULTIPLE MINI-INTERVIEW (MMI) AT
YOUR RESIDENCY PROGRAM
Saturday, April 28, 2018
Ryan P. Zimmerman, DO
Tower Health System: Reading Hospital
In Collaboration with:
Anthony A. Donato Jr., MD MHPE
Benjamin J. Lloyd, MD FACP
![Page 2: IMPLEMENTING MULTIPLE MINI-INTERVIEW (MMI) AT YOUR ...EDUCATIONAL OBJECTIVES Cite two reasons for implementing Multiple Mini-Interview (MMI) in residency Cite applicant interviewing](https://reader030.vdocuments.site/reader030/viewer/2022040918/5e92e695c114f0789b1ff7df/html5/thumbnails/2.jpg)
EDUCATIONAL OBJECTIVES
Cite two reasons for implementing Multiple Mini-Interview (MMI) in residency applicant interviewing.Cite
Produce a list of desired, non-cognitive traits of applicants to your GME program.Produce
List the structural elements and resources that are needed to utilize MMI GME programs (reference lecture materials).List
![Page 3: IMPLEMENTING MULTIPLE MINI-INTERVIEW (MMI) AT YOUR ...EDUCATIONAL OBJECTIVES Cite two reasons for implementing Multiple Mini-Interview (MMI) in residency Cite applicant interviewing](https://reader030.vdocuments.site/reader030/viewer/2022040918/5e92e695c114f0789b1ff7df/html5/thumbnails/3.jpg)
Text:
RyanZimmerma925To:
22333
Who is ready for recruitment
How do you interview
![Page 4: IMPLEMENTING MULTIPLE MINI-INTERVIEW (MMI) AT YOUR ...EDUCATIONAL OBJECTIVES Cite two reasons for implementing Multiple Mini-Interview (MMI) in residency Cite applicant interviewing](https://reader030.vdocuments.site/reader030/viewer/2022040918/5e92e695c114f0789b1ff7df/html5/thumbnails/4.jpg)
Interviews at Reading,
circa 2010
Applicant selected
for interview,
reports to us for
interview day
Two from full-time
faculty would
perform
unstructured
interview
Tremendous variety
of viewpoints,
experience,
observational skill
and emphasis…
Produced results
that were difficult to
interpret, and even
more difficult to
combine
![Page 5: IMPLEMENTING MULTIPLE MINI-INTERVIEW (MMI) AT YOUR ...EDUCATIONAL OBJECTIVES Cite two reasons for implementing Multiple Mini-Interview (MMI) in residency Cite applicant interviewing](https://reader030.vdocuments.site/reader030/viewer/2022040918/5e92e695c114f0789b1ff7df/html5/thumbnails/5.jpg)
Kevin Eva
McMaster Univ
2004
Reliability of a standard interview: r=0.1
Much too low for high-stakes decisions!
…yet 99% of programs do them.
Adding structure (same questions,
format) adds a little reliability: r=0.3
…yet still too low for high stakes.
r=0.123456>0.7
Adding more (structured) independent
observers, however, made all of the difference, and
got the reliability- > 0.7- necessary for high-stakes
decisions.
![Page 6: IMPLEMENTING MULTIPLE MINI-INTERVIEW (MMI) AT YOUR ...EDUCATIONAL OBJECTIVES Cite two reasons for implementing Multiple Mini-Interview (MMI) in residency Cite applicant interviewing](https://reader030.vdocuments.site/reader030/viewer/2022040918/5e92e695c114f0789b1ff7df/html5/thumbnails/6.jpg)
Faculty say:
• Outperforms traditional
interview (74%)
• More enjoyable (4.8/6 Likert)
• Independent of transcripts;
focus on non-cognitive skills
• ?Is there interviewer fatigue
IS IT ACCEPTABLE?
Students say:
• >90% “very acceptable”
• Prefer over standard
• More “fair”
• “Show off strengths”
• ?Does it favor extroverts
Bell K, et al. Med Teach 2007; 29:394-6.
Harris and Owen, Med Educ 2007; 41:234-41.
Humphrey S, et al. Med Educ 2008; 42:207-13.
Kumar, et al. Medical Education 2009; 43:360-7.
![Page 7: IMPLEMENTING MULTIPLE MINI-INTERVIEW (MMI) AT YOUR ...EDUCATIONAL OBJECTIVES Cite two reasons for implementing Multiple Mini-Interview (MMI) in residency Cite applicant interviewing](https://reader030.vdocuments.site/reader030/viewer/2022040918/5e92e695c114f0789b1ff7df/html5/thumbnails/7.jpg)
Interviews at Reading,
circa 2017
Applicant selected
for interview,
reports to us for
interview day
Six from full-time
faculty would
perform 8-min
structured MMI
interview
…and then we get together to
holistically reassemble the whole
from the parts each of us saw
![Page 8: IMPLEMENTING MULTIPLE MINI-INTERVIEW (MMI) AT YOUR ...EDUCATIONAL OBJECTIVES Cite two reasons for implementing Multiple Mini-Interview (MMI) in residency Cite applicant interviewing](https://reader030.vdocuments.site/reader030/viewer/2022040918/5e92e695c114f0789b1ff7df/html5/thumbnails/8.jpg)
• In my opinion, YES
• 30% of composite score to
rank candidates
• 2017-18 “Red Flag”
identification 9/~265 across
3 programs
• 1300+ observations
IS IT VALUABLE?
![Page 9: IMPLEMENTING MULTIPLE MINI-INTERVIEW (MMI) AT YOUR ...EDUCATIONAL OBJECTIVES Cite two reasons for implementing Multiple Mini-Interview (MMI) in residency Cite applicant interviewing](https://reader030.vdocuments.site/reader030/viewer/2022040918/5e92e695c114f0789b1ff7df/html5/thumbnails/9.jpg)
Text:
RyanZimmerma925To:
22333
MMI barriers
![Page 10: IMPLEMENTING MULTIPLE MINI-INTERVIEW (MMI) AT YOUR ...EDUCATIONAL OBJECTIVES Cite two reasons for implementing Multiple Mini-Interview (MMI) in residency Cite applicant interviewing](https://reader030.vdocuments.site/reader030/viewer/2022040918/5e92e695c114f0789b1ff7df/html5/thumbnails/10.jpg)
Step 1: Identify Your Ideal Doctors
What are the non-cognitive traits you and your staff want to identify?
![Page 11: IMPLEMENTING MULTIPLE MINI-INTERVIEW (MMI) AT YOUR ...EDUCATIONAL OBJECTIVES Cite two reasons for implementing Multiple Mini-Interview (MMI) in residency Cite applicant interviewing](https://reader030.vdocuments.site/reader030/viewer/2022040918/5e92e695c114f0789b1ff7df/html5/thumbnails/11.jpg)
WHO DO YOU WANT
IN YOUR GME
PROGRAM?
•Non-cognitive traits
of your ideal resident?
•How do we identify
these traits?
Word cloud for non-cognitive traits
![Page 12: IMPLEMENTING MULTIPLE MINI-INTERVIEW (MMI) AT YOUR ...EDUCATIONAL OBJECTIVES Cite two reasons for implementing Multiple Mini-Interview (MMI) in residency Cite applicant interviewing](https://reader030.vdocuments.site/reader030/viewer/2022040918/5e92e695c114f0789b1ff7df/html5/thumbnails/12.jpg)
Text:
RyanZimmerma925To:
22333
Word cloud for non-cognitive traits
![Page 13: IMPLEMENTING MULTIPLE MINI-INTERVIEW (MMI) AT YOUR ...EDUCATIONAL OBJECTIVES Cite two reasons for implementing Multiple Mini-Interview (MMI) in residency Cite applicant interviewing](https://reader030.vdocuments.site/reader030/viewer/2022040918/5e92e695c114f0789b1ff7df/html5/thumbnails/13.jpg)
SELECT DESIRED TRAIT(S)
Mindful
Capacity for self-reflection
Handles stressful situations skillfully
Adaptable/innovative
Professional
Appropriate level of confidence
Team player
Capacity for empathy
Enthusiasm for learning
Incorporates feedback
![Page 14: IMPLEMENTING MULTIPLE MINI-INTERVIEW (MMI) AT YOUR ...EDUCATIONAL OBJECTIVES Cite two reasons for implementing Multiple Mini-Interview (MMI) in residency Cite applicant interviewing](https://reader030.vdocuments.site/reader030/viewer/2022040918/5e92e695c114f0789b1ff7df/html5/thumbnails/14.jpg)
• Seek desired traits
• Do not test specific
medical knowledge
• Have no “correct” answer
• Allow interviewer flexibility
• Appropriate controversy
level with score dispersion
Step 2: Scenario Development
![Page 15: IMPLEMENTING MULTIPLE MINI-INTERVIEW (MMI) AT YOUR ...EDUCATIONAL OBJECTIVES Cite two reasons for implementing Multiple Mini-Interview (MMI) in residency Cite applicant interviewing](https://reader030.vdocuments.site/reader030/viewer/2022040918/5e92e695c114f0789b1ff7df/html5/thumbnails/15.jpg)
CONTROVERSY LEVEL
High
Low
Controversy
Score
Dispersion
![Page 16: IMPLEMENTING MULTIPLE MINI-INTERVIEW (MMI) AT YOUR ...EDUCATIONAL OBJECTIVES Cite two reasons for implementing Multiple Mini-Interview (MMI) in residency Cite applicant interviewing](https://reader030.vdocuments.site/reader030/viewer/2022040918/5e92e695c114f0789b1ff7df/html5/thumbnails/16.jpg)
EXAMPLE SCENARIO
You are an upper year resident rounding on a new admission of an 80 year old woman from an assisted living center with moderate dementia and neuropathy. Admitted last night by your night float for DM foot ulcer and rule out osteomyelitis.
You have interviewed and examined the patient on morning rounds, and the exam was consistent with your night float team’s assessment. Reviewing her studies you note that the MRI was completed on the wrong foot.
You are now summoned to the patient’s room as her daughter is requesting an update on her mother’s status and the results of the
tests so far.
![Page 17: IMPLEMENTING MULTIPLE MINI-INTERVIEW (MMI) AT YOUR ...EDUCATIONAL OBJECTIVES Cite two reasons for implementing Multiple Mini-Interview (MMI) in residency Cite applicant interviewing](https://reader030.vdocuments.site/reader030/viewer/2022040918/5e92e695c114f0789b1ff7df/html5/thumbnails/17.jpg)
PROBING QUESTIONS
Defend position while remaining adaptable
Clear verbal communication
Emotional quotient
Explore the “why”
![Page 18: IMPLEMENTING MULTIPLE MINI-INTERVIEW (MMI) AT YOUR ...EDUCATIONAL OBJECTIVES Cite two reasons for implementing Multiple Mini-Interview (MMI) in residency Cite applicant interviewing](https://reader030.vdocuments.site/reader030/viewer/2022040918/5e92e695c114f0789b1ff7df/html5/thumbnails/18.jpg)
PROBING QUESTIONS –
EXAMPLES
“My mother frequently sun-downs when in the hospital and I am angry that this will extend her stay.”
“We have a copay of 20%; who is to pay for this mistake?”
“How will you approach preventing this from happening again?”
![Page 19: IMPLEMENTING MULTIPLE MINI-INTERVIEW (MMI) AT YOUR ...EDUCATIONAL OBJECTIVES Cite two reasons for implementing Multiple Mini-Interview (MMI) in residency Cite applicant interviewing](https://reader030.vdocuments.site/reader030/viewer/2022040918/5e92e695c114f0789b1ff7df/html5/thumbnails/19.jpg)
Step 3: Applicant Assessment & Scoring
Dispersive and reliable
Capture “Red Flags”
Minimum passing standard
Score immediately after MMI
Record on paper or directly in ERAS
![Page 20: IMPLEMENTING MULTIPLE MINI-INTERVIEW (MMI) AT YOUR ...EDUCATIONAL OBJECTIVES Cite two reasons for implementing Multiple Mini-Interview (MMI) in residency Cite applicant interviewing](https://reader030.vdocuments.site/reader030/viewer/2022040918/5e92e695c114f0789b1ff7df/html5/thumbnails/20.jpg)
MMI 5: Overloaded Census Scoring APPLICANT: DATE:
Overall Performance: Interviewer:
1 2 3 4 5 6 7
Interpersonal and Communication Skills
1 2 3 4 5 6 7
Excellent listening, writing, or
nonverbal skills;
comprehensive, clear
explanations, fluent
(including euphemisms and
slang), easily understood
English (7)
Average listening,
writing, or nonverbal
skills; clear but not
comprehensive
explanations of
problems; moderately
accented and/or
understood English,
but had difficulties
with euphemisms and
slang (4)
Poor
listening,
writing,
nonverbal
skills; unable
to clearly
explain
complex
problems;
heavily
accented and
poorly
understood
English (1)
Personality
that is
difficulty to
work with;
Unable to
empathize
(1)
standard answers
without good
rationale; can
handle basic
conflicts (4)
top level EQ;
obvious team
player;
unusually
mature; strong
conflict
resolution skills
(7)
![Page 21: IMPLEMENTING MULTIPLE MINI-INTERVIEW (MMI) AT YOUR ...EDUCATIONAL OBJECTIVES Cite two reasons for implementing Multiple Mini-Interview (MMI) in residency Cite applicant interviewing](https://reader030.vdocuments.site/reader030/viewer/2022040918/5e92e695c114f0789b1ff7df/html5/thumbnails/21.jpg)
LOGISTICS: ASSISTANTS
• Schedule rooms/faculty
• Prep materials for interviewers (scenarios/score sheets)
Preparation
• MC: timekeeper, announcer
• Assistant to handle IT/Applicant/Schedule
Interview day
![Page 22: IMPLEMENTING MULTIPLE MINI-INTERVIEW (MMI) AT YOUR ...EDUCATIONAL OBJECTIVES Cite two reasons for implementing Multiple Mini-Interview (MMI) in residency Cite applicant interviewing](https://reader030.vdocuments.site/reader030/viewer/2022040918/5e92e695c114f0789b1ff7df/html5/thumbnails/22.jpg)
LOGISTIC: INTERVIEWERS
Orientation session prior to interview season
Send instructions & case before interview day
Each room: case, score sheet, pen, instructions
![Page 23: IMPLEMENTING MULTIPLE MINI-INTERVIEW (MMI) AT YOUR ...EDUCATIONAL OBJECTIVES Cite two reasons for implementing Multiple Mini-Interview (MMI) in residency Cite applicant interviewing](https://reader030.vdocuments.site/reader030/viewer/2022040918/5e92e695c114f0789b1ff7df/html5/thumbnails/23.jpg)
LOGISTICS –PHYSICAL
SPACE
# Rooms = # Applicants
Maximizes efficiency
Minimize travel time
(Offices in hallway, clinic exam rooms, retired hospital space?)
Must have PA system
(Conference phones?)
![Page 24: IMPLEMENTING MULTIPLE MINI-INTERVIEW (MMI) AT YOUR ...EDUCATIONAL OBJECTIVES Cite two reasons for implementing Multiple Mini-Interview (MMI) in residency Cite applicant interviewing](https://reader030.vdocuments.site/reader030/viewer/2022040918/5e92e695c114f0789b1ff7df/html5/thumbnails/24.jpg)
READING HOSPITAL FORMAT
Applicant reviews scenario
2 minScenario
discussion with interviewer
6 min2 min travel
One break session
80 min8 applicants
5 MMI
1 traditional
48 obs
![Page 25: IMPLEMENTING MULTIPLE MINI-INTERVIEW (MMI) AT YOUR ...EDUCATIONAL OBJECTIVES Cite two reasons for implementing Multiple Mini-Interview (MMI) in residency Cite applicant interviewing](https://reader030.vdocuments.site/reader030/viewer/2022040918/5e92e695c114f0789b1ff7df/html5/thumbnails/25.jpg)
Categorical Categorical Categorical Categorical
1
900 MMI 1 -- MMI 4 -- MMI 2 --
2910 PD Station BREAK MMI 5 -- MMI 3 --
3
920 MMI 2 -- MMI 1 -- MMI 4 --
4930 MMI 3 -- PD Station BREAK MMI 5 --
5
940 MMI 4 -- MMI 2 -- MMI 1 --
6
950 MMI 5 -- MMI 3 -- PD Station BREAK
7
1000 MMI 1 -- MMI 4 -- MMI 2 --
8
1010 BREAK MMI 5 -- MMI 3 -- PD Station
Osteo Osteo Osteo Osteo
1900 MMI 3 -- MMI 5 -- BREAK
2910 PD Station MMI 1 -- MMI 2 -- MMI 4 --
3
920 BREAK MMI 3 -- MMI 5 --
4
930 MMI 4 -- PD Station MMI 1 -- MMI 2 --
5940 MMI 5 -- BREAK MMI 3 --
6950 MMI 2 -- MMI 4 -- PD Station MMI 1 --
7
1000 MMI 3 -- MMI 5 -- BREAK
81010 MMI 1 -- MMI 2 -- MMI 4 -- PD Station
![Page 26: IMPLEMENTING MULTIPLE MINI-INTERVIEW (MMI) AT YOUR ...EDUCATIONAL OBJECTIVES Cite two reasons for implementing Multiple Mini-Interview (MMI) in residency Cite applicant interviewing](https://reader030.vdocuments.site/reader030/viewer/2022040918/5e92e695c114f0789b1ff7df/html5/thumbnails/26.jpg)
LOGISTICS –INFORM
APPLICANTS
Increasingly common in US U.M.E.
Unfamiliar to IMGs (beware of coaching)
Inform when invited; information on website to set expectations
Reinforce: no “correct answers;” seeking to understand thought process, communication skills, professionalism
![Page 27: IMPLEMENTING MULTIPLE MINI-INTERVIEW (MMI) AT YOUR ...EDUCATIONAL OBJECTIVES Cite two reasons for implementing Multiple Mini-Interview (MMI) in residency Cite applicant interviewing](https://reader030.vdocuments.site/reader030/viewer/2022040918/5e92e695c114f0789b1ff7df/html5/thumbnails/27.jpg)
Text:
RyanZimmerma925To:
22333
Questions from the audience
![Page 28: IMPLEMENTING MULTIPLE MINI-INTERVIEW (MMI) AT YOUR ...EDUCATIONAL OBJECTIVES Cite two reasons for implementing Multiple Mini-Interview (MMI) in residency Cite applicant interviewing](https://reader030.vdocuments.site/reader030/viewer/2022040918/5e92e695c114f0789b1ff7df/html5/thumbnails/28.jpg)
EDUCATIONAL OBJECTIVES
Cite two reasons for implementing Multiple Mini-Interview (MMI) in residency applicant interviewing.Cite
Produce a list of desired, non-cognitive traits of applicants to your GME program.Produce
List the structural elements and resources that are needed to utilize MMI GME programs (reference lecture materials).List
![Page 30: IMPLEMENTING MULTIPLE MINI-INTERVIEW (MMI) AT YOUR ...EDUCATIONAL OBJECTIVES Cite two reasons for implementing Multiple Mini-Interview (MMI) in residency Cite applicant interviewing](https://reader030.vdocuments.site/reader030/viewer/2022040918/5e92e695c114f0789b1ff7df/html5/thumbnails/30.jpg)
MOC QUESTION 1
Which of the following is the recommended first step in implementing MMI at
your program?
A: Create a list of key non-cognitive traits of potential house staff
B: Generate 5-7 MMI Scenarios
C: Gain house staff buy-in
D: Propose structure to Department Chair
![Page 31: IMPLEMENTING MULTIPLE MINI-INTERVIEW (MMI) AT YOUR ...EDUCATIONAL OBJECTIVES Cite two reasons for implementing Multiple Mini-Interview (MMI) in residency Cite applicant interviewing](https://reader030.vdocuments.site/reader030/viewer/2022040918/5e92e695c114f0789b1ff7df/html5/thumbnails/31.jpg)
MOC QUESTION 2
What is the role of probing questions in to each MMI scenario?
A: Probing questions test the applicants adaptability and emotional quotient
B: Probing questions increase assessment reliability
C: Probing questions test cognitive traits
D: Probing questions are designed to reduce interviewer fatigue
![Page 32: IMPLEMENTING MULTIPLE MINI-INTERVIEW (MMI) AT YOUR ...EDUCATIONAL OBJECTIVES Cite two reasons for implementing Multiple Mini-Interview (MMI) in residency Cite applicant interviewing](https://reader030.vdocuments.site/reader030/viewer/2022040918/5e92e695c114f0789b1ff7df/html5/thumbnails/32.jpg)
MOC QUESTION 3
Which of the following is a known limitation of the utility of MMI?
A: Applicants report that MMI is unfair
B: MMI only evaluated cognitive attributes
C: MMI scores tend to be higher in extroverted applicants
D: MMI requires FEWER support staff than do traditional interviews
![Page 33: IMPLEMENTING MULTIPLE MINI-INTERVIEW (MMI) AT YOUR ...EDUCATIONAL OBJECTIVES Cite two reasons for implementing Multiple Mini-Interview (MMI) in residency Cite applicant interviewing](https://reader030.vdocuments.site/reader030/viewer/2022040918/5e92e695c114f0789b1ff7df/html5/thumbnails/33.jpg)
BIBLIOGRAPHY
• Diaz Fraga, et. al. Reliability and acceptability of a five-station multiple mini-
interview model for residency program recruitment. Journal of Community
Hospital Internal Medicine Perspectives. 2013, 3: 21362
• Jerant. Et. Al. Does Applicant Personality Influence MMI performance and
Medical School Acceptance Offers? Academic Medicine. 2012, Vol 87-9:1250-9
• Dore, et. al. The Reliability and Acceptability of the Multiple Mini-Interview as a
Selection Instrument for Postgraduate Admissions. Acad Med. 2010;85:S60–S63