Mastery Learning:A Paradigm Shift in Health Professions Education

William C. McGaghie, PhDNorthwestern University Feinberg School of Medicine

Department of Medical EducationNorthwestern Simulation

No Conflict of Interest

“There are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns—the ones we don’t know we don’t know.”

Donald Rumsfeld

Goal: Educate Superb Clinicians

• Effective & Safe• Quality Patient Care• Good Patient Outcomes

Deep ProbeKnowledgeSkillsAttitudesProfessionalism

Simulation Lab

Transfer to Patient Care

Simulation Lab

Goals of this Presentation1. Transfer of training pathway from the

simulation education lab to patient care settings2. Address translational science and the role of

mastery learning with deliberate practice3. Highlight features of rigorous simulation-based

health professions education (SBHPE)4. Interpret data from selected research studies on

SBHPE transfer of training5. Coda: effective SBHPE interventions, lessons

learned, research opportunities, & academic teams

Medical Education Research as Translational ScienceContributions of powerful medical education interventions to T1 – T4 outcomes

Focus T1 T2 T3 T4Increased or Improved

Knowledge, skill, attitudes, professionalism

Patient care practices

Patient outcomes Collateral effects, e.g., skill retention, ROI, indirect outcomes

Target Groups Individuals and teams

Individuals and teams

Individuals and public health

Individuals, teams, public health

Setting Education setting, e.g., simulation lab

Clinic and bedside Clinic, bedside, and community

Clinic, bedside, and community

McGaghie 2010; Barsuk & Szmuilowicz, 2015

Level of Translation

Key SBHPE Research Concepts

• Mastery Learning

• Deliberate Practice

Mastery LearningFeatures1. Baseline, i.e., diagnostic testing;2. Clear learning objectives, units ordered by difficulty;3. Educational activities (e.g., deliberate skills practice)

focused on objectives;4. Minimum passing mastery standard (MPS) for each unit;5. Formative testing → mastery of each unit;6. Advancement if performance ≥ MPS; or7. Continued practice or study until MPS is reached8. Time varies, outcomes are uniform

McGaghie et al., Chest 2009

12

Source: Barsuk JH, Cohen ER, Caprio T, et al. Simulation-based education with mastery learning improves residents’ lumbar puncture skills. Neurology 2012; 79(2): 132-37. Reprinted with permission of Wolters Kluwer Health.

Clinical skills examination (checklist) pre-and final posttest performance of 58 first-year simulator-trained internal medicine residents and baseline performance of 36 traditionally trained neurology residents. Three internal medicine residents failed to meet the minimum passing score (MPS) at initial post-testing. PGY – postgraduate year.

Mastery Learning of Lumbar Puncture ((S((kills

Source: Barsuk JH et al. Neurology 2012; 79(2): 132-37

Deliberate Practice (DP)Features1. Highly motivated learners with good concentration;2. Engagement with a well-defined learning objective or task; at an3. Appropriate level of difficulty; with4. Focused, repetitive practice; that leads to5. Rigorous, precise measurements; that yield6. Informative feedback from educational sources (e.g., simulators, teachers); and

where7. Trainees also monitor their learning experiences and correct strategies, errors,

and levels of understanding, engage in more DP; and continue with8. Evaluation to reach a mastery standard; and then9. Advance to another task or unit10. Goal: constant improvement

Ericsson Acad Med. 2004; McGaghie et al., Chest 2009

SBHPE Translational Science (TS) Example Program

CVC T1 → T2 → T3 → T4 ($, R, CE)

Thematic → Sustained → Cumulative

SBHPE-TS Research Example Central Lines

1. CVC Placement in Simulation Lab & MICU (T1)

2. CVC Insertion → ↓ Complications in MICU (T2)

3. CVC Insertion → ↓ CRBSI in MICU (T3)

4. CVC Insertion → ↑ Cost Savings in MICU (T4)

5. CVC Insertion Skills Retention (T4)

6. Unexpected Collateral Effects (T4)

(T1 Outcomes)

J Hosp Med. 2009; 4: 397-403

T1 Outcomes

Copyright © 2009 Wolters Kluwer. 2

Simulation-based mastery learning reduces complications during central venous catheter insertion in a medical intensive care unit

Barsuk, Jeffrey; McGaghie, William; Cohen, Elaine; OLeary, Kevin; Wayne, DianeCritical Care Medicine. 37(10):2697-2701, October 2009.doi: 10.1097/CCM.0b013e3181a57bc1

(T2 Outcomes)

Clinical Outcomes: Complications

Barsuk et al. Critical Care Medicine 2009

0

5

10

15

20

25

Pneumothorax Arterial Puncture CVC Adjustment Insertion Failure Rate

% o

f Ins

erte

d Li

nes

Complications

Traditionally-Trained Simulator-Trained

p=.757 p<.0005 p=.002 p=.005

(T3 Outcomes)

Arch Intern Med. 2009; 169: 1420-23.

Copyright restrictions may apply.

Barsuk, J. H. et al. Arch Intern Med 2009;169:1420-1423.

Timeline of residents rotating in the medical intensive care unit (ICU) and a comparison ICU

Copyright restrictions may apply.

Barsuk, J. H. et al. Arch Intern Med 2009;169:1420-1423.

85% ↓ in CRBSI

Monthly catheter-related bloodstream infection rates in a medical intensive care unit (MICU) and a comparison intensive care unit (ICU) before and after a simulation-based educational intervention in the MICU

T3 Outcomes

Cost Savings

Compared with the cost of the intervention ($111,916) the net savings ranged from $704,034 to $711,248.

7-1 ROI

T4 Outcomes

(T4 Outcomes)

Academic Medicine 2010; 85(10, Suppl.): S9-S12

T4 Outcomes

T4 Outcomes

“Deliberate practice is a concept that is not currently found in the nursing lexicon.”

(Clapper & Kardong-Edgren, 2012)Clinical Simulation in Nursing

• Randomized trial• 6-min. monthly

practice sessions• Voice-activated

manikin (VAM)• Sig. improvement in

Practice vs. Control group

• Clinical outcome measures

Clinical Outcome Measures

• Compression rate and depth• Percent of compressions performed with

adequate depth• Percent performed with correct hand

placement• Ventilation rate and volume• Percent of ventilations with adequate

volume

Key Result

“By practicing only 6 min. a month, students maintained or improved their CPR skills over the 12-month period.”

Oermann, Kardong-Edgren, Odom-MaryonResuscitation 2011; 82: 447-453

CPR Skill RetentionChest compressions goal: depth of at least 38 mm

Control group: ~ ½ compressions adequate depthPractice group: sig. more compressions adequate

depthVentilation goal: volume greater than 500 ml

Control group: ventilation skills ↓ over 1 yearPractice group: maintained ventilation skills over

1 year

Deliberate Practice Works!

Coda (Secret Sauce)

• Effective SBHPE interventions• Lessons learned• Research opportunities• Productive academic teams

INTERVENTION

Learners

Instructors

Curriculum

Healthcare System

Interventions = Products of Context

Effective SimulationIntervention

=Training Resources(Simulator)

XMotivatedLearners

XTrained SimulationInstructors

CurriculumIntegration XXHealthcare

System

HealthcareSystem

Effective SimulationIntervention

=Training Resources(Simulator)

XMotivatedLearners

XTrained SimulationInstructors

CurriculumIntegration XX

1 1 1 1 1 1

[In]Effective SimulationIntervention

=Training Resources(Simulator)

XMotivatedLearners

XTrained SimulationInstructors

CurriculumIntegration XXHealthcare

System

1 1 .50 1 1 .50

[In]Effective SimulationIntervention

=Training Resources(Simulator)

XMotivatedLearners

XTrained SimulationInstructors

CurriculumIntegration XXHealthcare

System

1 .40 .50 1 1 .20

Issenberg (2006)

7%

16%

38%

11%

19%

38%

0%

5%

10%

15%

20%

25%

30%

35%

40%

2007 2008 2009

Pret

est

Pass

Rat

e

Internal Jugular Subclavianp=.004 p=.028

(T4 Outcomes)

Nine Lessons Learned1. ML with DP is a key feature of SBHPE programs aimed at boosting

skill and knowledge acquisition among medical learners at all levels—strong, integrated, sustained

2. Robust measures yielding reliable data that permit valid decisions or inferences are a key feature of SBHPE

3. Rater training and constant calibration are essential4. No correlation between medical knowledge (USMLE scores) and

clinical skill acquisition5. ML model depends on setting high MPSs6. Self-assessments are biased, poor relation to performance measured

objectively7. ML with DP cannot be done “on the cheap.”8. Follow-up testing and remediation are needed to assure retention9. Implementation science

McGaghie (2008)

Ten Research Opportunities1. Rely on randomized trials? No, mastery learning!2. Stretch the measurement endpoint: T1 → T2 → T3 →

T4 ($, R, CE)3. Study DP features and quality4. Rigorous research designs5. Measurement [metric] development6. SBHPE affective consequences7. Skill maintenance8. Faculty development9. UTOST model: unit + treatment + observation + setting

+ time10. Team science

McGaghie (2008)

Attributes of Productive Academic Teams• Shared goals—common mission & vision• Functional diversity• Clear leadership—may change or rotate• Shared mental models & language• High standards, recognition, & credit• Sustained hard work / commitment• Physical proximity• Minimize status differences within the team• Maximize status of the team• Shared activities that breed trust

Hong & Page (2004) Wuchty, Jones & Uzzi (2007) McGaghie (2009) Bennett & Gadlin (2012)

Six Challenges for the Health Professions

1. Technology works!2. Power of inertia3. Hard work4. SBHPE: more than skill acquisition5. Fun & exciting6. Scholarship opportunities

ReferencesBarsuk JH, Cohen ER, Feinglass J, McGaghie WC, Wayne DB. Use of simulation-based education to reduce catheter-related bloodstream

infections. Archives of Internal Medicine 2009; 169 (15): 1420-1423.Barsuk JH, Cohen ER, Feinglass J, McGaghie WC, Wayne DB. Unexpected collateral effects of simulation-based medical education.

Academic Medicine 2011; 86: 1513-1517.Barsuk JH, Cohen ER, McGaghie WC, Wayne DB. Long term retention of central venous catheter insertion skills after simulation-based

mastery learning. Academic Medicine 2010; 85 (10 Suppl.): S9-S12.Barsuk JH, McGaghie WC, Cohen ER, Balachandran JS, Wayne DB. Use of simulation-based mastery learning to improve the quality of

central venous catheter placement in a medical intensive care unit. Journal of Hospital Medicine 2009; 4: 397-403.Barsuk JH, McGaghie WC, Cohen ER, O’Leary KS, Wayne DB. Simulation-based mastery learning reduces complications during central

venous catheter insertion in a medical intensive care unit. Critical Care Medicine 2009; 37 (10): 2697-2701.Bennett LM, Gadlin H. Collaboration and team science: from theory to practice. Journal of Investigative Medicine 2012; 60 (5): 768-775.Cohen ER, Feinglass J, Barsuk JH, Barnard C, O’Donnell A, McGaghie WC, Wayne DB. Cost savings from reduced catheter-related

bloodstream infection after simulation-based education for residents in a medical intensive care unit. Simulation in Healthcare 2010; 5: 98-102.

Ericsson KA. Deliberate practice and the acquisition and maintenance of expert performance in medicine and related domains. Academic Medicine 2004; 79 (10, Suppl.): S70-S81.

Hong L, Page SE. Groups of diverse problem solvers can outperform groups of high-ability problem solvers. Proceedings of the National Academy of Science 2004; 101(46): 16385-16389.

Issenberg SB. The scope of simulation-based healthcare education. Simulation in Healthcare 2006; 1: 203-208.Jones BF, Wuchty S, Uzzi B. Multi-university research teams: Shifting impact, geography, and stratification in science. Science 21 November

2008; 322: 1259-1262.McGaghie WC. Scholarship, publication, and career advancement in health professions education. AMEE Guide No. 43. Medical Teacher

2009; 31(7): 574-590.McGaghie WC. Medical education research as translational science. Science Translational Medicine 2010; 2: 19cm8.McGaghie WC, Issenberg SB, Petrusa ER, Scalese RJ. A critical review of simulation-based medical education (SBME) research: 2003-2009.

Medical Education 2010; 44: 50-63.McGaghie WC, Siddall VJ, Mazmanian PE, Myers J. Lessons for continuing medical education from simulation research in undergraduate

and graduate medical education: Effectiveness of continuing medical education: American College of Chest Physicians evidence-based educational guidelines. CHEST 2009; 135 (3, Suppl.): 62S-68S.

Oermann MH, Kardong-Edgren SE, Odom-Maryon T. Effects of monthly practice on nursing students’ CPR psychomotor skill performance. Resuscitation 2011; 82: 447-53.

Salas E, Rosen MA, Burke CS, Goodwin GF, Fiore SM. The making of a dream team: when expert teams do best. In: Ericsson KA, Charness N, Feltovich PJ, Hoffman RR,eds. The Cambridge Handbook of Expertise and Expert Performance. New York: Cambridge University Press; 2006, 439-453.