Competency-Based Medical Education: Origins and Prospects

Ralph P. Leischner, Jr. MD Institute for Medical Education

William C. McGaghie, PhD

No Conflict of Interest

Competency-Based Medical Education:Origins and Prospects

Five QuestionsWhat is medical competence?What are the origins of mastery learning? Why is this an issue in 2014?What are the answers in medical education?What are the barriers?

What is medical competence?

ACGME competencies

CanMEDS framework

Scottish Doctor learning outcomes

Competency-Based Curriculum (1978)

Differs from subject centered and integrated course models in three ways:1. Grounded in functions (competencies)

required for medical practice in a specific setting

2. Mastery learning (high achievement for all, time can vary)

3. Education is an experiment (learning and teaching are hypotheses subject to test)

“A competency-based curriculum in any setting assumes that the many roles and functions involved in the doctor’s work can be defined and clearly expressed.”

“Careful delineation of these components of medical practice is the first and most critical step in designing a competency-based curriculum.”

Latest Incarnation

AAMCEntrustable Professional Activities

“ . . . a set of activities that entering residents should be expected (entrusted) to perform on day one of residency without direct supervision.”

The Core EPAs for Entering Residency

1. History & physical2. Prioritized ddx3. Recommend and

interpret dx and screening tests

4. Enter patient orders/prescriptions

5. Document clinical encounter

6. Patient presentation7. Form clinical questions

and retrieve evidence

8. Give or receive a patient handover

9. Interprofessionalteamwork

10. Recognize patient needing urgent/emergentcare

11. Obtain informed consent for tests and procedures

12. Perform procedures13. ID system failures and

contribute to safety

© 2013 AAMC. May not be reproduced without permission.

CoreEPAs

For Entering Residency

EPAsFor any

Practicing Physician

Expectationsfor the

Medical School Graduate

EPAsFor

Specialties

© 2013 AAMC. May not be reproduced without permission.

EPA

DOC

DOC

DOC

M1M2

M1M2

M1M2

M1M2

M1M2

M1M2

EPA: EntrustableProfessional ActivityDOC: Domain of CompetenceC: CompetencyM: Milestone

C2

C3

C1

C4

C2

C5

© 2013 AAMC. May not be reproduced without permission.

M1M2

M1M2

M1M2

M1M2

M1M2

M1M2

Narrative description of a pre-entrustable learner

Narrative description of the entrustable learner

Take Home Messages• Implementation of the Core EPAs is an

invitation, not a mandate• These EPAs represent a “floor” not a “ceiling”• EPAs are units of work, competencies are

abilities of individuals• Entrustment decisions require frequent direct

observation in the work place• Help-seeking behavior and trustworthiness are

foundational to all EPAs

What are the origins of mastery learning?

• Carroll, Keller, Block & Bloom• Carroll, A model of school learning• Block, Mastery Learning

Teachers College Record 1963; 64: 723-733.

John B. Carroll Harvard University

A model of school learning

Journal of Applied Behavior Analysis

1968; 1(1): 79-89

“Good-Bye, Teacher…”¹Fred S. Keller

Arizona State University²

1Mastery Learning

James H. Block and Robert B. Burns

University of California, Santa Barbara

1976

Time and Learning

Benjamin S. Bloom University of Chicago

682 • September 1974 • American Psychologist

Carroll JB. A model of school learningTeachers College Record 1963; 64: 723-733

Time Needed in LearningAptitudeAbility to Understand InstructionQuality of Instruction

Time Spent in LearningTime Allowed for Learning (Opportunity)Perseverance

Degree of TimeLearning = f actually spent

Time needed

Mastery LearningFeatures1. Baseline, i.e., diagnostic testing;2. Clear learning objectives, units ordered by difficulty;3. Educational activities (e.g., deliberate skills practice)

focused on objectives;4. Minimum passing mastery standard (MPS) for each unit;5. Formative testing → mastery of each unit;6. Advancement if performance ≥ MPS; or7. Continued practice or study until MPS is reached8. Time varies, outcomes are uniform

McGaghie et al., Chest 2009

Why is This an Issue in 2014? • Great conceptual advancements (EPAs,

DOC, competencies, milestones)• Great at measuring acquired knowledge• But, obsolete model of clinical medical

education and outcome assessment• Using 19th century ideas, knowledge, and

technology to educate 21st century physicians

Osler (1903) The hospital as a college . . . “natural method of teaching”

[Medical education] “Starts with the patient, continues with the patient, ends with the patient”• Clinical curriculum = patients seen• No standardization• Passive educational model• No rigorous personnel outcome measurement in

clinical years (clinical ratings, sources of bias)

Osler modelCurrent system of clinical medical education is broken, esp. when we measure educational outcomes with rigor

Examples

• Mangione studies (1990s) Annals of Internal Medicine, JAMA, AJRCCM

• Resident’s review course (2002) Teaching and Learning in Medicine

• Bell et al. (2009) Annals of Surgery• Cohen et al. (2013) Academic Medicine• Barsuk et al. (2012) Neurology

Surgical Experience

●254 general surgery residency programs inthe U.S. as of 2006

●300 operations graded A, B, or C●A = 121: graduating general surgery

residents should be competent to performindependently

●Graduating 2005 residents (n = 1,022)●18/121 more than 10 times●83/121 less than 5 times●31/121 less than once●63/121 mode experience = 0

Significant variation between residents in operative experience for specific procedures

Intern “Boot Camp”

Mastery OutcomesRecognition of physical examination

findings (cardiac auscultation)Performance of procedures

(paracentesis, lumbar puncture)Management of critically ill patients

(ICU skills)Communication with patients

(code status discussion)

No relationship between CSE scores and age, gender, prior experience, self-confidence, or USMLE Step 1 and Step 2 scores

Source: Barsuk JH, Cohen ER, Caprio T, et al. Simulation‐based education with mastery learning improves residents’ lumbar puncture skills. Neurology 2012; 79(2): 132‐37. Reprinted with permission of Wolters Kluwer Health.

Clinical skills examination (checklist) pre‐and final posttest performance of 58 first‐year simulator‐trained internal medicine residents and baseline performance of 36 traditionally trained neurology residents.  Three internal medicine residents failed to meet the minimum passing score (MPS) at initial post‐testing.  PGY – postgraduate year.

Mastery Learning of Lumbar Puncture Skills

What are the answers in clinical medical education?

• Simulation-based mastery learning (SBML) with deliberate practice (DP)

• Controlled, laboratory setting• Rigorous measurement and feedback• Translational, downstream outcomes• Data: tool, not a weapon• Complements clinical experience

Study N Competency Assessed Randomized Trials1. Wayne, et al, 2005 38 Advanced Cardiac Life Support2. Ahlberg, et al, 2007 13 Laparoscopic choleystectomy3. Andreatta, et al, 2006 19 Laparoscopic skills4. Korndorffer, et al, 2005 17 Laparoscopic suturing5. Korndorffer, et al, 2004 20 Laparoscopic camera navigation6. Van Sickle, et al, 2008 22 Intracorporeal SuturingCohort Studies7. Issenberg, et al, 2002 98 Cardiology skills8. Barsuk, et al, 2009 18 Dialysis catheter insertion9. Butter, et al 2010 108 Cardiac auscultation Case-Control Studies10. Wayne, et al, 2008 78 Advanced Cardiac Life SupportPre-Post Baseline Studies11. Wayne, et al, 2008 40 Thoracentesis skills12. Barsuk, et al, 2009 41 Central venous catheter insertion13. Barsuk, et al, 2009 103 Central venous catheter insertion14. Stefanidis, et al, 2006 18 Laparoscopic suturing

Simulation-based Medical Education with Deliberate Practice Yields Better Results than Traditional Clinical Education:

A Meta-Analytic Comparative Review of the EvidenceMcGaghie WC, Issenberg SB, Barsuk JH, Cohen ER, Wayne DB Academic Medicine 2011 86: 706-711

Research Design and Study

Statistics for Each Study

Correlation and 95% CILower Upper

Correlation Limit Limit P-Value

0.81 0.70 0.88 0.0000.80 0.56 0.91 0.0000.67 0.40 0.84 0.0000.62 0.29 0.82 0.0010.52 0.17 0.75 0.0060.51 0.17 0.74 0.005

0.78 0.73 0.82 0.0000.61 0.29 0.81 0.0010.59 0.47 0.69 0.000

0.51 0.29 0.68 0.000

0.80 0.72 0.86 0.0000.79 0.70 0.86 0.0000.77 0.71 0.82 0.0000.71 0.55 0.83 0.000

0.71 0.65 0.76 0.000

Favors Traditional Clinical Education

Favors SBME with DP

-1.00 -0.50 0.00 0.50 1.00Overall Effect Size

Randomized Trials1. Wayne et al, 20052. Ahlberg et al, 20073. Andreatta et al, 20064. Korndorffer et al, 20055. Korndorffer et al, 20056. Van Sickle et al, 2008

Cohort Studies7. Issenberg et al, 20028. Barsuk et al, 20099. Butter et al 2010

Case-Control Studies10. Wayne et al, 2008

Pre-Post Baseline Studies11. Wayne et al, 200812. Barsuk et al, 200913. Barsuk et al, 200914. Stefanidis et al, 2006

What are the Barriers?• Inertia: clinical culture, competence is

ascribed socially• Rigorous, reliable measurement• Evaluation apprehension• Teamwork• Implementation science• Not money• Measurement of very complex medical

problems and outcomes

ReferencesBarsuk JH, Cohen ER, Caprio T, et al. Simulation-based education with mastery learning improves residents’ lumbar puncture

skills. Neurology 2012; 79 (2): 132-37.Bell RH, Biester TW, Tabuenca A, et al. Operative experience of residents in US general surgery programs: a gap between

expectation and experience. Annals of Surgery 2009; 249: 719-24.Block JH, ed. Mastery Learning: Theory and Practice. New York: Holt, Rinehart and Winston, 1971.Block JH, Burns RB. Mastery learning. In: Shulman LS, ed. Review of Research in Education, 4th ed. Itasca, IL: F.E. Peacock

Publishers, 1974, 3-49.Bloom BS. Time and learning. American Psychologist 1974; 29: 682-88.Carroll JB. A model of school learning. Teachers College Record 1963; 64: 723-33.Cohen ER, Barsuk JH, Moazed F, et al. Making July safer: simulation-based mastery learning during intern boot camp. Academic

Medicine 2013; 88: 233-39.Flynn T (Drafting Panel Chair). Core Entrustable Professional Activities for Entering Residency (CEPAER). Washington, DC:

Association of American Medical Colleges, 2013.Issenberg SB, McGaghie WC, Gordon DL, et al. Effectiveness of a cardiology review course for internal medicine residents using

simulation technology and deliberate practice. Teaching and Learning in Medicine 2002; 14: 223-28.Keller FS. “Good-bye, teacher . . .” Journal of Applied Behavior Analysis 1968; 1: 79-89. Mangione S, Nieman LZ, Gracely E, Kaye D. The teaching and practice of cardiac auscultation during internal medicine and

cardiology training: a nationwide survey. Annals of Internal Medicine 1993; 119: 47-54.Mangione S, Nieman LZ. Cardiac auscultatory skills of internal medicine and family practice trainees: a comparison of diagnostic

proficiency. JAMA 1997; 278: 717-22.Mangione S, Nieman LZ. Pulmonary auscultatory skills during training in internal medicine and family practice. American Journal

of Respiratory and Critical Care Medicine 1999; 159: 1119-24.McGaghie WC, Issenberg SB, Cohen ER, et al. Does simulation-based medical education with deliberate practice yield better results

than traditional clinical education? A meta-analytic comparative review of the evidence. Academic Medicine 2011; 86: 706-11.McGaghie WC, Miller GE, Sajid A, Telder TV. Competency-Based Curriculum Development in Medical Education. Public Health

Paper No. 68. Geneva, Switzerland: World Health Organization, 1978.McGaghie WC, Siddall VJ, Mazmanian PE, Myers J. Lessons for continuing medical education from simulation research in

undergraduate and graduate medical education. CHEST 2009; 135: 62S-68S.Osler W. The hospital as a college. In: Osler W, ed. Aequanimitas. Philadelphia: P. Blakiston’s Son & Co., Inc., 1932, 313-24.