Pnina Weiss, MD

pnina.weiss@yale.edu

Bruce Herman, MD

Kathleen McGann, MD

Angela Myers, MD

Geoff Fleming, MD

Chris Kennedy, MD

Objectives Increase understanding of milestone-based

competency assessment in pediatric fellowship programs

Demonstrate how to incorporate milestone-based competency assessment into evaluations and feedback to fellows

Milestone reporting to ACGME Resident milestones

May - June 2014

Fellow milestones

Nov - Dec 2014

Why Milestones? For accreditation…..

Allow for continuous monitoring of programs

Provide data for evaluation and research for continuous improvement of graduate medical education

Ensure public accountability by reporting national aggregate competency outcomes by specialty

acgme.org

Why are Milestones Important to YOU?

For fellowship programs…… Provide a descriptive, developmental framework for

clinical competency committees

Support better assessment practices

Enhance opportunities for identification of struggling residents and fellows

Guide curriculum development

acgme.org

For fellows…… Provide more explicit and transparent expectations of

performance

Facilitate better feedback for professional development

Support better self-directed assessment and learning

acgme.org

What is a competent physician?

I know it when I see it

Justice Potter Stewart, in Jacobellis v. Ohio 378 U.S. 184 (1964), regarding pornography

Advances in medical education:

Competency

Outcome

Evaluation/Assessment

Systematic appraisal of skills, behaviors and attitudes of a learner

Info about actual performance can be compared to intended performance

Highlight gaps

Motivate learning

Milestones: part of Pediatrics!

Dreyfus Model of Professional Development

Medical

Student

Master

Clinician

Milestone-based competencies

Dreyfus, Stuart E.; Dreyfus, Hubert L. (February 1980). A Five-Stage Model of the Mental Activities Involved in Directed Skill Acquisition. Washington, DC: Storming Media.

www.abp.org/abpwebsite/publicat/milestones.pdf

p. 125 Great Resource

www.acgme.org/acgmeweb/Portals/0/PDFs/Milestones/PediatricsSubspecialtyMilestones.pdf

March-April 2014

Vol. 14 No. 28

Milestone-based Clinical Competency Assessment: Building blocks!

Domains (of Competence)* Patient Care

Medical Knowledge

Practice-Based Learning Improvement

Interpersonal & Communication Skills

Professionalism

System-Based Practice

Personal and Professional Development

*“Competencies”

Domains Divided into Competencies* Patient Care

1. Develop and carry out management plans

2. Gather essential and accurate information about the patient

3. Provide transfer of care that ensures seamless transitions

* “Subcompetencies”

Competencies divided into milestones

PC7. Develop and carry out management plans

Master

Clinician

Novice

Domains, Competencies & Milestones

Entrustable professional activities (EPA)

Entrustable professional activities (EPA)

Essential routine care activities that define a specialty or competency; describes a unit of work

Can the trainee be entrusted to perform them without direct supervision?

EPAs for Subspecialties Shared with Pediatrics

Apply public health principles and improvement methodology to improve care for populations

Facilitate handovers to another health care provider

Common to all subspecialties Engage in scholarly activities through the discovery,

application and dissemination of new knowledge Lead within the subspecialty profession

Subspecialty -specific Care for patients with acute _____ problems Provide care for patients with chronic ______ problems

EPAs mapped to competencies and milestones

Milestones

21 Milestone-based competencies for reporting to ACGME

“Milestone assessment”

Selecting a response in between levels: milestones in lower levels have been substantially demonstrated as well as some milestones in higher level

Selecting a response in the middle of a level: milestones in that level and lower levels have been substantially demonstrated

www.acgme.org/acgmeweb/Portals/0/PDFs/Milestones/PediatricsSubspecialtyMilestones.pdf

PC7.

Clinical Competency Committee:

July 2013

Composition

At least 3 faculty

May include non-physician members of the health care team

Responsibilities

Review all fellow evaluations semi-annually

Prepare and assure the reporting of Milestones evaluations of each fellow semi-annually to ACGME

Advise the program director regarding fellow progress, including promotion, remediation, and dismissal

CCC maps fellow to the milestones

www.acgme.org/acgmeweb/Portals/0/PDFs/Milestones/PediatricsSubspecialtyMilestones.pdf

PC7.

You are the CCC! Materials WORKSHOP 17: FELLOW VIGNETTES for CCC Packet: CCC Evaluation Form (Fellow) APPD JJ-TT Evals Process Choose Chair Map fellows to milestones in each competency (7)- 15 mins List challenges/potential solutions-10 mins Chair will report to us

Challenges?

Challenges Useful evaluation tool?

CCC- inefficient?

Milestones -difficult to understand

Faculty

Lack of knowledge

Faculty fatigue

Lack of data

Usefulness of an evaluation tool

Reliability

Validity

Acceptability by learner and faculty

Impact on future learning

Costs

Van Der Vleuten CPM. The assessment of professional competence: developments, research and practical implications. Adv Health Sci Educ 1996;1:41-67 Epstein RM. Assessment in Medical Education. NEJM 2007;356:387-96

Improve effectiveness of CCC?

Improve effectiveness of CCC?

Suggestions to improve effectiveness Faculty & Self Education Process

PLAN!!!!! Budget at least 40 mins/fellow the first time Distribute CCC eval forms ahead of time; have faculty fill out

beforehand Decide on how to reach consensus Have copy of CCC eval form available for each member at meeting Review evals at start of meeting to highlight problem areas Go by competency and map all fellows before going to next competency Record comments after each competency is mapped Include comments and competencies outside of the 21 if needed to

document problem areas Feedback to fellow!

Understandability: Subspecialty “milestones” 101

Suggestions to improve your familiarity with milestones Read them

Self-evaluation using milestones

Resources http://www.pedsubs.org/Webinar/webinar.cfm

http://pedsubs.org/issues/pdfs/SlidesForFeb182014CoPS%20WebinarV20.pdf

http://www.acgme.org/acgmeweb/tabid/442/GraduateMedicalEducation/SlidePresentationsforFacultyDevelopment.aspx

Faculty development Education

Minimize evaluator fatigue

Suggestions for faculty education Generic training for ALL faculty More intensive training for CCC members

Prepare CCC members for time commitment!

Venue: faculty meeting, special meetings Self-evaluations Resources (previous slide) Walk through CCC meeting Use vignettes (written or video)

Evaluator fatigue Don’t use all competencies; limit to a few Phase in the # of milestone-based competencies

Not enough data?

Milestone-based evaluations

Milestone-based evaluations

Milestone-based evaluations

Evaluations

Milesto

ne

Ro

tati

on

Sem

ian

nu

al

facu

lty

Clin

ic

Am

bu

lato

ry

CE

X

Resid

en

t

Stu

den

t

Sta

fr

Pati

en

t

Pro

ced

ure

Teach

ing

Sig

no

ut

J C

lub

CC

C

Self

12 23 10 5 6 5 3 2 1 2 1 2 22 22

Transfer PC3 x x x x x

Make informed decision PC6 x x x x x x

Carry out plan PC7 x x x x x

Role model PC12 x x x x x

M edical procedures PC8 x x x x

Evidence MK2 x x x x x x x

Identify strength PBLI1 x x x x

Analyze practice using QI PBLI4 x x x

Information technology PBLI7 x x x x x

Educate pts, students PBLI9 x x x x x x x x

Effective teacher PBLI8 x

Communicate w/ pts ICS 1 x x x

Communicate w / MDs ICS3 x x x x x

Work as team member/leader ICS4 x x x x x x

Consultant ICS5 x x x x x

M edical records ICS6 x x x x x

Humanism, compassion PROF1 x x x x

High standards ethics PROF2 x x x x x x x x

TrustworthyPPD5 x x x x

Ambiguity PPD8 x x x

Leadership skills PPD6 x x x x x

Coordinate healthcare SBP2 x x x x x

Work in teams/pt safety SBP5 x x x x

Work in health care settings SBP1 x x x

Incorporate cost SBP3 x x x x

Identify systems error SBP 6 x x x

Feedback

Information communicated to a learner to modify thinking or behavior to improve learning

Perfect timing for semiannual meetings with fellow!

CCC meet (Nov, May)

Report milestones to ACGME (Nov/Dec, May/June)

Semiannual meeting with fellow (Dec, June)

Better formative feedback

Needs improvement

Develops and carries out management plans based on directives from others, either from the health care organization or the supervising physician. Unable to adjust plans based on individual patient differences or preferences. Communication about the plan is unidirectional from the practitioner to the patient and family.

To improve you should strive for: Develops and carries out management

plans based on one’s theoretical knowledge and/or directives from others. Can adapt plans to the individual patient, but only within the framework of one’s own theoretical knowledge. Unable to focus on key information, so conclusions are often from arbitrary, poorly prioritized, and time-limited information gathering. Management plans based on the framework of one’s own assumptions and values.

Develop and carry out management plans

Effective feedback Constructive Timely Ongoing Consistent By authoritative person over extended duration of time Face-to-face Concrete Goal-oriented Focused on product, not learner Understandable

“Complex feedback is likely to be ignored or its main messages lost”

Archer JC. State of the Science in Health Professional Education: Effective Feedback. Med Ed 2010; 44: 101-108

Giving your (problem) fellow feedback….. Materials

B. WORKSHOP 17: FELLOW MINIVIGNETTES for SELECTION of MILESTONE-BASED COMPETENCY

Table of Contents-Appendix of Abbreviated Pediatric Milestones

Milestones- paper, on-line (www.abp.org) or APPD meeting website

Process

Groups assigned Fellows #1-7

Select 2 competencies and plot to competency/milestone level(1 will NOT be part of 21)- as a group

Break into pairs

Give your fellow constructive feedback using the milestones

Report back the competency/milestone that you used

#1: Your fellow always runs behind the 8-ball She comes in 30 minutes before morning rounds. She examines the patients on your service, but when the service is busy does not always get to examine the consults before you round together. When paged in the afternoon for a nonurgent consult, she often puts it off till the following day (a day that you both have clinic). She starts to write the notes immediately after rounds instead of heading over to discuss the discharge plans on your patients with the residents. She sometimes seems to “forget” to follow up on important labs or tasks. Sometimes she dismissively says that she’s “pretty sure they were normal.” You find yourself checking on the critical labs and ensuring that the “to do” list is complete.

PC2 Organize and prioritize responsibilities to provide patient care that is safe, effective and efficient

PPD5. Demonstrate trustworthiness that makes colleagues feel secure when one is responsible for the care of patients

#2: Your new fellow has trouble getting along with others After the first month, your nurse pages you with an update because she says jokingly that she’d rather ask you than the fellow. Upon questioning, she confides that the fellow has been abrupt with her and makes her feel stupid for calling. According to your faculty, he is distant. He is respectful to them but has a short fuse with support staff. He is efficient but doesn’t seem to do much outside reading. During your next rotation with him, you overhear him snap at the respiratory therapist for suggesting a ventilator change. When you sit down with him to discuss his behavior, he breaks down and says that he is having issues with his personal life.

PPD2 Use healthy coping mechanisms to respond to stress PC12: Provide appropriate role modeling Prof2: Professional conduct SBP5. Work in interprofessional teams to enhance patient safety and

improve patient care quality

#3: Your fellow is intellectually lazy She accurately obtains and presents the history on new patients and comes up with a basic plan. For your new admission, you discuss the reason for starting an additional medication. You describe the pharmacokinetics and evidence-based rationale for it. You later read her progress note and find that she has rewritten almost verbatim your discussion. The next day after discussing a new consult, you suggest some additional reading material. Again, you find that she has just rewritten your own words in the discussion in the consult note. She says that she didn’t get to the outside reading. This pattern continues for the rest of your week together.

PBLI3. Identify and perform appropriate learning activities to guide personal and professional development

MK2 Critically evaluate and apply current medical information and scientific evidence for patient care

ICS5. Act in a consultative role to other physicians and health professionals

#4: Your fellow just isn’t doing enough outside reading Medical knowledge isn’t where it should be. In-training exam scores were low, particularly in pathophysiology. He can regurgitate the rules of patient care that you’ve given him, but has read few of the primary sources. When asked why he chose a particular medication, he doesn’t know the rationale. You have suggested multiple times that he read the NIH guidelines. He copied one of the tables and does carry it around to help with management.

MK1: Demonstrate sufficient knowledge of the basic and clinically supportive sciences appropriate to (pediatrics)

PBLI10: Take primary responsibility for lifelong learning to improve knowledge, skills, and practice performance through familiarity with general and experience-specific goals and objectives

MK2: Critically evaluate and apply current medical information and scientific evidence for patient care

#5. Your fellow is passive-aggressive in response to your feedback She worked for two years as a pediatrician in private practice before coming to your program. You had a long discussion with her at the end of her last rotation. You told her that she needs to preround on all the patients on your service and the consults. She needs to read more primary material and expand the depth of her discussions. You correct her choice of medications and explain why they’re less preferred. You are on service with her one month later and you notice the exact same issues. When approached about her behavior, she tells you that that’s the way they did it in her old institution or when she was in private practice.

PBLI5: Incorporate formative evaluation feedback into daily practice

PPD4: Practice flexibility and maturity in adjusting to change with the capacity to alter behavior

PBLI1. Identify strengths, deficiencies, and limits in one’s knowledge and expertise

#6. Your new fellow doesn’t call you when he should He just started and in the space of one week, he admitted a patient to your service without telling you, ordered (incorrect) antibiotics on another without consulting you and failed to contact you for help when one of your patients deteriorated overnight. When you came in that morning, you found the patient in severe distress and had to transfer her to the ICU. When you discussed it with him, he promised to notify you in the future, but didn’t seem convinced that he had done anything wrong.

PPD1. Develop the ability to use self-awareness of knowledge, skills and emotion limitations to engage in appropriate help seeking behaviors

PBLI1: Identify strengths, deficiencies and limits in one’s knowledge and expertise

PPD5: Demonstrate trustworthiness that makes colleagues feel secure when one is responsible for the care of patients

#7. Your fellow is careless with notes Your fellow’s inpatient notes ramble and are full of errors. The HPI is overly detailed. Because of “copy forward,” the interval history is from one week ago, medications that were discontinued are still in medication list, and the problem list is outdated. The physical exam appeared before the interval history. The notes are done in a timely fashion, but you still have to edit for accuracy and correct the many grammatical and spelling errors.

ICS6 Maintain comprehensive, timely and legible medical records

ICS3 Communicate effectively with physicians, other health professionals and health-related agencies

Importance of self-evaluation

Comments?

pnina.weiss@yale.edu

References Assessment • Epstein RM. Assessment in Medical Education. NEJM 2007;356:387-96 • Downing SM. Reliability: on the reproducibility of assessment data. Med Ed

2004;38:1006-12 • Van Der Vleuten CPM. The assessment of professional competence: developments,

research and practical implications. Adv Health Sci Educ 1996;1:41-67 Milestones • Hicks PJ, Englander R, Schumacher DJ, et al. Pediatrics milestone project: next steps

toward meaningful outcomes assessment. J Grad Med Educ 2010;2:577-84 • Jones MD, Jr., Rosenberg AA, Gilhooly JT, Carraccio CL. Perspective: Competencies,

outcomes, and controversy--linking professional activities to competencies to improve resident education and practice. Acad Med: JAAMC 2011;86:161-5

• Schumacher DJ, Lewis KO, Burke AE, et al. The Pediatrics Milestones: Initial Evidence for Their Use as Learning Road Maps for Residents. Acad Pediatr;13:40-7

• The Pediatric Milestone Project www.abp.org/abpwebsite/publicat/milestones.pdf

References Feedback

Archer JC. State of the science in health professional education: effective feedback. Med Ed 2010;44:101-8

Hewson MG, Little ML. Giving feedback in medical education: verification of recommended techniques. J Gen Intern Med 1998;13:111-6.

Self-assessment

Gordon MJ. A Review of the validity and accuracy of self-assessments in health professions training. Acad Med 1991; 66:762-769.

Davis DA, Mazmanian PE, Fordis M, Thorpe KE , Perrier L. Accuracy of physician self-assessment compared with observed measures of competence: a systematic review. JAMA 2006; 296:1094-1102.