acgme milestones project update

59
Accreditation Council for Graduate Medical Education The Next Accreditation System TSDA Los Angeles, CA--January 28, 2013 Walter Merrill, MD Vice Chair Thoracic Surgery RRC Peggy Simpson, EdD Executive Director Thoracic Surgery RRC Ara Vaporciyan, MD TS Milestones Working Group Stephen Yang, MD TS Milestones Working Group

Upload: vuongquynh

Post on 02-Jan-2017

231 views

Category:

Documents


2 download

TRANSCRIPT

Page 1: ACGME Milestones Project Update

Accreditation Council for Graduate Medical Education

The Next Accreditation System

TSDA

Los Angeles, CA--January 28, 2013

Walter Merrill, MD Vice Chair

Thoracic Surgery RRC

Peggy Simpson, EdD Executive Director

Thoracic Surgery RRC

Ara Vaporciyan, MD TS Milestones Working

Group

Stephen Yang, MD TS Milestones Working

Group

Page 2: ACGME Milestones Project Update

Disclosures

• Financial

• None

• Fiduciary

• Full-time employee of ACGME

Page 3: ACGME Milestones Project Update

Agenda

• Next Accreditation System

• Milestones

• Milestones Beta Test Results

© 2012 Accreditation Council for Graduate Medical Education (ACGME)

Page 4: ACGME Milestones Project Update

Next Accreditation System

Page 5: ACGME Milestones Project Update

Next Accreditation System

• Goals

• Program activities

• Overview of accreditation in NAS

• Timeline for transition

Page 6: ACGME Milestones Project Update

Next Accreditation System Goals

• Reduce the burden of accreditation

• Free good programs to innovate

• Assist poor programs to improve

• Realize the promise of Outcomes

• Provide public accountability for outcomes

© 2012 Accreditation Council for Graduate Medical Education (ACGME)

Page 7: ACGME Milestones Project Update

Conceptual change from…

The Current Accreditation System

Rules

Corresponding Questions

“Correct or Incorrect”

Answer

Citations and Accreditation

Decision

Rules

Corresponding Questions

“Correct or Incorrect”

Answer

Citation and Accreditation

Decision

© 2012 Accreditation Council for

Graduate Medical Education (ACGME)

Page 8: ACGME Milestones Project Update

To...

The Next Accreditation System

© 2012 Accreditation Council for Graduate Medical Education (ACGME)

Continuous

Observations

Potential

Problems

Diagnose

the Problem

(if there is one)

Ensure Program

Fixes the Problem Promote

Innovation

Page 9: ACGME Milestones Project Update

NAS-Annual Inputs

RRC

Board Pass Rate

Case Logs/Clinical

Experience

ADS Annual Update

Resident

Survey

Core Faculty Survey

Milestone

Semi Annual

Reporting

Page 10: ACGME Milestones Project Update

Next Accreditation System

• No PIF’s

• No site visits (as we know them)

• Focused site visits for “issues”

• Self-study visits every ten years

© 2012 Accreditation Council for Graduate Medical Education (ACGME)

Page 11: ACGME Milestones Project Update

Internal Reviews

• Currently required at mid-cycle

• In NAS, NO requirement for routine IRs

• GMEC must demonstrate program oversight

• Annual Program Reviews

• Special Review for programs when warranted

© 2012 Accreditation Council for Graduate Medical Education (ACGME)

NOTE: New Institutional Requirements available for comment on 11/19/2012

Page 12: ACGME Milestones Project Update

Streamlined ADS Annual Update

• 33 questions removed

• 14 questions simplified

• Very few essay questions

• Self-reported board pass rate removed

• Faculty CVs removed

• 11 MCQ or Y/N questions added

© 2012 Accreditation Council for Graduate Medical Education (ACGME)

Page 13: ACGME Milestones Project Update

Current PIF Faculty CV

© 2012 Accreditation Council for Graduate Medical Education (ACGME)

Page 14: ACGME Milestones Project Update

Scholarly Activity Template

© 2012 Accreditation Council for Graduate Medical Education (ACGME)

Page 15: ACGME Milestones Project Update

Scholarly Activity Template

• For each core faculty member enter:

–x Pub Med ID’s

–Four numbers

–Answer two Y/N questions

• For each resident with scholarly activity enter:

–x Pub Med ID’s

–Two numbers

–Answer two Y/N question

© 2012 Accreditation Council for Graduate Medical Education (ACGME)

Page 16: ACGME Milestones Project Update

NAS Timeline

Phase I specialties • Neurological Surgery

• Orthopedic Surgery

• Urology

• Diagnostic Radiology

• Emergency Medicine

• Internal Medicine

• Pediatrics

• Subspecialties of all above

© 2012 Accreditation Council for Graduate Medical Education (ACGME) JGME 2012; 4:399

Page 17: ACGME Milestones Project Update

NAS: Program Activities

2014-2015

Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun

Milestones Yr 0 Yr 1 Yr 1

Faculty Survey Yr 1

Resident Survey Yr 1

ADS Update Yr 1

Case Logs Yr 0 Yr 1

Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun

© 2012 Accreditation Council for Graduate Medical Education (ACGME)

Page 18: ACGME Milestones Project Update

Conceptual Model of Standards Implementation

Across the Continuum of Programs in a Specialty

STANDARDS

Outcomes

Core Process

Detail Process

© 2012 Accreditation Council for

Graduate Medical Education (ACGME)

Continued

Accreditation

Withdrawal of Accreditation

Accreditation

with Warning

Probationary

Accreditation

2-4% 10-15% 75-80%

<1%

Application

for

New Program

Outcomes

Core Process

Detail Process

Outcomes

Core Process

Detail Process

Outcomes

Core Process

Detail Process

Initial

Accreditation

Page 19: ACGME Milestones Project Update

NAS in a Nutshell

• Continuous Accreditation Model

• Based on review of annually submitted data

• SVs replaced by 10-year Self-Study Visit

• Program Requirements revised every 10 yrs.

• Program Requirements organized by:

• Outcomes

• Core Processes

• Detailed Processes

© 2012 Accreditation Council for Graduate Medical Education (ACGME)

Page 20: ACGME Milestones Project Update

Summary: NAS

• Focus on outcomes

• Reduce burden

• Allow programs to innovate

© 2012 Accreditation Council for Graduate Medical Education (ACGME)

Page 21: ACGME Milestones Project Update

Milestones

Milion of

Constantinople

Gemas

Malaysia

Apian Way

Yorkshire Moors

Portadon Ireland

County Cork

Key West, FL

Boston, MA

Via Ignatia

Page 22: ACGME Milestones Project Update

Creation of Milestones

ABTS

Milestones

RRC TSDA

AATS

STS

JCTSE

Residents

© 2012 Accreditation Council for Graduate Medical Education (ACGME)

Page 23: ACGME Milestones Project Update

TS Milestones

Advisory Group

• Douglas Wood, MD

• William Baumgartner, MD

• John Calhoon, MD

• David Fullerton, MD

• John R. Potts, III, MD

• Peggy Simpson, EdD

Working Group • Carolyn E. Reed, MD

• Edward Bove, MD

• Andrea J. Carpenter, MD

• Robert Higgins, MD

• Richard Lee, MD

• J. Wayne Meredith, MD

• Walter Merrill, MD

• Tom C. Nguyen, MD

• Peggy Simpson, EdD

• Ara Vaporciyan, MD

• Thomas Varghese, MD

• Edward Verrier, MD

• Cameron Wright, MD

• Stephen Yang, MD

Page 24: ACGME Milestones Project Update

Timeline of Activities

• Kick-Off Meeting for Working Group • March 30-31, 2012

• Conference Calls • April 11, 2012

• May 16, 2012

• May 23, 2012

• Working Group Meetings • September 7-8, 2012

• January 25, 2013

• Beta Test • November-December 2012

Page 25: ACGME Milestones Project Update

Milestones Creation Status

Completed

• CRS • GS • IM • MG • Neuro • OB/Gyn • Ophthal • OS • Path • Peds • PS • Radiology • TY • Urology • Expert Panel

Drafts

• A&I • Derm • EM • NS • NM • TS • PMR • Psych • RO • TS • Anes • FM

Dec 2012

All specialties to have first draft of milestones

July 2013

Implement

EM

IM

NS

Ortho

Ped

Diag rad

Urology

July 2014

Implement All Remaining Specialties

(i.e. THORACIC SURGERY)

25

Page 26: ACGME Milestones Project Update

CT Surgery Milestones

26

ACGME

ABTS

Advisory Group

Thoracic Surgery

Milestones Group

Pilot Sites

July 2014

All Programs

Page 27: ACGME Milestones Project Update

Thoracic Milestones

Advisory Group

• William Baumgartner

• John Calhoon

• David Fullerton

• John Potts

• Peggy Simpson

• Doug Wood

27

Page 28: ACGME Milestones Project Update

CT Surgery Milestones

28

ACGME

ABTS

Advisory Group

Thoracic Surgery

Milestones Group

Pilot Sites

July 2014

All Programs

Page 29: ACGME Milestones Project Update

Thoracic Milestones Group

Carolyn Reed/ Walter Merrill (Chair)

• Andrea J. Carpenter

• Jim Fann

• Robert Higgins

• Rick Lee

• J. Wayne Meredith

• Tom Nguyen (TSRA)

• Peggy Simpson

• Ara Vaporciyan

• Tom Varghese

• Ed Verrier

• Cam Wright

• Steve Yang

• Laura Edgar (ACGME)

29

Page 30: ACGME Milestones Project Update

Milestone Development

Sponsorship

• ACGME • Certification

Boards

Who

• Working Group

• Advisory Group

When/How

• 4 meetings • Weekly conf

calls • Pre/Post

meeting briefings

30

Page 31: ACGME Milestones Project Update

Guiding Principles

31

Feasibility

• Balance costs with benefits

• Manageable number of milestones - <36 (actually have 26)

Quality

• Improvement over current approaches

• Meaningful & substantive

• “Measurable”

Reporting

• Specialty-wide use of 5-level template for milestones reporting

• Central data repository

Page 32: ACGME Milestones Project Update

32

Page 33: ACGME Milestones Project Update

Framework Options

• Anatomic area

• Disease states/Diagnoses

• Disease classification

• Domains

• Subspecialty areas

33

Page 34: ACGME Milestones Project Update

CT Surgery Milestones

Each resident will be evaluated on the following milestones:

34

Patient Care and Medical

Knowledge

Ischemic Heart Disease

CPB, Myocardial Protection and

Temporary Circulatory Support

Valvular Disease

Great Vessel Disease

End Stage Cardiopulmonary Disease

Congenital Heart Disease

Critical Care

Esophagus

Lung and Airway

Chest Wall/Pleura/Mediastinum

General Competencies

Professionalism – Ethics and Values;

Personal Accountability

Practice Based Learning and Improvement

– Learning; Research and Teaching

Interpersonal and Communication Skills

Systems Based Practice – Patient Safety;

Resource Allocation; Practice Management

Page 35: ACGME Milestones Project Update

Key Points

• The levels DO NOT correspond to year of

residency

• Residents are expected to reach Level 4 by

the end of training (NOT level 5)

35

Page 36: ACGME Milestones Project Update

Milestone Template

36

Page 37: ACGME Milestones Project Update

Sample Milestone 1

37

Page 38: ACGME Milestones Project Update

Sample: PC Valvular Disease

38

Page 39: ACGME Milestones Project Update

Sample Milestone 3

39

General Competencies

Professionalism – Ethics and Values; Personal

Accountability

Practice Based Learning and Improvement –

Learning; Research and Teaching

Interpersonal and Communication Skills

Systems Based Practice – Patient Safety; Resource

Allocation; Practice Management

Page 40: ACGME Milestones Project Update

EVALUATION

METHOD

Record

Review

Chart

Stimulation

Recall Check List Global Rating

Standardized

Patients

OSCE/

CASPE

Simulations

and Models

360 Global

Rating

Learning

Portfolios ITE

Mock Oral

Exam

Procedures

and Case

Logs

Patient

Survey

COMPETENCY

Patient Care/Technical Skills 1 1 2 2 1 1 2 2 2 2 1

Medical Knowledge 1 2 2 3 1 1

Practice-Based Learning and

Improvement

Evaluate care/Self-Improvement 2 2 2 2 3 1 1 1 2 2

Research and Teaching 1 2 3 3 1 3

Interpersonal and Communication

Skills 3 1 1 2 1

Professionalism

Ethics and Values 2 2 1 2 1 2 1

Personal Accountability 2 2 2 2

System-Based Practice

Patient Safety 1 3 1 2 1

Resource Allocation 3 3 2 3

Practice Management 2 3 2 2 3 1

Key:

1 = most desirable

2 = next best method

3 = potentially applicable

Adapted from Toolbox of Assessment Methods, ACGME and ABMS, v 1.1, Sept 2000

Assessment Toolbox Matrix

(modified ACGME, 2000)

Page 41: ACGME Milestones Project Update

SESATS

Moodle

Courses

Simulation /

Video

Assessment

Database

Patient

Outcomes

?Senior Tour

Assessments Apple Apps

1 2

1 1 2

2 1

2

2

2

Specific for

CT Surgery

COMPETENCY

Patient Care/Technical Skills

Medical Knowledge

Practice-Based Learning and

Improvement

Evaluate care/Self-Improvement

Research and Teaching

Interpersonal and Communication

Skills

Professionalism

Ethics and Values

Personal Accountability

System-Based Practice

Patient Safety

Resource Allocation

Practice Management

Assessment Toolbox Matrix

Specific for CT Surgery

Specific for Thoracic

SESATS

Moodle Courses

Simulation/Video Assessment

Database Pt Outcomes

M&M presentation

Residents as Educators Assessment

Chart Audit

Mock orals

Page 42: ACGME Milestones Project Update

CT Surgery Milestones

42

ACGME

ABTS

Advisory Group

Thoracic Surgery

Milestones Group

Pilot Sites

July 2014

All Programs

Page 43: ACGME Milestones Project Update

Future Strategies

• Beta testing on larger scale

• Publish manuscript:

• Intro, Background, Assessment

Tools

• Pilot results, Implementation

Guidelines

• Produce manual like Int Med

• Faculty development on a large

scale

• Make links to WebBrain

Curriculum

Page 44: ACGME Milestones Project Update

How will the milestones affect

me?

► This is a COMPOSITE

Review for the RRC for

program review

effectiveness (not end of

rotation, does not replace

rotation evaluation)

► Assesses resident’s

progression through the

program

► RRC WILL NOT look at

individual resident reports

Page 45: ACGME Milestones Project Update

When will the Milestones be

implemented?

► “Go live” with TS Milestones in 7/2014

► Begin preparatory effort and faculty development

with PD/rep

● Beta testing 10 more institutions

● Sessions at AATS’13/STS’14

● Educate the Educator’13 and ‘14

● Webinars

Page 46: ACGME Milestones Project Update

Initial Milestones Project Pilot

Ara Vaporciyan, M.D.

Page 47: ACGME Milestones Project Update

Goals of the Pilot Study

• Assess their adequacy

• Assess time requirements

• Assess feasibility/methods

• Collect general feedback

All collected

from end users

Page 48: ACGME Milestones Project Update

Results

• What was the time burden?

Median Mode

Longest time spent on a single trainee? 15.7 10

Shortest time spent on a single trainee? 12.3 5

Page 49: ACGME Milestones Project Update

Results

• What tools were used to assign

milestones? • 63 different tools used (average of 9 tools per

program) 360o feedback 26%

Direct observation w immediate written assessment 19%

Peer evaluation 13%

Structured case w immediate written assessment 10%

End of rotation evaluations 10%

Simulators w built in or faculty assessment 6%

Mock orals/Milestones as an assessment tool 5%/5%

Patient Survey/In-training exam 3%/3%

Chart audit or review 2%

Other written tests/QI projects 0%

Page 50: ACGME Milestones Project Update

Results

• Adequacy and feasibility

Positive Negative

(SA/A) (D/SD)

They allow discrimination between levels of competency.

They accurately represent the scope of our practice

I thoroughly read and reviewed each description

They represent a realistic progression of knowledge,

skills and behaviors

They provide a meaningful evaluation of the trainee

I have enough information about my residents to assign

a level

It is easy to select the single best level

Page 51: ACGME Milestones Project Update

Results

• Positive findings, generally accepted

Positive Negative

(SA/A) (D/SD)

They allow discrimination between levels of competency. 90% (15/75) 10% (10/0)

They accurately represent the scope of our practice 98% (25/73) 2% (2/0)

I thoroughly read and reviewed each description 95% (38/57) 5% (5/0)

They represent a realistic progression of knowledge,

skills and behaviors

They provide a meaningful evaluation of the trainee

I have enough information about my residents to assign

a level

It is easy to select the single best level

Page 52: ACGME Milestones Project Update

Results

• Less agreement, some outliers

Positive Negative

(SA/A) (D/SD)

They allow discrimination between levels of competency. 90% (15/75) 10% (10/0)

They accurately represent the scope of our practice 98% (25/73) 2% (2/0)

I thoroughly read and reviewed each description 95% (38/57) 5% (5/0)

They represent a realistic progression of knowledge,

skills and behaviors 90% (10/80) 10% (8/2)

They provide a meaningful evaluation of the trainee 82% (8/74) 18% (18/0)

I have enough information about my residents to assign

a level

It is easy to select the single best level

Page 53: ACGME Milestones Project Update

Results

• Least agreement, more evenly split

Positive Negative

(SA/A) (D/SD)

They allow discrimination between levels of competency. 90% (15/75) 10% (10/0)

They accurately represent the scope of our practice 98% (25/73) 2% (2/0)

I thoroughly read and reviewed each description 95% (38/57) 5% (5/0)

They represent a realistic progression of knowledge,

skills and behaviors 90% (10/80) 10% (8/2)

They provide a meaningful evaluation of the trainee 82% (8/74) 18% (18/0)

I have enough information about my residents to assign

a level 71% (5/64) 19 (19/0)

It is easy to select the single best level 51% (0/51) 49% (44/5)

Page 54: ACGME Milestones Project Update

What Changes are Possible?

What Can We Change

• Specifics within

milestones

• Refined

• Moved up or down a level

• Additional examples

provided

What Can’t We Change

• General categories

• Must remain aligned with

competencies

• 5 Levels

• Common to all specialties

Page 55: ACGME Milestones Project Update

Pilot Summary

• The majority of pilot participants agreed that the milestones

• discriminate trainees by competency

• reflect what we want them to know.

• Must distinguish “Reporting of Competency” from “Assessment of Competency”

• The Milestones are a reporting tool

• Assessment must still be done locally, usually with multiple tools

• There is time for additional piloting and changes before we go live

• Project initiation July 1, 2014

• Initial report due approx. 6 months later

Page 56: ACGME Milestones Project Update

Summary: Milestones

• Created by the specialty

• Provide narratives to describe trajectory

from neophyte to practitioner

• Uses existing tools to track outcomes

• Will soon provide normative data

© 2012 Accreditation Council for Graduate Medical Education (ACGME)

Page 57: ACGME Milestones Project Update

What’s Next For NAS and

Thoracic Surgery?

• Look for information on ACGME Monthly

“Webinars”

• General Information

• Specialty Specific Webinars

• Live sessions for Q&A participation

• Look for announcements in ACGME e-

Communication

• Registration required

• Will be recorded for future use

Page 58: ACGME Milestones Project Update

What’s Next For NAS and

Thoracic Surgery? (cont.)

• Educational Sessions for PDs, faculty,

coordinators at AATS and STS

• Second set of Pilot testing

• Refinement of Milestones

• Publications

• Toolbox development

• Contact [email protected] to volunteer

your interest in second pilot ….

Page 59: ACGME Milestones Project Update

QUESTIONS????????