CLINICAL COMPETENCY COMMITTEE

Wendy M. Helkowski, M.D.

Program Director

University of Pittsburgh Medical Center (UPMC)

ACGME COMMON PROGRAM REQUIREMENTS

The program director must appoint the Clinical Competency Committee. (Core)

V.A.1.a) At a minimum the Clinical Competency Committee must be composed of three members of the program faculty. (Core)

The Clinical Competency Committee should: V.A.1.b).(1).(a) review all resident evaluations semi-

annually; (Core) V.A.1.b).(1).(b) prepare and assure the reporting of

Milestones evaluations of each resident semi-annually to ACGME; and, (Core)

V.A.1.b).(1).(c) advise the program director regarding resident progress, including promotion, remediation, and dismissal. (Detail)

PROGRAM BACKGROUND Large program (34 residents) Half enter program as PGY-1 Large faculty(approx 30) 1 PD, 1 Associate PD, 1 Assistant PD Bi-monthly Program Evaluation

Committee (PEC) meetingsAll teaching faculty are invited to the

meeting

CLINICAL COMPETENCY COMMITTEE (CCC) PRE-MILESTONES

9 faculty members (selected by PD) and 3 PDs

Meetings held after Bi-monthly PEC Evaluation of individual resident

competency concerns including remediation and dismissal

Mid-year and end of year progress assessments for all residents

NEW CCC RESPONSIBILITIES Semi-annual review of evaluations for all

34 residents Selection of 19 milestone levels for all

34 residents Total estimated time per resident = 30

minutes Total estimated time for all 34 residents

= 17 hours semi-annually! 12 faculty members spending 2 full days

evaluating milestones twice a year??!!

GOALS Educate the CCC faculty about the

milestones Improve efficiency and accuracy of

milestone level selectionEducate the entire residency faculty about

the milestones Improve efficiency of evaluation review Reduce time commitment for non-PD

faculty Reduce time commitment for PD faculty

EDUCATE THE CCC FACULTY ABOUT THE MILESTONES What are they?

Actual milestones tables emailed to the CCC Selected milestones samples reviewed at PEC meetings CCC members asked to do a pilot where they

completed the set of 19 milestones for one resident using recent evaluations

How will they be used? Submitted to ACGME semi-annually Eventual comparison of program data with national

data What are the CCC responsibilities related to the

milestones? Group evaluation semi-annually to determine an

appropriate milestone level for each resident in the program

IMPROVE EFFICIENCY AND ACCURACY OF MILESTONE LEVEL SELECTION Pilot of CCC milestones assessment 1 year ago

our current evaluations did not provide sufficient data on specific milestones to accurately score a level

Modification of end of rotation evaluations to include milestones language

Modification of semi-annual self evaluations to include milestones language

Repeat pilot using modified evaluations 6 months ago Improved ability to select a milestone level Still challenging for the PGY-1 residents because of

variable evaluation formats

END OF ROTATION EVALUATION

IMPROVE EFFICIENCY AND ACCURACY OF MILESTONE REVIEW

Divided the CCC into 3 subgroups with separate meeting times PGY-1 and PGY-2 (3 faculty and 3 PDs) PGY-3 (3 faculty and 3 PDs) PGY-4 (3 faculty and 3 PDs)

Prior to the meeting, each member is assigned 1-2 residents to review and complete the draft electronic milestone form All evaluations (global, peer, self, multi-source,

patient) SAE scores

Utilization of institution’s electronic evaluation system which includes a milestones program

CCC MILESTONE EVALUATION

CCC MEETINGS Each CCC member presented the

information on his/her assigned residents

CCC determined actual level Level was changed in about 1/3 of the

items based on discussion and personal experiences with the residents

9-10 residents reviewed in 2 hours Total time estimate – about 10 hours for

PDs and 3 hours for faculty semi-annually

OUR LEARNING POINTS

Don’t expect all CCC faculty to come to the first meeting completely prepared

It is helpful to have faculty on the CCC who have worked with the resident during the evaluation periodHowever, if there was one negative

experience by a CCC member and other evaluations are good, it is challenging to have the one CCC member agree to a higher score

OUR LEARNING POINTS Use first CCC milestones assessment as

a learning opportunity for the groupHelpful hints about the electronic evaluation

programDiscuss differences in end-of-rotation

evaluations and semi-annual evaluations of the milestones OUR end of rotation evaluation language leaves

out “across a spectrum of ages and impairments” – so the resident may reach a level 4 on one rotation but the CCC should not assign a 4 until the resident has demonstrated this ability across the spectrum

OUR LEARNING POINTS It is helpful to have the actual

milestones available at the meetings with review of the language and discussion when there is a discrepancy on the various rotation evaluationsHelpful with “halo effect” and when limited

data is available Choose faculty who reliably complete

documentation for the residency program

Consider scheduling CCC meetings early (Nov/May)

Thank you!