Correlation of Milestones Ratings by Objective Structured Clinical Examination, Rotation Evaluations, and Resident

Self-Evaluation

Aisha B Davis MD1, Dewesh Agrawal MD1, Kathleen Wortmann3, Ashraf Harahsheh MD1, Jerri Curtis MD2, Cara Lichtenstein MD MPH1, Mary Ottolini MD

MPH1, Joseph Lopreiato MD MPH3

1Children's National Health System, 2Walter Reed National Military Medical

Center, 3Uniformed Services University of the Health Sciences

Association of Pediatric Program Directors 2015 Annual Spring Meeting

Orlando, FL

Disclosures

2

No conflicts of interest or financial relationships to disclose

Background – The Pediatric Milestones

3

ACGME requires semi-annual reporting of each resident’s milestones for 21 of the pediatric subcompetencies Milestone attainment is often assessed using clinical rotation evaluations In a busy clinical setting there may be barriers to accumulating sufficient direct observation data to reliably assess residents’ progress

Background – Objective Structured Clinical Exam

4

Clinical assessment that has been shown to correlate with other traditional performance measures An OSCE offers advantages for assessment of milestones • Increased opportunities for observation • Standardized patient encounters • Ability to design encounters to evaluate specific

subcompetencies In 2013 we implemented an annual intern OSCE designed specifically to assess milestones

Objective

5

To compare milestone attainment of end-of-year interns measured by an OSCE to milestone attainment measured by clinical rotation evaluations and resident self-assessment

Design – The OSCE

6

Setting Occurred over 5 hours in May 2014 Included 5 standardized scenarios designed to emphasize milestones Utilized standardized patients, parents, nurses, and/ or mannequins

Assessment Milestone sets were used directly as rating tools Assessed 8 subcompetencies from semi-annual reporting list

Rater Training All faculty - milestones grand rounds and faculty development OSCE faculty - 45 minute standard setting session using recordings

Design – Clinical Rotation Evaluations

7

Setting Included all available end-of-rotation evaluations in Med Hub Rotations January through June 2014 Included inpatient and outpatient rotations

Assessment Each end-of-rotation evaluation utilized milestones in 2 ways: Directly - subcompetencies were rated using milestones as anchors Indirectly - milestones were mapped to rotation objective ratings

Rater Training All faculty participated in grand rounds and faculty development

Design – Resident Self-Assessment

8

Setting Occurred in May or June 2014

Assessment Med Hub Each rated all 21 subcompetencies using milestones as anchors

Rater Training Grand Rounds Noon conferences

Methods

9

Included subcompetencies based on availability: PC1, PC4, PC5, ICS1, ICS2, SBP1, SBP3,Prof1 Calculated each intern’s mean milestone rating for each subcompetency for the OSCE and rotation evaluations Self-assessment provided 1 rating per subcompetency Calculated Pearson Correlation with significance assigned at p <0.05

Results – Correlation

10

Correlations were not statistically significant

Results – Weakly Correlated Items

11

Results – Mean Rating by Assessment Type

12

Mean OSCE ratings were notably lower than rotation evaluation and self-assessment ratings

Results – Subcompetency Ratings per Intern by Assessment Method

13

Sub competency

OSCE Rotation (median)

Self-Assessment

PC 1 4 23.5 1 PC 4 2 24.5 1 PC 5 2 23.5 1 ICS 1 2 11.5 1 ICS 2 5 9 1 SBP 1 1 7.5 1 SBP 3 1 8 1 Prof 1 2 4 1

Discussion - Summary

14

OSCE milestone ratings did not correlate positively or negatively with rotation evaluations or self-assessments OSCE milestone ratings were notably lower than rotation evaluations and self-assessments Lack of correlation could be due to at least 2 factors: • OSCE controls for many variables so it may be measuring

different behaviors than clinical encounters. The consistency of measurements within assessment methods suggest this explanation.

• OSCE allows for direct observation while rotation evaluations are contributed to by inference from verbal communication and documentation

Discussion – Comparing Assessment Methods

15

OSCE

Timely faculty

development Real-time ratings Controlled stations Designed for

milestone assessment

Observe full encounters

Less assessment opportunities

Rotation Evaluations

Longitudinal More data points Encounters are

unpredictable Accuracy of

mapped objectives Delayed ratings

Self-Assessment

Most longitudinal

viewpoint May be less

familiar with milestone language

May not be as objective

Discussion – Next Steps

16

Direct comparison of faculty and resident milestone ratings during OSCE and clinical encounters Qualitative evaluation of faculty experience using milestones

Acknowledgements National Capital Consortium Uniformed Services University Medical Simulation Center Amy Flanagan Andrea Creel Children’s National Health System Simulation Learning Center

Background – The Pediatric Milestones

19

Patient Care 1 Gather essential and accurate information about the patient.

Results

20

Design - OSCE Blueprint

21

PC ICS Prof PBLI 1 4 5 6 7 9 10 1 2 1 9

SP - Abdominal pain √ √ √ √ √ Parent call - URI √ √ √ √ √ √ SP – Asthma √ √ √ √ √ √ √ √ SP - Well Baby √ √ √ √ √ √ √ SP - Syncope √ √ √ √ √ √

PC – Patient care ICS – Interpersonal and communication skills Prof – Professionalism PBLI – Practice based learning and improvement SP – Standardized patient URI – Upper respiratory infection

References Nasca TJ, Philibert I, Brigham T, Flynn TC. The next GME accreditation system—rationale and benefits. NEJM 2012 Mar 15;366(11):1051-6 Carraccio C, Englander R. The objective structured clinical examination – A step in the direction of competency-based evaluation. Arch Pediatr Adolesc Med. 2000;154:736-741 Pediatrics Milestone Project Working Group. The Pediatrics Milestone Project. Available at: https://www.abp.org/abpwebsite/publicat/milestones.pdf. Accessed March 1, 2014

Results – Mean Milestone Attainment

23

1

2

3

4

5

PC 1 PC 4 PC 5 PC 6 PC 7 PC 9 PC 10 PBLI 9 ICS 1 ICS 2 Prof 1

Competency

Mile

ston

e Ra

ting Physical

exam 2.2 ± 0.6

Humanism 3.0 ± 0.8

Design – Competencies

24

PC 1 Gather essential and accurate information about the patient.

PC 4 Interview patients and families about the particulars of the medical condition for which they seek care, with specific attention to behavioral, psychosocial, environmental, and family unit correlates of disease.

PC 5 Perform complete and accurate physical examinations.

PC 6 Make informed diagnostic and therapeutic decisions that result in optimal clinical judgment.

PC 7 Develop and carry out management plans.

PC 9 Counsel patients and families.

PC 10 Provide effective health maintenance and anticipatory guidance.

ICS 1 Communicate effectively with patients, families, and the public, as appropriate, across a broad range of socioeconomic and cultural backgrounds.

ICS 2 Demonstrate the insight and understanding into emotion and human response to emotion that allow one to appropriately develop and manage human interactions.

Prof 1 Demonstrate humanism, compassion, integrity, and respect for others based on the characteristics of an empathetic practitioner.

PBLI 9 Participate in the education of patients, families, students, residents, and other health professionals.

Design – The OSCE

25

Reliability & Validity • Recruited OSCE planning team with varied areas of expertise • Focused on observable clinical skills rather than written items • 4 hours • Used a checklist format for SP rating • Used original milestones language in the faculty rating tool • Performed standard setting for faculty rating • Assessed 8 of 11 competencies across multiple cases

Discussion - Limitations

26

• Observation of a single encounter to assess milestone

• OSCEs are not real patients but performance correlates to other performance evaluations

Design - The OSCE

27

SP & parent - abdominal pain

SP & parent - asthma

SP & parent - well infant

SP & parent - recent syncopal episode

Telephone Triage - Parent of febrile infant

Procedure - Intraosseous line insertion

Procedure - Intravenous line insertion

Handoff of three previous patients