correlation of milestones ratings by objective … of milestones ratings by objective structured...
Post on 19-Apr-2018
218 Views
Preview:
TRANSCRIPT
Correlation of Milestones Ratings by Objective Structured Clinical Examination, Rotation Evaluations, and Resident
Self-Evaluation
Aisha B Davis MD1, Dewesh Agrawal MD1, Kathleen Wortmann3, Ashraf Harahsheh MD1, Jerri Curtis MD2, Cara Lichtenstein MD MPH1, Mary Ottolini MD
MPH1, Joseph Lopreiato MD MPH3
1Children's National Health System, 2Walter Reed National Military Medical
Center, 3Uniformed Services University of the Health Sciences
Association of Pediatric Program Directors 2015 Annual Spring Meeting
Orlando, FL
Background – The Pediatric Milestones
3
ACGME requires semi-annual reporting of each resident’s milestones for 21 of the pediatric subcompetencies Milestone attainment is often assessed using clinical rotation evaluations In a busy clinical setting there may be barriers to accumulating sufficient direct observation data to reliably assess residents’ progress
Background – Objective Structured Clinical Exam
4
Clinical assessment that has been shown to correlate with other traditional performance measures An OSCE offers advantages for assessment of milestones • Increased opportunities for observation • Standardized patient encounters • Ability to design encounters to evaluate specific
subcompetencies In 2013 we implemented an annual intern OSCE designed specifically to assess milestones
Objective
5
To compare milestone attainment of end-of-year interns measured by an OSCE to milestone attainment measured by clinical rotation evaluations and resident self-assessment
Design – The OSCE
6
Setting Occurred over 5 hours in May 2014 Included 5 standardized scenarios designed to emphasize milestones Utilized standardized patients, parents, nurses, and/ or mannequins
Assessment Milestone sets were used directly as rating tools Assessed 8 subcompetencies from semi-annual reporting list
Rater Training All faculty - milestones grand rounds and faculty development OSCE faculty - 45 minute standard setting session using recordings
Design – Clinical Rotation Evaluations
7
Setting Included all available end-of-rotation evaluations in Med Hub Rotations January through June 2014 Included inpatient and outpatient rotations
Assessment Each end-of-rotation evaluation utilized milestones in 2 ways: Directly - subcompetencies were rated using milestones as anchors Indirectly - milestones were mapped to rotation objective ratings
Rater Training All faculty participated in grand rounds and faculty development
Design – Resident Self-Assessment
8
Setting Occurred in May or June 2014
Assessment Med Hub Each rated all 21 subcompetencies using milestones as anchors
Rater Training Grand Rounds Noon conferences
Methods
9
Included subcompetencies based on availability: PC1, PC4, PC5, ICS1, ICS2, SBP1, SBP3,Prof1 Calculated each intern’s mean milestone rating for each subcompetency for the OSCE and rotation evaluations Self-assessment provided 1 rating per subcompetency Calculated Pearson Correlation with significance assigned at p <0.05
Results – Mean Rating by Assessment Type
12
Mean OSCE ratings were notably lower than rotation evaluation and self-assessment ratings
Results – Subcompetency Ratings per Intern by Assessment Method
13
Sub competency
OSCE Rotation (median)
Self-Assessment
PC 1 4 23.5 1 PC 4 2 24.5 1 PC 5 2 23.5 1 ICS 1 2 11.5 1 ICS 2 5 9 1 SBP 1 1 7.5 1 SBP 3 1 8 1 Prof 1 2 4 1
Discussion - Summary
14
OSCE milestone ratings did not correlate positively or negatively with rotation evaluations or self-assessments OSCE milestone ratings were notably lower than rotation evaluations and self-assessments Lack of correlation could be due to at least 2 factors: • OSCE controls for many variables so it may be measuring
different behaviors than clinical encounters. The consistency of measurements within assessment methods suggest this explanation.
• OSCE allows for direct observation while rotation evaluations are contributed to by inference from verbal communication and documentation
Discussion – Comparing Assessment Methods
15
OSCE
Timely faculty
development Real-time ratings Controlled stations Designed for
milestone assessment
Observe full encounters
Less assessment opportunities
Rotation Evaluations
Longitudinal More data points Encounters are
unpredictable Accuracy of
mapped objectives Delayed ratings
Self-Assessment
Most longitudinal
viewpoint May be less
familiar with milestone language
May not be as objective
Discussion – Next Steps
16
Direct comparison of faculty and resident milestone ratings during OSCE and clinical encounters Qualitative evaluation of faculty experience using milestones
Acknowledgements National Capital Consortium Uniformed Services University Medical Simulation Center Amy Flanagan Andrea Creel Children’s National Health System Simulation Learning Center
Background – The Pediatric Milestones
19
Patient Care 1 Gather essential and accurate information about the patient.
Design - OSCE Blueprint
21
PC ICS Prof PBLI 1 4 5 6 7 9 10 1 2 1 9
SP - Abdominal pain √ √ √ √ √ Parent call - URI √ √ √ √ √ √ SP – Asthma √ √ √ √ √ √ √ √ SP - Well Baby √ √ √ √ √ √ √ SP - Syncope √ √ √ √ √ √
PC – Patient care ICS – Interpersonal and communication skills Prof – Professionalism PBLI – Practice based learning and improvement SP – Standardized patient URI – Upper respiratory infection
References Nasca TJ, Philibert I, Brigham T, Flynn TC. The next GME accreditation system—rationale and benefits. NEJM 2012 Mar 15;366(11):1051-6 Carraccio C, Englander R. The objective structured clinical examination – A step in the direction of competency-based evaluation. Arch Pediatr Adolesc Med. 2000;154:736-741 Pediatrics Milestone Project Working Group. The Pediatrics Milestone Project. Available at: https://www.abp.org/abpwebsite/publicat/milestones.pdf. Accessed March 1, 2014
Results – Mean Milestone Attainment
23
1
2
3
4
5
PC 1 PC 4 PC 5 PC 6 PC 7 PC 9 PC 10 PBLI 9 ICS 1 ICS 2 Prof 1
Competency
Mile
ston
e Ra
ting Physical
exam 2.2 ± 0.6
Humanism 3.0 ± 0.8
Design – Competencies
24
PC 1 Gather essential and accurate information about the patient.
PC 4 Interview patients and families about the particulars of the medical condition for which they seek care, with specific attention to behavioral, psychosocial, environmental, and family unit correlates of disease.
PC 5 Perform complete and accurate physical examinations.
PC 6 Make informed diagnostic and therapeutic decisions that result in optimal clinical judgment.
PC 7 Develop and carry out management plans.
PC 9 Counsel patients and families.
PC 10 Provide effective health maintenance and anticipatory guidance.
ICS 1 Communicate effectively with patients, families, and the public, as appropriate, across a broad range of socioeconomic and cultural backgrounds.
ICS 2 Demonstrate the insight and understanding into emotion and human response to emotion that allow one to appropriately develop and manage human interactions.
Prof 1 Demonstrate humanism, compassion, integrity, and respect for others based on the characteristics of an empathetic practitioner.
PBLI 9 Participate in the education of patients, families, students, residents, and other health professionals.
Design – The OSCE
25
Reliability & Validity • Recruited OSCE planning team with varied areas of expertise • Focused on observable clinical skills rather than written items • 4 hours • Used a checklist format for SP rating • Used original milestones language in the faculty rating tool • Performed standard setting for faculty rating • Assessed 8 of 11 competencies across multiple cases
Discussion - Limitations
26
• Observation of a single encounter to assess milestone
• OSCEs are not real patients but performance correlates to other performance evaluations
top related