measuring non-technical aspects of surgical clinician development in an otolaryngology residency...
TRANSCRIPT
Measuring Non-technical Aspects of Surgical Clinician Development in an
Otolaryngology Residency Training Program
Jennifer J. Shin, M.D. S.M.Michael J. Cunningham, M.D. Kevin G. Emerick, M.D.Stacey T. Gray, M.D.
Society for University OtolaryngologistsNovember 13, 2015
The Constitution of a Surgeon
It takes five years to learn how to operate and twenty years to learn when and when not to.
-Anonymous
The Constitution of a Surgeon
It takes five years to learn how to operate and twenty years to learn when and when not to.
-Anonymous
I would like to see the day when somebody would be appointed surgeon somewhere who had no hands, for the operative part is the least part of the work.
-Harvey Williams Cushing, Letter to Dr Henry Christian,
1911. Quoted in 'The Best Hope of All', Time
(3 May 1963).
Validated Instruments
…Clinical Outcomes…Evidence-based Practice
Validated Instruments
…Educational Outcomes…Evidence-based Residency
Novice Intermediate Competent
Instrument to measure the development of
clinical practice ability?
Instrument Design
Rapid/real time evaluation
Insight into each trainee’s thought process
Concrete feedback and data compilation
Benefits beyond the learning/measurement itself
Instrument AdministrationMock oral board exam Case criteria:
1) Single patient with a single main diagnostic issue.
2) Case culminates in an intervention, with one resulting complication
3) Enough detail available so that a full history, physical examination, and complement of diagnostic testing may be obtained -- actual radiologic images, lab values, audiometric studies, or other diagnostic test results are available
Instrument Development Iterative process:
22 successive drafts
311 candidate questions
Face validity evaluation: Residency program director
Division chairman
Professionalism director
Resident education curriculum supervisor
Epidemiologist/Instrument validation scientist
Validation phase: Instrument assessment
Inter-rater Reliability• Cohen Kappa 0.66 (SE 0.03)• 72.5% agreement
Internal Consistency• Cronbach alpha >0.87
Responsive to Change
OBJECTIVES
• To utilize the clinical practice instrument (CPI) to measure non-technical diagnostic and management skills during otolaryngology residency training
• To determine whether there is demonstrable change in these skills between PGY-2, 4, and 5 residents
• To evaluate whether results vary according to subspecialty topic or method of administration.
METHODS
• Prospective study of an otolaryngology residency training program
• Institutional review board approved
• n=248 evaluations of 45 otolaryngology resident trainees at regular intervals
• Analysis of variance with nesting and post-estimation pairwise comparisons to evaluate total and domain scores according to training level, subspecialty topic, and method of administration
• Examination preparation and security
PGY-4 & 5
RESULTS
Total scores were significantly different among PGY-levels of training, with lower scores seen in the PGY-2 level compared to the PGY-4 or PGY-5 level (p<0.0001).
PGYLevel
n MeanStandard Deviation
Minimum Maximum25th Percentile
75th Percentile
2 51 44.2 16.3 23 87 32 53
4 101 63.6 12.8 39 94 54 72
5 96 68.8 13.3 34 92 60 78.5
All 248 61.6 16.5 23 94 50 74
RESULTS
Residents scored higher in general otolaryngology than in the subspecialties (ANOVA, p<0.003).
There was a significant difference in between general otolaryngology and pediatric otolaryngology (p<0.0001), and between general otolaryngology and head and neck surgery (post-ANOVA pairwise comparison, p<0.0033)
RESULTS
Administering the examination with an electronic scoring system, rather than a paper-based scoring system, did not affect these results.
The calendar year of administration did not affect these results.
RESULTS
CONCLUSIONS
• Standardized interval evaluation with the CPI demonstrates improvement in qualitative diagnostic and management capabilities as PGY-levels advance.
• Administration of the CPI has been formally incorporated into the Harvard otolaryngology residency curriculum.
• The CPI can potentially be adapted for use in any otolaryngology training program, and is potentially applicable to any surgical specialty.
FUTURE DIRECTIONS
• Short-term educational missions…
• Milestones…
• Electronic case library…
• Program update follow up?
• Benchmarks?
• Other sites?
Thank you