![Page 1: 360 Degree Evaluation Craig McClure, MD May 15, 2003 Educational Outcomes Service Group](https://reader030.vdocuments.site/reader030/viewer/2022032600/56649db55503460f94aa5d2d/html5/thumbnails/1.jpg)
360 Degree Evaluation
Craig McClure, MDMay 15, 2003
Educational Outcomes Service Group
![Page 2: 360 Degree Evaluation Craig McClure, MD May 15, 2003 Educational Outcomes Service Group](https://reader030.vdocuments.site/reader030/viewer/2022032600/56649db55503460f94aa5d2d/html5/thumbnails/2.jpg)
Description
• Use of rating forms to report frequency of observed behavior
• Multiple people in contact with resident act as evaluators
• Often survey type form• Ratings summarized by topic• Include goal-setting
![Page 3: 360 Degree Evaluation Craig McClure, MD May 15, 2003 Educational Outcomes Service Group](https://reader030.vdocuments.site/reader030/viewer/2022032600/56649db55503460f94aa5d2d/html5/thumbnails/3.jpg)
Background
• Human resources in business• ACGME found no published reports
of use in GME
![Page 4: 360 Degree Evaluation Craig McClure, MD May 15, 2003 Educational Outcomes Service Group](https://reader030.vdocuments.site/reader030/viewer/2022032600/56649db55503460f94aa5d2d/html5/thumbnails/4.jpg)
Use for “Soft” Areas
• More accurate for formative than summative feedback
• Interpersonal & communication• Professional behavior• Limited
– Patient care– Systems-based practice
![Page 5: 360 Degree Evaluation Craig McClure, MD May 15, 2003 Educational Outcomes Service Group](https://reader030.vdocuments.site/reader030/viewer/2022032600/56649db55503460f94aa5d2d/html5/thumbnails/5.jpg)
Decision to Utilize
• Accepted and used by residents, faculty, staff?
• Develop or purchase?• Cost? • Who are the raters? • How will the tool be used?
![Page 6: 360 Degree Evaluation Craig McClure, MD May 15, 2003 Educational Outcomes Service Group](https://reader030.vdocuments.site/reader030/viewer/2022032600/56649db55503460f94aa5d2d/html5/thumbnails/6.jpg)
Decision to Utilize (2)
• To whom is the information available?
• What core competencies will be evaluated with this tool?
• How nurture trust the process remains confidential?
• Platform of evaluation
![Page 7: 360 Degree Evaluation Craig McClure, MD May 15, 2003 Educational Outcomes Service Group](https://reader030.vdocuments.site/reader030/viewer/2022032600/56649db55503460f94aa5d2d/html5/thumbnails/7.jpg)
Acceptance
• Will all potential evaluators fully participate?
• Will raters be fair & honest?• Will residents accept the feedback
from non-faculty?
![Page 8: 360 Degree Evaluation Craig McClure, MD May 15, 2003 Educational Outcomes Service Group](https://reader030.vdocuments.site/reader030/viewer/2022032600/56649db55503460f94aa5d2d/html5/thumbnails/8.jpg)
Develop or Purchase
• Development permits tailoring• Development time may be
considerable• Purchasing gives a ready-made
product• Purchasing: computer based
![Page 9: 360 Degree Evaluation Craig McClure, MD May 15, 2003 Educational Outcomes Service Group](https://reader030.vdocuments.site/reader030/viewer/2022032600/56649db55503460f94aa5d2d/html5/thumbnails/9.jpg)
Developing
• Expert in educational testing• Programming expertise• Pilot period
![Page 10: 360 Degree Evaluation Craig McClure, MD May 15, 2003 Educational Outcomes Service Group](https://reader030.vdocuments.site/reader030/viewer/2022032600/56649db55503460f94aa5d2d/html5/thumbnails/10.jpg)
Purchase
• Items measured appropriate?• Does it perform as claimed?• Inter-rater reliability?• Degree of support and ability to
customize
![Page 11: 360 Degree Evaluation Craig McClure, MD May 15, 2003 Educational Outcomes Service Group](https://reader030.vdocuments.site/reader030/viewer/2022032600/56649db55503460f94aa5d2d/html5/thumbnails/11.jpg)
Cost
• If purchasing, monetary cost• If developing, personnel support• Data management system• Personnel time to complete forms• Annual development plan
![Page 12: 360 Degree Evaluation Craig McClure, MD May 15, 2003 Educational Outcomes Service Group](https://reader030.vdocuments.site/reader030/viewer/2022032600/56649db55503460f94aa5d2d/html5/thumbnails/12.jpg)
Cost (2)
• Addressing EEOC/grievance complaints
• Handling disputes over data• Divisive & counterproductive for
those resistant
![Page 13: 360 Degree Evaluation Craig McClure, MD May 15, 2003 Educational Outcomes Service Group](https://reader030.vdocuments.site/reader030/viewer/2022032600/56649db55503460f94aa5d2d/html5/thumbnails/13.jpg)
Personnel Evaluation Time
• 5 to 10 nurse evaluators per resident to give reproducible results
• More for faculty• More for patients
![Page 14: 360 Degree Evaluation Craig McClure, MD May 15, 2003 Educational Outcomes Service Group](https://reader030.vdocuments.site/reader030/viewer/2022032600/56649db55503460f94aa5d2d/html5/thumbnails/14.jpg)
Identify Raters
• Patients (how explain process)• Nursing staff• Clerical staff members• Physician faculty members• Non-physician faculty members• Residents
![Page 15: 360 Degree Evaluation Craig McClure, MD May 15, 2003 Educational Outcomes Service Group](https://reader030.vdocuments.site/reader030/viewer/2022032600/56649db55503460f94aa5d2d/html5/thumbnails/15.jpg)
Identify Raters (2)
• Medical students• Allied Health Personnel• Self-assessment
![Page 16: 360 Degree Evaluation Craig McClure, MD May 15, 2003 Educational Outcomes Service Group](https://reader030.vdocuments.site/reader030/viewer/2022032600/56649db55503460f94aa5d2d/html5/thumbnails/16.jpg)
Patients as Raters
• Literacy• Language• Culture (medical and otherwise)• Personality
![Page 17: 360 Degree Evaluation Craig McClure, MD May 15, 2003 Educational Outcomes Service Group](https://reader030.vdocuments.site/reader030/viewer/2022032600/56649db55503460f94aa5d2d/html5/thumbnails/17.jpg)
Intended Utility
• Intervals: monthly, quarterly, yearly
• Summative versus formative• To support high stakes decisions?
![Page 18: 360 Degree Evaluation Craig McClure, MD May 15, 2003 Educational Outcomes Service Group](https://reader030.vdocuments.site/reader030/viewer/2022032600/56649db55503460f94aa5d2d/html5/thumbnails/18.jpg)
Access to Information
• Resident• Advisor• Program Director
![Page 19: 360 Degree Evaluation Craig McClure, MD May 15, 2003 Educational Outcomes Service Group](https://reader030.vdocuments.site/reader030/viewer/2022032600/56649db55503460f94aa5d2d/html5/thumbnails/19.jpg)
Confidentiality & Trust
• Raters require anonymity• Residents require confidentiality• Both need the process to be
positive & constructive• Prior history conditions
expectations• Education to process aids current
participation
![Page 20: 360 Degree Evaluation Craig McClure, MD May 15, 2003 Educational Outcomes Service Group](https://reader030.vdocuments.site/reader030/viewer/2022032600/56649db55503460f94aa5d2d/html5/thumbnails/20.jpg)
Platform of Evaluation
• PDA• Paper• Computer
![Page 21: 360 Degree Evaluation Craig McClure, MD May 15, 2003 Educational Outcomes Service Group](https://reader030.vdocuments.site/reader030/viewer/2022032600/56649db55503460f94aa5d2d/html5/thumbnails/21.jpg)
Challenges
• Securing appropriate instruments for variety of evaluators
• Managing data successfully
![Page 22: 360 Degree Evaluation Craig McClure, MD May 15, 2003 Educational Outcomes Service Group](https://reader030.vdocuments.site/reader030/viewer/2022032600/56649db55503460f94aa5d2d/html5/thumbnails/22.jpg)
Advantages
• Electronic database for documentation
• Ease of access for raters• Rapid turnaround for feedback• “Gap” analysis (self perception
versus image of others)
![Page 23: 360 Degree Evaluation Craig McClure, MD May 15, 2003 Educational Outcomes Service Group](https://reader030.vdocuments.site/reader030/viewer/2022032600/56649db55503460f94aa5d2d/html5/thumbnails/23.jpg)
Disadvantages
• Hardware/software costs• Lack of validation in GME• Potential information overload• Selection bias• Discoverability• Potential for invalid feedback
![Page 24: 360 Degree Evaluation Craig McClure, MD May 15, 2003 Educational Outcomes Service Group](https://reader030.vdocuments.site/reader030/viewer/2022032600/56649db55503460f94aa5d2d/html5/thumbnails/24.jpg)
References
• Assessment of Communication and Interpersonal Skills Competencies, C.C. Hobgood, et.al. Academic Emergency Medicine 2002;9: 1257-69
• ACGME/ABMS Joint Initiative Toolbox of Assessment Methods, September 2000
![Page 25: 360 Degree Evaluation Craig McClure, MD May 15, 2003 Educational Outcomes Service Group](https://reader030.vdocuments.site/reader030/viewer/2022032600/56649db55503460f94aa5d2d/html5/thumbnails/25.jpg)
References (2)
• 360-degree Feedback, K.G. Rodgers,et.al. Academic Emergency Medicine 2002;9:1300-1304
• Letter from ADFM listserv, Goldsmith to Kikano