![Page 1: A Novel Approach for Developing Rating Scales for a Practice Ready OSCE](https://reader035.vdocuments.site/reader035/viewer/2022062511/54c34d144a795960188b4608/html5/thumbnails/1.jpg)
A Novel Approach for Developing Rating Scales for a Practice Ready OSCE
Bruce HolmesDivision of Medical Education, Dalhousie University
Saad ChahineMount Saint Vincent University
![Page 2: A Novel Approach for Developing Rating Scales for a Practice Ready OSCE](https://reader035.vdocuments.site/reader035/viewer/2022062511/54c34d144a795960188b4608/html5/thumbnails/2.jpg)
Clinician Assessment for Practice Program (CAPP)
• A program of the College of Physicians and Surgeons of Nova Scotia (CPSNS)
• A 3-part program assessing IMG physicians (residents in Canada, without formal Canadian training) for practice readiness.
![Page 3: A Novel Approach for Developing Rating Scales for a Practice Ready OSCE](https://reader035.vdocuments.site/reader035/viewer/2022062511/54c34d144a795960188b4608/html5/thumbnails/3.jpg)
The CAPP ProgramPart A: Initial assessment• Assessment of competence via OSCE & therapeutics
exam
Part B: 1 year mentorship with a family physician• Defined license for 1 year with performance assessment
Part C: Additional 3 years of defined license untilcertified by The College of Family Physicians
(CCFP)
![Page 4: A Novel Approach for Developing Rating Scales for a Practice Ready OSCE](https://reader035.vdocuments.site/reader035/viewer/2022062511/54c34d144a795960188b4608/html5/thumbnails/4.jpg)
Purpose of the CAPP OSCE
1. To assess the clinical competence of IMG candidates for readiness to practice.
2. To provide feedback on candidates' performance for their continuing professional development.
• The OSCE is not a “pass or fail” examination, but rather one factor in the decision to grant a defined license to practice in Nova Scotia.
![Page 5: A Novel Approach for Developing Rating Scales for a Practice Ready OSCE](https://reader035.vdocuments.site/reader035/viewer/2022062511/54c34d144a795960188b4608/html5/thumbnails/5.jpg)
Examining the Constructs of Rating Form?
• Re-design rating form
• Provide rater guidelines on how to be an examiner
• Two online practice videos for raters– Results submitted– Group data reported to compare with individual rating– Personal feedback available
![Page 6: A Novel Approach for Developing Rating Scales for a Practice Ready OSCE](https://reader035.vdocuments.site/reader035/viewer/2022062511/54c34d144a795960188b4608/html5/thumbnails/6.jpg)
OSCE Competencies
• History Taking• Physical Exam• Communication Skills• Quality of Spoken English• Counselling• Professional Behaviour• Problem Definition & Diagnosis*• Investigation & Management*
• Overall Global*
![Page 7: A Novel Approach for Developing Rating Scales for a Practice Ready OSCE](https://reader035.vdocuments.site/reader035/viewer/2022062511/54c34d144a795960188b4608/html5/thumbnails/7.jpg)
Example of Elements for Competency
![Page 8: A Novel Approach for Developing Rating Scales for a Practice Ready OSCE](https://reader035.vdocuments.site/reader035/viewer/2022062511/54c34d144a795960188b4608/html5/thumbnails/8.jpg)
Purpose of Elements“Guided Conceptualization”
• The elements act as a guide to forming a conceptualization of the competency
• Theory the elements help the examiner in self regulatory process to build an
“awareness and management of one’s own thought” (Kuhn & Dean, 2004)
• Provide “standardized” comments as feedback for candidates and Credentials Committee of College
![Page 9: A Novel Approach for Developing Rating Scales for a Practice Ready OSCE](https://reader035.vdocuments.site/reader035/viewer/2022062511/54c34d144a795960188b4608/html5/thumbnails/9.jpg)
![Page 10: A Novel Approach for Developing Rating Scales for a Practice Ready OSCE](https://reader035.vdocuments.site/reader035/viewer/2022062511/54c34d144a795960188b4608/html5/thumbnails/10.jpg)
![Page 11: A Novel Approach for Developing Rating Scales for a Practice Ready OSCE](https://reader035.vdocuments.site/reader035/viewer/2022062511/54c34d144a795960188b4608/html5/thumbnails/11.jpg)
How are Elements Related to Competencies?
• Reviewed the ratings in 2012 OSCE– 36 candidates
– Total of 432 observations
• Each set of elements correlated with competency– e.g., History; Investigation & Management
• Spearman Correlation between the element ratings and the respective competency rating was conducted one at a time
![Page 12: A Novel Approach for Developing Rating Scales for a Practice Ready OSCE](https://reader035.vdocuments.site/reader035/viewer/2022062511/54c34d144a795960188b4608/html5/thumbnails/12.jpg)
Results• History Taking; Counselling; Investigation &
Management:– High positive correlation for all element ratings
with the respective competency rating (r-values 0.74-0.82)
• Problem Definition and Diagnosis: – High positive correlation (r-values 0.70-0.81) except for
alternate diagnosis (r=0.61)
• Physical Exam – High positive correlation (r-values 0.70-0.75) except for
sensitivity (r=0.61)
![Page 13: A Novel Approach for Developing Rating Scales for a Practice Ready OSCE](https://reader035.vdocuments.site/reader035/viewer/2022062511/54c34d144a795960188b4608/html5/thumbnails/13.jpg)
Future Analysis for Competencies – CFA
History Taking
Breadth
Relevance
Focus
Sequencing
Chronology
• Include data from 2013 & 2014
• Confirmatory Factor Analysis conducted through Structural Equation Modeling
![Page 14: A Novel Approach for Developing Rating Scales for a Practice Ready OSCE](https://reader035.vdocuments.site/reader035/viewer/2022062511/54c34d144a795960188b4608/html5/thumbnails/14.jpg)
Take Home Messages
• Strong positive associations between elements and competency rating– Appears to be helpful to guide examiner ratings– Helpful feedback on competencies
• Follow-up with refinement of elements using examiner feedback
• Follow-up with data from June 2013 & 2014 OSCE