assessing clinical judgment using the script concordance test: the importance of using...
TRANSCRIPT
![Page 1: Assessing clinical judgment using the script concordance test: The importance of using specialty-specific experts to develop the scoring key Petrucci AM](https://reader030.vdocuments.site/reader030/viewer/2022032606/56649e9f5503460f94ba1207/html5/thumbnails/1.jpg)
Assessing clinical judgment using the script concordance test: The
importance of using specialty-specific experts to develop the scoring key
Petrucci AM. MD1, Nouh T. MD1, Boutros M. MD1, Gagnon R. PhD3, Meterissian S. MD1, 2
Department of Surgery, McGill University 1
Center for Medical Education, McGill University 2
Faculty of Medicine, University of Montreal 3
Association for Surgical Education
Paper Session 1
March 22, 2012
![Page 2: Assessing clinical judgment using the script concordance test: The importance of using specialty-specific experts to develop the scoring key Petrucci AM](https://reader030.vdocuments.site/reader030/viewer/2022032606/56649e9f5503460f94ba1207/html5/thumbnails/2.jpg)
Disclosures
The authors have no disclosures to declare
![Page 3: Assessing clinical judgment using the script concordance test: The importance of using specialty-specific experts to develop the scoring key Petrucci AM](https://reader030.vdocuments.site/reader030/viewer/2022032606/56649e9f5503460f94ba1207/html5/thumbnails/3.jpg)
![Page 4: Assessing clinical judgment using the script concordance test: The importance of using specialty-specific experts to develop the scoring key Petrucci AM](https://reader030.vdocuments.site/reader030/viewer/2022032606/56649e9f5503460f94ba1207/html5/thumbnails/4.jpg)
![Page 5: Assessing clinical judgment using the script concordance test: The importance of using specialty-specific experts to develop the scoring key Petrucci AM](https://reader030.vdocuments.site/reader030/viewer/2022032606/56649e9f5503460f94ba1207/html5/thumbnails/5.jpg)
Current Assessment Tools
• MCQ
• Rich context MCQ
• Orals
• Short answer questions
• Long answer questions
The perfect assessment tool?
![Page 6: Assessing clinical judgment using the script concordance test: The importance of using specialty-specific experts to develop the scoring key Petrucci AM](https://reader030.vdocuments.site/reader030/viewer/2022032606/56649e9f5503460f94ba1207/html5/thumbnails/6.jpg)
Graduating general surgery residents
Multiple choice examination Oral examination+
![Page 7: Assessing clinical judgment using the script concordance test: The importance of using specialty-specific experts to develop the scoring key Petrucci AM](https://reader030.vdocuments.site/reader030/viewer/2022032606/56649e9f5503460f94ba1207/html5/thumbnails/7.jpg)
Institute of Applied Research in Health Sciences Educationhttp://www.script.md/portal/en/script_en.html
What is Reasoning?
![Page 8: Assessing clinical judgment using the script concordance test: The importance of using specialty-specific experts to develop the scoring key Petrucci AM](https://reader030.vdocuments.site/reader030/viewer/2022032606/56649e9f5503460f94ba1207/html5/thumbnails/8.jpg)
SCT: from memorizing to reasoning
• New tool intended to assess mental processes in medical practice
• Can be used to evaluate a trainee’s approach to ill-defined problems including those encountered by expert surgeons
![Page 9: Assessing clinical judgment using the script concordance test: The importance of using specialty-specific experts to develop the scoring key Petrucci AM](https://reader030.vdocuments.site/reader030/viewer/2022032606/56649e9f5503460f94ba1207/html5/thumbnails/9.jpg)
Example
-2 Contra-indicated or almost totally-1 Not useful or even detrimental 0 Nor less nor more useful+1 Useful+2 Necessary or absolutely necessary
![Page 10: Assessing clinical judgment using the script concordance test: The importance of using specialty-specific experts to develop the scoring key Petrucci AM](https://reader030.vdocuments.site/reader030/viewer/2022032606/56649e9f5503460f94ba1207/html5/thumbnails/10.jpg)
The Scoring Grid (Modal Experts’ Choice)
-2 -1 0 +1 + 2
Number of experts choosing answer 0 0 5 4 1
Score 0 0 5/10 4/10 1/10
Transformed score 0 0 5/5 4/5 1/5
Credit per item 0 0 1 0.8 0.2
Nouh, T., et al. (2012). The script concordance test as a measure of clinicalreasoning: A national validation study. The American Journal of Surgery, In press
![Page 11: Assessing clinical judgment using the script concordance test: The importance of using specialty-specific experts to develop the scoring key Petrucci AM](https://reader030.vdocuments.site/reader030/viewer/2022032606/56649e9f5503460f94ba1207/html5/thumbnails/11.jpg)
What have we done in the past?
![Page 12: Assessing clinical judgment using the script concordance test: The importance of using specialty-specific experts to develop the scoring key Petrucci AM](https://reader030.vdocuments.site/reader030/viewer/2022032606/56649e9f5503460f94ba1207/html5/thumbnails/12.jpg)
Preliminary Study:McGill University
n= 36
Meterissian, S. et al. (2007). Is the script concordance test a valid instrument for assessment ofintraoperative decision-making skills? The American Journal of Surgery, 193, 248-251.
![Page 13: Assessing clinical judgment using the script concordance test: The importance of using specialty-specific experts to develop the scoring key Petrucci AM](https://reader030.vdocuments.site/reader030/viewer/2022032606/56649e9f5503460f94ba1207/html5/thumbnails/13.jpg)
Pan Canadian Studyn= 202
Nouh, T., et al. (2012). The script concordance test as a measure of clinicalreasoning: A national validation study. The American Journal of Surgery, In press
![Page 14: Assessing clinical judgment using the script concordance test: The importance of using specialty-specific experts to develop the scoring key Petrucci AM](https://reader030.vdocuments.site/reader030/viewer/2022032606/56649e9f5503460f94ba1207/html5/thumbnails/14.jpg)
![Page 15: Assessing clinical judgment using the script concordance test: The importance of using specialty-specific experts to develop the scoring key Petrucci AM](https://reader030.vdocuments.site/reader030/viewer/2022032606/56649e9f5503460f94ba1207/html5/thumbnails/15.jpg)
Purpose
To determine whether using a specialty-specific scoring key would result in a progression of scores from R1 to R5, with the R5 residents having the highest scores and therefore improving the validity of the scoring key
![Page 16: Assessing clinical judgment using the script concordance test: The importance of using specialty-specific experts to develop the scoring key Petrucci AM](https://reader030.vdocuments.site/reader030/viewer/2022032606/56649e9f5503460f94ba1207/html5/thumbnails/16.jpg)
Methods
![Page 17: Assessing clinical judgment using the script concordance test: The importance of using specialty-specific experts to develop the scoring key Petrucci AM](https://reader030.vdocuments.site/reader030/viewer/2022032606/56649e9f5503460f94ba1207/html5/thumbnails/17.jpg)
Participants
n= 25McGill
n= 202(Nouh, T., et al, 2012)
n tot= 227
![Page 18: Assessing clinical judgment using the script concordance test: The importance of using specialty-specific experts to develop the scoring key Petrucci AM](https://reader030.vdocuments.site/reader030/viewer/2022032606/56649e9f5503460f94ba1207/html5/thumbnails/18.jpg)
Specialty Specific Scoring Key
Colorectal
Hepatobiliary
Endocrine
Surgical Oncology
Thoracic
Trauma/ ACS
EXPERTS
![Page 19: Assessing clinical judgment using the script concordance test: The importance of using specialty-specific experts to develop the scoring key Petrucci AM](https://reader030.vdocuments.site/reader030/viewer/2022032606/56649e9f5503460f94ba1207/html5/thumbnails/19.jpg)
Colorectal 1 1 1 1 1
Hepatobiliary 1 1 1 1 1
Endocrine1 1 1 1 1
Surgical Oncology 1 1 1 1 1
Thoracic 1 1 1 1 1
Trauma/ ACS1 1 1 1 1
EX
PE
RT
2
EX
PE
RT
3
EX
PE
RT
4
EX
PE
RT
5
EX
PE
RT
1
Creation of amalgamated expert exams
![Page 20: Assessing clinical judgment using the script concordance test: The importance of using specialty-specific experts to develop the scoring key Petrucci AM](https://reader030.vdocuments.site/reader030/viewer/2022032606/56649e9f5503460f94ba1207/html5/thumbnails/20.jpg)
• Reliability: Cronbach alpha coefficient
• Analysis of item-to-total item correlation used to select the best items for the final analysis
• Construct validity tested with a one-way ANOVA with post-hoc comparisons test and planned contrasts
• All p values at alpha<5% were considered significant
Statistical Analysis
![Page 21: Assessing clinical judgment using the script concordance test: The importance of using specialty-specific experts to develop the scoring key Petrucci AM](https://reader030.vdocuments.site/reader030/viewer/2022032606/56649e9f5503460f94ba1207/html5/thumbnails/21.jpg)
RESULTS
![Page 22: Assessing clinical judgment using the script concordance test: The importance of using specialty-specific experts to develop the scoring key Petrucci AM](https://reader030.vdocuments.site/reader030/viewer/2022032606/56649e9f5503460f94ba1207/html5/thumbnails/22.jpg)
Final exam100-question testCronbach α = 0.81
153-question test Cronbach α = 0.67
Elimination of items with a negative item-to-total item correlation
![Page 23: Assessing clinical judgment using the script concordance test: The importance of using specialty-specific experts to develop the scoring key Petrucci AM](https://reader030.vdocuments.site/reader030/viewer/2022032606/56649e9f5503460f94ba1207/html5/thumbnails/23.jpg)
Results
?
Scores rising with increasing level of residency: p< 0.001
![Page 24: Assessing clinical judgment using the script concordance test: The importance of using specialty-specific experts to develop the scoring key Petrucci AM](https://reader030.vdocuments.site/reader030/viewer/2022032606/56649e9f5503460f94ba1207/html5/thumbnails/24.jpg)
Number of expert panel members matters!
Gagnon et al. Medical Education, 2005:
• The mean of residents’ scores increased with increasing panel size.
“…the number of points earned on an SCT is influenced by the number of members used in the panel”.
![Page 25: Assessing clinical judgment using the script concordance test: The importance of using specialty-specific experts to develop the scoring key Petrucci AM](https://reader030.vdocuments.site/reader030/viewer/2022032606/56649e9f5503460f94ba1207/html5/thumbnails/25.jpg)
Mean Score Graph
Scores rising with increasing level of residency: p< 0.001
![Page 26: Assessing clinical judgment using the script concordance test: The importance of using specialty-specific experts to develop the scoring key Petrucci AM](https://reader030.vdocuments.site/reader030/viewer/2022032606/56649e9f5503460f94ba1207/html5/thumbnails/26.jpg)
Juniors vs Seniors
Total N
R1/ R2 45.1 (7.6) 109
R3/ R4/ R5 50.4 (8.0) 118
The average score of junior residents proved to be significantly lower than the average score of senior residents: p< 0.001
![Page 27: Assessing clinical judgment using the script concordance test: The importance of using specialty-specific experts to develop the scoring key Petrucci AM](https://reader030.vdocuments.site/reader030/viewer/2022032606/56649e9f5503460f94ba1207/html5/thumbnails/27.jpg)
Discussion
• The results of this study suggest that the scoring key should be developed through the use of specialty- specific experts
• This may be due to the increasing subspecialization of General Surgery
• This has important implications in the application of the SCT on a wider level
![Page 28: Assessing clinical judgment using the script concordance test: The importance of using specialty-specific experts to develop the scoring key Petrucci AM](https://reader030.vdocuments.site/reader030/viewer/2022032606/56649e9f5503460f94ba1207/html5/thumbnails/28.jpg)
Limitations• We used only 5 amalgamated general
surgery experts for each surgical discipline instead of the recommended >10
Gagnon et al. Medical Education 2005
• The resident pool consisted mostly of the same residents from our last study with the addition of 25 new residents from our institution
![Page 29: Assessing clinical judgment using the script concordance test: The importance of using specialty-specific experts to develop the scoring key Petrucci AM](https://reader030.vdocuments.site/reader030/viewer/2022032606/56649e9f5503460f94ba1207/html5/thumbnails/29.jpg)
Future implications…
National wide study using the specialty-specific expert scoring key.
The SCT might be ready for a national in-training formative examination.
![Page 30: Assessing clinical judgment using the script concordance test: The importance of using specialty-specific experts to develop the scoring key Petrucci AM](https://reader030.vdocuments.site/reader030/viewer/2022032606/56649e9f5503460f94ba1207/html5/thumbnails/30.jpg)
Acknowledgements
• Dr. Sarkis Meterissian• Dr. Robert Gagnon• Dr. Thamer Nouh• Dr. Marylise Boutros
![Page 31: Assessing clinical judgment using the script concordance test: The importance of using specialty-specific experts to develop the scoring key Petrucci AM](https://reader030.vdocuments.site/reader030/viewer/2022032606/56649e9f5503460f94ba1207/html5/thumbnails/31.jpg)
Thank you