wbas: the northern ireland school of surgery experience richard mayes 1 robert gilliland 1, 2...
TRANSCRIPT
![Page 1: WBAs: the Northern Ireland School of Surgery Experience Richard Mayes 1 Robert Gilliland 1, 2 Jeffrey Campbell 1, 2 Helen Holscher 3 Department of Surgery,](https://reader035.vdocuments.site/reader035/viewer/2022062301/56649f1c5503460f94c31dc1/html5/thumbnails/1.jpg)
WBAs: the Northern Ireland School of Surgery
ExperienceRichard Mayes1 Robert Gilliland1, 2
Jeffrey Campbell1, 2 Helen Holscher3
Department of Surgery, Ulster Hospital Dundonald, Northern Ireland1
School of Surgery, Northern Ireland Medical & Dental Training Agency2,Royal College of Surgeons of England3..
The Royal College of Surgeons of England
Regional Representatives Meeting
Wednesday 10th November 2010
![Page 2: WBAs: the Northern Ireland School of Surgery Experience Richard Mayes 1 Robert Gilliland 1, 2 Jeffrey Campbell 1, 2 Helen Holscher 3 Department of Surgery,](https://reader035.vdocuments.site/reader035/viewer/2022062301/56649f1c5503460f94c31dc1/html5/thumbnails/2.jpg)
Introduction of a Curriculum
• Web based curricula • Workplace Based Assessments (WBAs)
• Valid and reliable
• Formative and summative
• Clear indicators of professional progress
GMC April 2010
![Page 3: WBAs: the Northern Ireland School of Surgery Experience Richard Mayes 1 Robert Gilliland 1, 2 Jeffrey Campbell 1, 2 Helen Holscher 3 Department of Surgery,](https://reader035.vdocuments.site/reader035/viewer/2022062301/56649f1c5503460f94c31dc1/html5/thumbnails/3.jpg)
AimsTo determine if WBAs:
• Differentiate between trainees of different abilities (summative)
• Demonstrate progression towards competence (summative)
• Are completed at regular intervals through training (formative)
![Page 4: WBAs: the Northern Ireland School of Surgery Experience Richard Mayes 1 Robert Gilliland 1, 2 Jeffrey Campbell 1, 2 Helen Holscher 3 Department of Surgery,](https://reader035.vdocuments.site/reader035/viewer/2022062301/56649f1c5503460f94c31dc1/html5/thumbnails/4.jpg)
Methods
• Core Surgical trainees appointed Aug 2008
• Trainees not completing 24 months excluded
• Trainees were ranked according to interview
score (Q1, Q2, Q3 and Q4)
• 4 x 6 month placements (P1, P2, P3 and P4)
• Median (range): ANOVA
![Page 5: WBAs: the Northern Ireland School of Surgery Experience Richard Mayes 1 Robert Gilliland 1, 2 Jeffrey Campbell 1, 2 Helen Holscher 3 Department of Surgery,](https://reader035.vdocuments.site/reader035/viewer/2022062301/56649f1c5503460f94c31dc1/html5/thumbnails/5.jpg)
Methods
• CBDs, DOPS, Mini-CEXs – score 1-6• 1-2 - Below expectations• 3 - Borderline• 4 - Meets expectations• 5-6 - Above expectations
• PBAs – score 0-4• Level 0 – Insufficient evidence to support judgement• Level 1 – Unable to perform procedure• Level 2 – Able to perform procedure, or part under
supervision• Level 3 – Minimum supervision• Level 4 – Unsupervised, able to deal with complications
![Page 6: WBAs: the Northern Ireland School of Surgery Experience Richard Mayes 1 Robert Gilliland 1, 2 Jeffrey Campbell 1, 2 Helen Holscher 3 Department of Surgery,](https://reader035.vdocuments.site/reader035/viewer/2022062301/56649f1c5503460f94c31dc1/html5/thumbnails/6.jpg)
Results
• 39 trainees • 12 excluded• n=27 trainees
• 1,256 assessments• 381 CBDs• 408 DOPS• 346 Mini-CEXs• 121 PBAs
![Page 7: WBAs: the Northern Ireland School of Surgery Experience Richard Mayes 1 Robert Gilliland 1, 2 Jeffrey Campbell 1, 2 Helen Holscher 3 Department of Surgery,](https://reader035.vdocuments.site/reader035/viewer/2022062301/56649f1c5503460f94c31dc1/html5/thumbnails/7.jpg)
Number of assessments
0
50
100
150
200
250
300
350
P1 P2 P3 P4
Placement
Nu
mb
er PBA
CEX
DOPS
CBD
p=0.39
![Page 8: WBAs: the Northern Ireland School of Surgery Experience Richard Mayes 1 Robert Gilliland 1, 2 Jeffrey Campbell 1, 2 Helen Holscher 3 Department of Surgery,](https://reader035.vdocuments.site/reader035/viewer/2022062301/56649f1c5503460f94c31dc1/html5/thumbnails/8.jpg)
Can WBAs differentiate between trainees of different abilities?
![Page 9: WBAs: the Northern Ireland School of Surgery Experience Richard Mayes 1 Robert Gilliland 1, 2 Jeffrey Campbell 1, 2 Helen Holscher 3 Department of Surgery,](https://reader035.vdocuments.site/reader035/viewer/2022062301/56649f1c5503460f94c31dc1/html5/thumbnails/9.jpg)
Can WBAs differentiate between trainees of different abilities?
Trainees from Q1 would out perform trainees from Q4
• number of assessments performed • scores obtained
![Page 10: WBAs: the Northern Ireland School of Surgery Experience Richard Mayes 1 Robert Gilliland 1, 2 Jeffrey Campbell 1, 2 Helen Holscher 3 Department of Surgery,](https://reader035.vdocuments.site/reader035/viewer/2022062301/56649f1c5503460f94c31dc1/html5/thumbnails/10.jpg)
Median (range) Scores in CBD and Mini-CEXs
0
2
4
6
Quartile
p=0.11
Med
ian
Sco
re
Median (range) Number of WBAs Completed per Trainee
0
20
40
60
80
Quartile
p=0.92
Num
ber
of A
sses
smen
ts
![Page 11: WBAs: the Northern Ireland School of Surgery Experience Richard Mayes 1 Robert Gilliland 1, 2 Jeffrey Campbell 1, 2 Helen Holscher 3 Department of Surgery,](https://reader035.vdocuments.site/reader035/viewer/2022062301/56649f1c5503460f94c31dc1/html5/thumbnails/11.jpg)
Can WBAs demonstrate progression towards competence?
![Page 12: WBAs: the Northern Ireland School of Surgery Experience Richard Mayes 1 Robert Gilliland 1, 2 Jeffrey Campbell 1, 2 Helen Holscher 3 Department of Surgery,](https://reader035.vdocuments.site/reader035/viewer/2022062301/56649f1c5503460f94c31dc1/html5/thumbnails/12.jpg)
Can WBAs demonstrate progression towards competence?
Scores should show improvement over
4 placements
![Page 13: WBAs: the Northern Ireland School of Surgery Experience Richard Mayes 1 Robert Gilliland 1, 2 Jeffrey Campbell 1, 2 Helen Holscher 3 Department of Surgery,](https://reader035.vdocuments.site/reader035/viewer/2022062301/56649f1c5503460f94c31dc1/html5/thumbnails/13.jpg)
CBD, DOPS and Mini-CEX Median (range) Scores
P1 P2 P3 P40
2
4
6
DOPS
Mini-CEX
CBD
Placement
Sco
re
CBD p=0.47, DOPS p=0.19, Mini-CEX p=0.96
![Page 14: WBAs: the Northern Ireland School of Surgery Experience Richard Mayes 1 Robert Gilliland 1, 2 Jeffrey Campbell 1, 2 Helen Holscher 3 Department of Surgery,](https://reader035.vdocuments.site/reader035/viewer/2022062301/56649f1c5503460f94c31dc1/html5/thumbnails/14.jpg)
PBA Median (range) Scores
P1* P2 P3
P4*0
1
2
3
4
I----NS----I I----NS----I I----NS----I
I-------------*p=0.0007-------------I
Placements
Sco
re
![Page 15: WBAs: the Northern Ireland School of Surgery Experience Richard Mayes 1 Robert Gilliland 1, 2 Jeffrey Campbell 1, 2 Helen Holscher 3 Department of Surgery,](https://reader035.vdocuments.site/reader035/viewer/2022062301/56649f1c5503460f94c31dc1/html5/thumbnails/15.jpg)
Are WBAs being used in formative assessment?
![Page 16: WBAs: the Northern Ireland School of Surgery Experience Richard Mayes 1 Robert Gilliland 1, 2 Jeffrey Campbell 1, 2 Helen Holscher 3 Department of Surgery,](https://reader035.vdocuments.site/reader035/viewer/2022062301/56649f1c5503460f94c31dc1/html5/thumbnails/16.jpg)
Are WBAs being used in formative assessment?
WBAs should be completed at regular intervals through training placements
![Page 17: WBAs: the Northern Ireland School of Surgery Experience Richard Mayes 1 Robert Gilliland 1, 2 Jeffrey Campbell 1, 2 Helen Holscher 3 Department of Surgery,](https://reader035.vdocuments.site/reader035/viewer/2022062301/56649f1c5503460f94c31dc1/html5/thumbnails/17.jpg)
Total Number of Assessments per Month
A S O N D J F M A M J J A S O N D J F M A M J J0
50
100
150
200
P1 P2 P3 P4
*
*
*
*
Months of Placement
Nu
mb
er o
f A
sses
smen
ts
![Page 18: WBAs: the Northern Ireland School of Surgery Experience Richard Mayes 1 Robert Gilliland 1, 2 Jeffrey Campbell 1, 2 Helen Holscher 3 Department of Surgery,](https://reader035.vdocuments.site/reader035/viewer/2022062301/56649f1c5503460f94c31dc1/html5/thumbnails/18.jpg)
Summary
• No inter-quartile differences with regards to the numbers of WBAs performed or scores obtained
• CBD, DOPS, Mini-CEX scores do not improve with time
• PBAs scores improve with time
• WBAs are clustered immediately prior to appraisal
![Page 19: WBAs: the Northern Ireland School of Surgery Experience Richard Mayes 1 Robert Gilliland 1, 2 Jeffrey Campbell 1, 2 Helen Holscher 3 Department of Surgery,](https://reader035.vdocuments.site/reader035/viewer/2022062301/56649f1c5503460f94c31dc1/html5/thumbnails/19.jpg)
Conclusions
• WBAs do not fulfil GMC requirements
• Need modification
• Assessors require to be more robust in their evaluations
![Page 20: WBAs: the Northern Ireland School of Surgery Experience Richard Mayes 1 Robert Gilliland 1, 2 Jeffrey Campbell 1, 2 Helen Holscher 3 Department of Surgery,](https://reader035.vdocuments.site/reader035/viewer/2022062301/56649f1c5503460f94c31dc1/html5/thumbnails/20.jpg)
![Page 21: WBAs: the Northern Ireland School of Surgery Experience Richard Mayes 1 Robert Gilliland 1, 2 Jeffrey Campbell 1, 2 Helen Holscher 3 Department of Surgery,](https://reader035.vdocuments.site/reader035/viewer/2022062301/56649f1c5503460f94c31dc1/html5/thumbnails/21.jpg)
![Page 22: WBAs: the Northern Ireland School of Surgery Experience Richard Mayes 1 Robert Gilliland 1, 2 Jeffrey Campbell 1, 2 Helen Holscher 3 Department of Surgery,](https://reader035.vdocuments.site/reader035/viewer/2022062301/56649f1c5503460f94c31dc1/html5/thumbnails/22.jpg)
Is one assessor type more robust than the other?
![Page 23: WBAs: the Northern Ireland School of Surgery Experience Richard Mayes 1 Robert Gilliland 1, 2 Jeffrey Campbell 1, 2 Helen Holscher 3 Department of Surgery,](https://reader035.vdocuments.site/reader035/viewer/2022062301/56649f1c5503460f94c31dc1/html5/thumbnails/23.jpg)
Median (range) Scores in WBAswith HST/Consultant assessors
HSTsCons
HSTsCons
HSTsCons
0
2
4
6
P1 P2 P3
Placements
Sco
re
ANOVA p=0.42
![Page 24: WBAs: the Northern Ireland School of Surgery Experience Richard Mayes 1 Robert Gilliland 1, 2 Jeffrey Campbell 1, 2 Helen Holscher 3 Department of Surgery,](https://reader035.vdocuments.site/reader035/viewer/2022062301/56649f1c5503460f94c31dc1/html5/thumbnails/24.jpg)
Distribution of assessment scores
1 2 3 4 5 60
200
400
600
Score
Nu
mb
er