world conference nursing dubai 2015
TRANSCRIPT
“Back to the future: An online OSCE management Information system for nursing OSCEs"
Dr Thomas JB Kropmans
Back to the future, the movie
Developed by Ronald Harden in 1975 Medicine, nursing, dentistry, veterinary medicine, law, health sciences Student competency being assessed in consecutive series of stations Paper burden
Objective Structured Clinical Examinations e.g. OSCE
Admission interviews (Students) Selection interviews (HR) Objective Structured Clinical Examinations (Medicine, Nursing, Health Care, Law) Workplace Based Assessment
Where to be used
100 students x 15 stations = 1500 paper forms Not cost effective A 6-station OSCE report 327.5 hours of staff and faculty
time for each rotation of students (Ratanawongsa et al., 2008; Walsh & Smith, 2006)
US$ 6.90, equivalent to €4.70 per student per station (Cusimano et al., 1994)
We administer 11, 7-12 station OSCEs for a cohort of 670 students, produces 9380 assessment forms. To produce final OSCE results, the administrative cost of this procedure is €29,500, which is €2.80 per paper form.
Due to high error rates 30% (unpublished data)
Disadvantages
Only student ability to pass an examination is being assessed Difficulty of the assessment is not taken into account Heterogeneity of measurement instruments (stations) limits comparability Comparison of student competence across institutions and assessment (OSCE) settings is required Universal adoption of a standardized instrument is recommended (Clinical Teacher September 2014)
Without standard setting
Objective Structured Clinical Examinations e.g. OSCE
Of 1000 forms 300 contains errors Low Cronbach’s Alpha (< 0.8 poor) Pass mark of 50%? Introduction Online Marking Tool in December 2008
30% based upon error
Assess clinical skills or non-cognitive skills Determine between ‘good’ and ‘bad’ performance To provide adequate and timely feedback Reliable decision on ‘pass’ or ‘fail’
Purpose of a (formative or summative) clinical skills exam?
Real time solution
Real Time Data
Cronbach’s Alpha Internal consistency
Issues with examiners
Skew distribution • Median = 74%; min score = 4 (20%) and max 20 (100%)
Distribution of scores
74% 4
100%
Nr o
f stu
dent
s
20 20%
50%
Professional impression examiners
0.0
1.0
2.0
3.0
4.0
5.0
6.0
7.0
8.0
9.0
10.0
0 1 2 3 4
Stat
ion
scor
es
Method 1: Global score (Fail=0, Borderline=1, Clear Pass=2, Good Pass=3, Excellent=4)
Method 1 Single borderline score
Borderline Regression Analysis
Improved pass/fail decisions
User Acceptance Test n=18
Satisfaction using the E-OSCE package 88.40% (min- 74; max = 100)
22 prestigious universities & professional bodies >200 OSCEs successfully administered Student and Examiner results being analysed Cost reduction of 70% Error reduction by 30%
19% MORE NUIG students fail after introducing BRA Internal consistency varies from 0.45 – 0.85 Cut-off score varies between stations from 40 – 75% Generalisability Kappa between departments 0.4 – 0.9
References