Progress Testing
Prof Adrian Freeman
Director of Assessments UEMS
Progress Testing
• What is it?
• What we do?
• How it helps
• Possible future
Progress Testing
• Same test paper delivered to all students in all the years of the course.
• Different test papers for each testing occasion
Progress Testing
• Progress testing measures progression
• That is it!
• Five year course
• How much medical knowledge exists now?
• How much more in 5 years?
• How much more in 10/15/20?
• No fixed syllabus
• Learn how to learn
Progress Testing
• The progression curve
• Know end point (e.g. knowledge of newly qualified doctors)
• Know start point (Less than 10%, mostly)
• Discover rate of progression.
• Freeman A, Ricketts C. Choosing and Designing Knowledge Assessments. Medical Teacher 2010;32(7):578-81
Longitudinal Testing
P50
P25
P75
Tests
Scores
Cohort Mean Score
Progress Testing
• It is a predictive process.• Vleuten CPMVD, Verwijnen GM, Wijnen WHFW. Fifteen years of experience with progress
testing in a problem-based learning curriculum. Medical teacher. 1996;18(2):103-9.
• Norman, G, Neville, A, Blake, JM & Mueller, B (2010) Assessment steers learning down the right road: impact of progress testing on licensing examination performance. Med Teach, 32,496-9.
Progress Testing at UEMS
• No other summative knowledge testing
• Other assessment modules
– SSU
– Clinical Skills
– Professionalism
• Other institutions formative/summative
Progress Testing at UEMS
• PBL Course
• Spiral Curriculum
• Frequent look, rapid remediation
• No end of course surprises
• No cram and dump
Progress Test
• Test delivered 4 times a year to all students
• 125 questions in each test, blueprinted.
• 3 hours testing time
• Single Best Answer, 1 correct answer from 5 options + Don’t Know option.
• +1 –correct answer
• -0.25 – incorrect answer
• 0 – Don’t Know
• Knowledge level of foundation year doctor (First year after graduation)
• All questions should be applied knowledge, not factual recall
• All questions must be in clinical vignette
• We do test basic science in this format
0.00
10.00
20.00
30.00
40.00
50.00
60.00
PT
1
PT
2
PT
3
PT
4
PT
5
PT
6
PT
7
PT
8
PT
9
PT
10
PT
11
PT
12
PT
13
PT
14
PT
15
PT
16
PT
17
PT
18
PT
19
PT
20
PT
21
Me
an
Sco
re (
%)
Progress Test
02/03 Cohort
Year5
Year4
Year3
Year2
Year1
0.0
10.0
20.0
30.0
40.0
50.0
60.0
70.0
80.0
PT0
1P
T02
PT0
3P
T04
PT0
5P
T06
PT0
7P
T08
PT0
9P
T10
PT1
1P
T12
PT1
3P
T14
PT1
5P
T16
PT1
7P
T18
PT1
9P
T20
PT2
1P
T22
PT2
3P
T24
PT2
5P
T26
PT2
7P
T28
PT2
9P
T30
PT3
1P
T32
PT3
3P
T34
PT3
5P
T36
PT3
7P
T38
PT3
9P
T40
Pe
rce
nta
ge
Student Progress02/03 Cohort
03/04 Cohort
04/05 Cohort
05/06 Cohort
06/07 Cohort
07/08 Cohort
08/09 Cohort
09/10 Cohort
10/11 Cohort
11/12 Cohort
• Students receive results electronically
• Atlas
Standard Setting. Norm referencing
• Bottom 5 % - Unsatisfactory grade
• Next 10 % - Borderline
• Remainder - Satisfactory, except top 5% graded Excellent
Unsatisfactory
Borderline Satisfactory
Excellent
Year1 5% 10% 80% 5%
Year 2 4% 8% 83% 5%
Year 3 3% 6% 86% 5%
Year 4 2% 4% 89% 5%
Year 5 Criterion reference
Standard Setting.
• There are other standard setting methods
• Beware complexity
• Consider purpose
The Questions
• All questions reviewed by CAP
• Clinical Assessment Panel
• All practising clinicians who are daily working with newly qualified doctors
• Review questions for appropriate content and difficulty
• Review and either adapt or reject questions that have performed badly in tests
Question Banks and Security
• QB size about 4000
• Do use some MSC-AA questions
• Don’t repeat an individual question for at least 3 years
• Each question has an assigned “learning point”
• Students receive results electronically
• Atlas gives feedback
UEMS ATLAS
Feedback & Retesting Knowledge
• Autumn 2008.
• Question – A footballer falls in a tackle twisting his R knee. He presents with pain in the knee, the knee is swollen, flexion is limited and he is tender to palpation over the medial joint line.
• What is the injury?
• Answer – Medial Meniscus Injury
• Learning point – Knee injuries
25
Progress Test Learning Cycle?
Result
Learning
Points
Revised
Learning
Test
26
Feedback & Retesting Knowledge
• Autumn 2008. Question – A footballer falls in a tackle twisting his R knee. He presents with pain in the knee, the knee is swollen, flexion is limited and he is tender to palpation over the medial joint line. What is the injury?
• Answer – Medial Meniscus Injury
• Learning point – Knee injuries
• Spring 2009. Question – A skier falls twisting his L knee. He presents with pain in the knee, the knee is swollen, extension is limited, the anterior drawer sign is positive. What is the injury?
• Answer – Anterior Cruciate Injury
• Learning point – Knee injuries
Atlas On Line feedback
Progression decisions
• This is a longitudinal assessment and individual test scores aggregate. Decisions are made on aggregate scores NOT individual tests
• Each test gives further information about student profile
• Use all information to arrive at end of year progress decisions
Aggregation
Test 3 Aggregate Test 4 Score New Aggregate
Excellent Satisfactory
Doubtful Satisfactory Satisfactory
Borderline Doubtful
Unsatisfactory Unsatisfactory
Progression Decisions
5.004.003.002.001.00
year
80.00
60.00
40.00
20.00
0.00
perc
en
tag
e
Excellent
Satisfactory
Borderline
Unsatisfactory
gradecode
Remediation
• At several levels
• Student themselves
• Academic Tutor
• Locality Team
• Dedicated remediation team, single identifiable intervention.
Remediation
0
5
10
15
20
25
30
35
40
PT1 PT2 PT3 PT4 PT5 PT6 PT7 PT8 PT9 PT10 PT11 PT12 PT13 PT14 PT15
Perc
enta
ge
Feedback to faculty
0
10
20
30
40
50
60
70
80
90
100
Test
1
Test
2
Test
3
Test
4
Test
5
Test
6
Test
7
Test
8
Test
9
Test
10
Test
11
Test
12
Test
13
Test
14
Test
15
Test
16
Test
17
Test
18
Test
19
Test
20
Applied Medical Knowledge ScoresScore
Correct
Incorrect
Don't Know
Y5 Y4 Y3 Y2 Y1 items
ENT 0.69 0.64 0.55 0.39 0.20 14
Blood & Lymph 0.66 0.61 0.60 0.43 0.22 15
Skin 0.66 0.59 0.51 0.43 0.28 16
Respiratory 0.66 0.61 0.52 0.39 0.17 34
Renal 0.65 0.61 0.53 0.43 0.22 27
Mental Health 0.63 0.61 0.57 0.49 0.33 27
Cardiovascular 0.61 0.55 0.49 0.34 0.19 28
Musculoskeletal 0.60 0.53 0.45 0.33 0.17 51
Neurological 0.59 0.54 0.47 0.37 0.18 37
Digestive 0.58 0.53 0.50 0.37 0.19 46
Women's Health 0.56 0.51 0.47 0.39 0.23 45
Eyes 0.55 0.51 0.38 0.25 0.10 9
Homeostatic 0.54 0.46 0.37 0.25 0.11 14
Seriously Ill Patient 0.53 0.46 0.45 0.33 0.28 9
General Duties 0.52 0.45 0.42 0.28 0.16 21
Infectious Diseases 0.45 0.45 0.45 0.42 0.06 7
Child Health 0.42 0.40 0.43 0.35 0.34 13
non classified 0.00 0.00 0.00 0.00 0.00 0
Feedback to faculty
Feedback to faculty
How long does progress continue?
• At times we have asked qualified doctors to take the tests
40
test
2118111098
Per
cen
tage
80.0
70.0
60.0
50.0
40.0
30.0
F2
F1
GP
Consultant
PRHO
status
41
Summary DataStatus N Mean Standard
Deviation
PRHO 47 54.5 7.9
F1 25 55.6 10.3
F2 23 57.5 11.5
Consultant 33 62.2 8.0
GP 54 66.5 7.1
Total 182 60 9.8
Developments
• Has been on paper
• Venues
• Absences
• Now on line
• Equated tests
Possible developments
• Peninsula has become Exeter and Plymouth Medical Schools
• Gone from 2 medical schools in UK to 7
• International progress test in Maastricht and looking for collaborations
• Possibilities to share question banks/tests
• Test equating
Issues of Test Design
• Test delivered 4 times a year to all students
• Initially paper test with machine marked answer sheets
• Delivering summative test to 800+ students at the same time has significant resource implications
• Test equating – allows asynchronous testing
• On line delivery of tests
• Allows consortia of schools to share resources
Test Equating
• Create 4 papers
• Questions delivered in random order
Test Equating
• Each test 125 items• 75 unique, 25 shared• Allows numerical
comparison of difficulties of tests
• Then a scaled score for each test
Progress Testing
• Undergraduate - Yes
• Foundation – Looks possible
• Specialist Training – Useful for early remediation
• Recertification – Politically unacceptable?
• Combining resources. Computer adaptive testing
U tube
• Couple of useful clips
• Feedback description to students
• https://youtu.be/l_x-5haIVRs
• Brief descriptor of progress testing
• https://www.youtube.com/watch?v=Mc_-fVmEf1M
Questions?
Keynote speakersProfessor Cees van der Vleuten, Professor of Education, Maastricht University, The NetherlandsProfessor Adrian Freeman, Professor of Medical Education, University of Exeter Medical School, UKProfessor Carlos Collares, Assistant Professor, Maastricht University, The Netherlands
CONTINUING
PROFESSIONAL
DEVELOPMENT
Friday 22 June 2018, 9am – 5pm
Xfi Building, University of Exeter, Streatham Campus,
Exeter, United Kingdom
Topics will include:• Understanding progress testing•The psychometrics of
progress testing•Different contexts for
progress testing•The assessment tool for the
future?
AudienceOpen to all. Those experienced with PT can come and present academic findings, those new to PT can learn more about it and present their academic questions.
For a full programme for the day (including speakers, workshop themes and facilitators) visit www.exeter.ac.uk/medicine/cpd
For general enquiries: email [email protected] or call +44 (0) 1392 722964
CPD points applied through the European Accreditation Council for Continuing Medical Education
[Progress Testing (PT) is a powerful assessment tool to evaluate the continuing growth of student knowledge during the length of the course. It provides frequent and regular observations that permit rapid remediation interventions. It delivers regular feedback to students on their performance and how to direct their learning. After a lengthy and often expensive course this testing methodology prevents any surprises at students’ performance at finals exams at end of course. It repeatedly consolidates learning in curriculum areas and provides faculty feedback on how effective teaching is in curriculum areas. It is being implemented in more schools and institutions around the world and sits comfortably within the concepts of programmatic assessments. It is used in undergraduate and postgraduate training in many disciplines and for many purposes.]
This one day conference, organised by Prof Adrian Freeman, is a joint collaboration between the University of Exeter Medical School and the European Board of Medical Assessors, and will take the form of interactive plenaries and short oral presentations.
LOOKING AHEAD IN
PROGRESS TESTING
Cost: - Standard delegate rate £220
22nd June 2018