progress testing - mededworldprogress testing •it is a predictive process. • vleuten cpmvd,...

Progress Testing

Prof Adrian Freeman

Director of Assessments UEMS

A.C.Freeman@exeter.ac.uk

Progress Testing

• What is it?

• What we do?

• How it helps

• Possible future

Progress Testing

• Same test paper delivered to all students in all the years of the course.

• Different test papers for each testing occasion

Progress Testing

• Progress testing measures progression

• That is it!

• Five year course

• How much medical knowledge exists now?

• How much more in 5 years?

• How much more in 10/15/20?

• No fixed syllabus

• Learn how to learn

Progress Testing

• The progression curve

• Know end point (e.g. knowledge of newly qualified doctors)

• Know start point (Less than 10%, mostly)

• Discover rate of progression.

• Freeman A, Ricketts C. Choosing and Designing Knowledge Assessments. Medical Teacher 2010;32(7):578-81

Longitudinal Testing

P50

P25

P75

Tests

Scores

Cohort Mean Score

Progress Testing

• It is a predictive process.• Vleuten CPMVD, Verwijnen GM, Wijnen WHFW. Fifteen years of experience with progress

testing in a problem-based learning curriculum. Medical teacher. 1996;18(2):103-9.

• Norman, G, Neville, A, Blake, JM & Mueller, B (2010) Assessment steers learning down the right road: impact of progress testing on licensing examination performance. Med Teach, 32,496-9.

Progress Testing at UEMS

• No other summative knowledge testing

• Other assessment modules

– SSU

– Clinical Skills

– Professionalism

• Other institutions formative/summative

Progress Testing at UEMS

• PBL Course

• Spiral Curriculum

• Frequent look, rapid remediation

• No end of course surprises

• No cram and dump

Progress Test

• Test delivered 4 times a year to all students

• 125 questions in each test, blueprinted.

• 3 hours testing time

• Single Best Answer, 1 correct answer from 5 options + Don’t Know option.

• +1 –correct answer

• -0.25 – incorrect answer

• 0 – Don’t Know

• Knowledge level of foundation year doctor (First year after graduation)

• All questions should be applied knowledge, not factual recall

• All questions must be in clinical vignette

• We do test basic science in this format

0.00

10.00

20.00

30.00

40.00

50.00

60.00

PT

1

PT

2

PT

3

PT

4

PT

5

PT

6

PT

7

PT

8

PT

9

PT

10

PT

11

PT

12

PT

13

PT

14

PT

15

PT

16

PT

17

PT

18

PT

19

PT

20

PT

21

Me

an

Sco

re (

%)

Progress Test

02/03 Cohort

Year5

Year4

Year3

Year2

Year1

0.0

10.0

20.0

30.0

40.0

50.0

60.0

70.0

80.0

PT0

1P

T02

PT0

3P

T04

PT0

5P

T06

PT0

7P

T08

PT0

9P

T10

PT1

1P

T12

PT1

3P

T14

PT1

5P

T16

PT1

7P

T18

PT1

9P

T20

PT2

1P

T22

PT2

3P

T24

PT2

5P

T26

PT2

7P

T28

PT2

9P

T30

PT3

1P

T32

PT3

3P

T34

PT3

5P

T36

PT3

7P

T38

PT3

9P

T40

Pe

rce

nta

ge

Student Progress02/03 Cohort

03/04 Cohort

04/05 Cohort

05/06 Cohort

06/07 Cohort

07/08 Cohort

08/09 Cohort

09/10 Cohort

10/11 Cohort

11/12 Cohort

• Students receive results electronically

• Atlas

Standard Setting. Norm referencing

• Bottom 5 % - Unsatisfactory grade

• Next 10 % - Borderline

• Remainder - Satisfactory, except top 5% graded Excellent

Unsatisfactory

Borderline Satisfactory

Excellent

Year1 5% 10% 80% 5%

Year 2 4% 8% 83% 5%

Year 3 3% 6% 86% 5%

Year 4 2% 4% 89% 5%

Year 5 Criterion reference

Standard Setting.

• There are other standard setting methods

• Beware complexity

• Consider purpose

The Questions

• All questions reviewed by CAP

• Clinical Assessment Panel

• All practising clinicians who are daily working with newly qualified doctors

• Review questions for appropriate content and difficulty

• Review and either adapt or reject questions that have performed badly in tests

Question Banks and Security

• QB size about 4000

• Do use some MSC-AA questions

• Don’t repeat an individual question for at least 3 years

• Each question has an assigned “learning point”

• Students receive results electronically

• Atlas gives feedback

UEMS ATLAS

Feedback & Retesting Knowledge

• Autumn 2008.

• Question – A footballer falls in a tackle twisting his R knee. He presents with pain in the knee, the knee is swollen, flexion is limited and he is tender to palpation over the medial joint line.

• What is the injury?

• Answer – Medial Meniscus Injury

• Learning point – Knee injuries

25

Progress Test Learning Cycle?

Result

Learning

Points

Revised

Learning

Test

26

Feedback & Retesting Knowledge

• Autumn 2008. Question – A footballer falls in a tackle twisting his R knee. He presents with pain in the knee, the knee is swollen, flexion is limited and he is tender to palpation over the medial joint line. What is the injury?

• Answer – Medial Meniscus Injury

• Learning point – Knee injuries

• Spring 2009. Question – A skier falls twisting his L knee. He presents with pain in the knee, the knee is swollen, extension is limited, the anterior drawer sign is positive. What is the injury?

• Answer – Anterior Cruciate Injury

• Learning point – Knee injuries

Atlas On Line feedback

Progression decisions

• This is a longitudinal assessment and individual test scores aggregate. Decisions are made on aggregate scores NOT individual tests

• Each test gives further information about student profile

• Use all information to arrive at end of year progress decisions

Aggregation

Test 3 Aggregate Test 4 Score New Aggregate

Excellent Satisfactory

Doubtful Satisfactory Satisfactory

Borderline Doubtful

Unsatisfactory Unsatisfactory

Progression Decisions

5.004.003.002.001.00

year

80.00

60.00

40.00

20.00

0.00

perc

en

tag

e

Excellent

Satisfactory

Borderline

Unsatisfactory

gradecode

Remediation

• At several levels

• Student themselves

• Academic Tutor

• Locality Team

• Dedicated remediation team, single identifiable intervention.

Remediation

0

5

10

15

20

25

30

35

40

PT1 PT2 PT3 PT4 PT5 PT6 PT7 PT8 PT9 PT10 PT11 PT12 PT13 PT14 PT15

Perc

enta

ge

Feedback to faculty

0

10

20

30

40

50

60

70

80

90

100

Test

1

Test

2

Test

3

Test

4

Test

5

Test

6

Test

7

Test

8

Test

9

Test

10

Test

11

Test

12

Test

13

Test

14

Test

15

Test

16

Test

17

Test

18

Test

19

Test

20

Applied Medical Knowledge ScoresScore

Correct

Incorrect

Don't Know

Y5 Y4 Y3 Y2 Y1 items

ENT 0.69 0.64 0.55 0.39 0.20 14

Blood & Lymph 0.66 0.61 0.60 0.43 0.22 15

Skin 0.66 0.59 0.51 0.43 0.28 16

Respiratory 0.66 0.61 0.52 0.39 0.17 34

Renal 0.65 0.61 0.53 0.43 0.22 27

Mental Health 0.63 0.61 0.57 0.49 0.33 27

Cardiovascular 0.61 0.55 0.49 0.34 0.19 28

Musculoskeletal 0.60 0.53 0.45 0.33 0.17 51

Neurological 0.59 0.54 0.47 0.37 0.18 37

Digestive 0.58 0.53 0.50 0.37 0.19 46

Women's Health 0.56 0.51 0.47 0.39 0.23 45

Eyes 0.55 0.51 0.38 0.25 0.10 9

Homeostatic 0.54 0.46 0.37 0.25 0.11 14

Seriously Ill Patient 0.53 0.46 0.45 0.33 0.28 9

General Duties 0.52 0.45 0.42 0.28 0.16 21

Infectious Diseases 0.45 0.45 0.45 0.42 0.06 7

Child Health 0.42 0.40 0.43 0.35 0.34 13

non classified 0.00 0.00 0.00 0.00 0.00 0

Feedback to faculty

Feedback to faculty

How long does progress continue?

• At times we have asked qualified doctors to take the tests

40

test

2118111098

Per

cen

tage

80.0

70.0

60.0

50.0

40.0

30.0

F2

F1

GP

Consultant

PRHO

status

41

Summary DataStatus N Mean Standard

Deviation

PRHO 47 54.5 7.9

F1 25 55.6 10.3

F2 23 57.5 11.5

Consultant 33 62.2 8.0

GP 54 66.5 7.1

Total 182 60 9.8

Developments

• Has been on paper

• Venues

• Absences

• Now on line

• Equated tests

Possible developments

• Peninsula has become Exeter and Plymouth Medical Schools

• Gone from 2 medical schools in UK to 7

• International progress test in Maastricht and looking for collaborations

• Possibilities to share question banks/tests

• Test equating

Issues of Test Design

• Test delivered 4 times a year to all students

• Initially paper test with machine marked answer sheets

• Delivering summative test to 800+ students at the same time has significant resource implications

• Test equating – allows asynchronous testing

• On line delivery of tests

• Allows consortia of schools to share resources

Test Equating

• Create 4 papers

• Questions delivered in random order

Test Equating

• Each test 125 items• 75 unique, 25 shared• Allows numerical

comparison of difficulties of tests

• Then a scaled score for each test

Progress Testing

• Undergraduate - Yes

• Foundation – Looks possible

• Specialist Training – Useful for early remediation

• Recertification – Politically unacceptable?

• Combining resources. Computer adaptive testing

U tube

• Couple of useful clips

• Feedback description to students

• https://youtu.be/l_x-5haIVRs

• Brief descriptor of progress testing

• https://www.youtube.com/watch?v=Mc_-fVmEf1M

Questions?

Keynote speakersProfessor Cees van der Vleuten, Professor of Education, Maastricht University, The NetherlandsProfessor Adrian Freeman, Professor of Medical Education, University of Exeter Medical School, UKProfessor Carlos Collares, Assistant Professor, Maastricht University, The Netherlands

CONTINUING

PROFESSIONAL

DEVELOPMENT

Friday 22 June 2018, 9am – 5pm

Xfi Building, University of Exeter, Streatham Campus,

Exeter, United Kingdom

Topics will include:• Understanding progress testing•The psychometrics of

progress testing•Different contexts for

progress testing•The assessment tool for the

future?

AudienceOpen to all. Those experienced with PT can come and present academic findings, those new to PT can learn more about it and present their academic questions.

For a full programme for the day (including speakers, workshop themes and facilitators) visit www.exeter.ac.uk/medicine/cpd

For general enquiries: email UEMS_CPD@exeter.ac.uk or call +44 (0) 1392 722964

CPD points applied through the European Accreditation Council for Continuing Medical Education

[Progress Testing (PT) is a powerful assessment tool to evaluate the continuing growth of student knowledge during the length of the course. It provides frequent and regular observations that permit rapid remediation interventions. It delivers regular feedback to students on their performance and how to direct their learning. After a lengthy and often expensive course this testing methodology prevents any surprises at students’ performance at finals exams at end of course. It repeatedly consolidates learning in curriculum areas and provides faculty feedback on how effective teaching is in curriculum areas. It is being implemented in more schools and institutions around the world and sits comfortably within the concepts of programmatic assessments. It is used in undergraduate and postgraduate training in many disciplines and for many purposes.]

This one day conference, organised by Prof Adrian Freeman, is a joint collaboration between the University of Exeter Medical School and the European Board of Medical Assessors, and will take the form of interactive plenaries and short oral presentations.

LOOKING AHEAD IN

PROGRESS TESTING

Cost: - Standard delegate rate £220

22nd June 2018