ASSESSMENT IN GP/FM

Mladenka Vrcic-Keglevic, MD, PhD

Head of Family Medicine Department, Medical School, Univerity of Zagreb

mvrcic@snz.hr

Content of the session

• Definition of the terms• How to use assessment (summative,

formative)• Assessment methods (AM)• What can be assessed by different AM• Metric’s characteristics of AM• …..• Interactive session

Definition of the terms

• Evaluation: A process that attempts to systematically determine the relevance, effectiveness, and impact of activities in light of their objectives. Can be related to structure, process or outcome of (programme, person).

• Assessment: A system of evaluation of professional accomplishments in light of educational objectives, using defined criteria and usually including an attempt at measurement either by grading on a rough scale or by assigning numerical values.

AMEE – Glosary of Medical Education Terms

Well defined educational objectives - prerequsite for assessment

Example for this session:

At the end of this session participants will be able:• To named at list three differences between

summative and formative assessment• To make a list of at least three written AM• To name the most effective AM to assess clinical

skills• To describe the most effective AM to assess

attitudes

How to use assessment?

• Summative: This is usually undertaken at the end of a training programme or teaching course and determines whether the educational objectives have been successfully achieved. With summative assessment the students usually receives a grade or a mark. Exam

• Formative: This is testing that is part of developmental or ongoing teaching / learning process. It should emanate from a wish to foster learning and understanding. It should include delivery of feedback to the student.

Summative - Examination

What the exams are?

• For students – A difficult and unpleasent steeplechase to run on a way for diploma

• For teachers – A less desirable teaching activity

• For public – An important protection from un-competent doctors

Summative assessment

The reasons:• A statement of achievement - university degree

(diploma)• An entrance requirement to an educational

institution• A guide as to the wisdom of continuing with

further study• A certification of competence – public

responsibility (licence)• A determinant of programme effectiveness

Assessment methods (AM)

Written:• MCQ - Multiple Choice

Questions

• Un-Q – An-cued Questions

• Essay• MEQ – Modified Essay

Questions

• EMI - extended matching items

• PMP – Patient Management Problem

• Oral, viva• Traditional practical• OSCE – Objective

Structured Clinical Examination

• SP - Standardised Patient

• Portfolio• ACC – Assessment of

Clinical Competence

What can be assessed by different AM

• Factual knowledge

• Interpretations

• Problem-solving skills

• Ethical

• Clinical skills

• Emotional reactions

• Communication

What can be assessed?

Fact Inter P-S Ethi Skill Emo Com

Oral ++ + +/- - - - - + +

Practic. + ++ ++ + - + + - + -

MCQ ++ + - + - - - - - - - - -

Un-Q ++ + - + - - - - - - - - -

MEQ + ++ ++ + - - - - - - -

Essay ++ + - + - + - + - - - - -

OSCE + + + + - + - ++ + - + -

PMP +- ++ ++ +- - - - - - -

ACC ++ ++ ++ ++ ++ ++ ++

Metric characteristics of AM

• Validity - the degree to which a measurement instrument truly measures what it is intended to measure

• Reliability – it is an expression of the precision, consistency and reproducibility. Ideally, measurements should be the same when repeated by the same student or made by the different assessors.

• Relevance – it is a degree to which the assessment

questions and educational objectives are in concordance • Objectivity – it is a level of agreement among independent

assessors (experts) about the right answer to certain question

Metric’s characteristics?

Validity Reliability Relevent Objectivity

Oral - - - - - - - -

Practic. - - + -

MCQ ++ ++ ++ ++

Un-Q ++ ++ ++ ++

MEQ ++ ++ ++ ++

Essay - - + -

OSCE ++ ++ ++ ++

PMP/SP ++ ++ ++ ++

ACC - + - + + - +

Practicality?

• Preparation – resources (time, human…)

• Easy to perform, aplay

• NC – number of candidates

• NC/teacher – number of candidates per one teacher

Practicality

Preparation Ease to applay

NC- No of candidates

NC / teacher

Oral ++ - - - - - -

Practic. ++ + - - - -

MCQ +++ ++ ++ ++

Un-Q +++ ++ ++ ++

MEQ - ++ ++ -

Essay + + + - -

OSCE + + + +

PMP/SP + + - ++ +

ACC ++ + - - - -

MCQ – Multiple Choice Questions

• Difficulty index

• Discrimination index

• Minimum pass level

Indeks of difficultyQD = (G + P)/N

• QD – question’s difficulty• G – number of right

answers in good group• P – number of right

answers in poor group• N – number of students in

both groups

a b c d e 0 U

1

2

3

Difficulty Index

0,3 0,5 0,6 0,7

------/---------------(------------)----------/--------- recommended

--------------------------------------------------------------

acceptable

too difficult too difficult

Discrimination Index

• DI = 2 (G-P)/N• DI – discrimination Index• G – number of right

answer in “good” group• P – number of right

answer in “poor” group• N – number of students in

both group

a b c d e 0 U

1

2

3

Diskrimination Index

0.15 0.25 0.35

----------/----------/----------/---------- throw off to check good excelent

What AM to use in reality?

• There is no perfect AM• A choice depends of what is to be measured –

educational objectives (knowledge, skills, competences, attitudes)

• Careful preparation, trained assessors• Multiple methods – better then one

Specialistic exam in GP in Croatia

Should be submeted in advance:• Portfolio• 8 essays• Questionary of professional competence –

signed by trainer• Trainer’s report about trainee’s achievements• Trainee’s report – How to organise my practice,

what to invent?• Medical records (videos) of 3 patients / family

Specialistic exam in GP in Croatia

• MCQ test – 150 questions

• OSCE – 20-25 stations

• Oral – jury (3 members): 1 academic GP + 1 well recognised GP (publication) + 1 trainer

Formative assessment

The reasons:

• Information for the student about his/her achievement of educational objectives

• Repetitive – progress measurement

• Discover “week points”–teacher’s support

• Help to a teacher to correct programme

• The results should not be used in summative assessment

Formative assesssment -which AM?

• The same as for summative – progress test• Protocols for observing clinical work

(consultation, communication, p-s abilities, professional competences)

• Checklists• Diaries – what has been done• Appraising projects, audit…• Portfolio• Self and peer assessment

Formative assesssment

• Feedback

• Feedback

• Feedback

• Feedback

• Feedback

Educational objectives in this session – achieved?

At the end of this session participants will be able to:

• name at least three differences between summative and formative assessment

• make a list of at least three written AM• name the most effective AM to assess

clinical skills• describe the most effective AM to assess

attitudes