gordon page.pdf

Gordon PageEmeritus Professor, Faculty of Medicine

University of British ColumbiaVancouver Canada

Outline

The Rationale – Why developnational examinations for thehealth professions?

How is ‘competence’ defined?

What are the desired features ofnational examinations for healthprofessionals

Why develop nationalexaminations in the health

professions?

Why develop nationalexaminations in the health

professions?

A Quality AssuranceStrategy

‘Assurance’ of theQuality of What?

1.The competence of the graduates ofthe Health Professions SchoolsDo the graduates possess at least the minimallevels of competence (a) to fulfill the roles thatsociety has assigned to their professions, or (b) toadvance to the next level of training in theirprofession?

Summative Formative

‘Assurance’ of theQuality of What?

1.The competence of the graduates ofthe Health Professions Schools

2. The quality of the training programsthat prepare students for these roles

Summative Formative

Why develop nationalexaminations in the healthprofessions?

The educational rationale

To convey to students what is important to learn,and to motivate them to learn it (‘Assessmentdrives learning!’)

To influence the curricula of health professionsschools by providing an operational definition ofthe competencies expected of graduatingstudents.

To provide a mechanism for evaluating the qualityof and inequities between the schools in eachhealth profession, identifying S&W in each

Why develop nationalexaminations in the healthprofessions?

The health care rationale

To ensure that the graduates of healthprofessional programs are competent – toprovide a basis for decisions on theirlicensure/registration

To define a national standard – thoseabilities that together define thecompetencies and levels of performanceexpected of all graduates

Why develop national healthprofessions examinations?

In summary …

To provide guidance and motivation tostudents and schools on what isimportant to learn

To pass judgement the competence ofgraduating students, and the educationaleffectiveness of schools

To define a national standard – thoseabilities that define the competencies andlevels of performance expected of allgraduates

Outline




What is ‘competence’?

Competence is the “ability to do something”

In the context of health professionseducation and practice, Epstein (2002)defined competence as, “the habitual andjudicious use of communication, knowledge,technical skills, clinical reasoning, emotions,values, and reflections in daily practice toimprove the health of the individual patientand community”

A national examination of‘competence’ must therefore testthe ability to use/apply knowledgeand skills in the context of caringfor patients.

A test of the recall of knowledge is not a testof ‘competence’. While possessing and beingable to recall knowledge is ‘the cornerstone’ ofbecoming competent, it is not sufficient.Students who memorize and can recallknowledge are often not able to effectivelyapply or use that knowledge.

Possessing Knowledgeversus

Possessing Competence

There is a bigdifference!

Non VignetteWhat is the most likely renalabnormality in children with nephroticsyndrome and normal renal function?

acute poststreptococcal glomerulonephritis

hemolytic-uremic syndrome

minimal change nephrotic syndrome

nephrotic syndrome due to focal andsegmental glomerulosclerosis

Schönlein-Henoch purpura with nephritis

(A)

(B)

*(C)

(D)

(E)

Case SM, Swanson DB. Constructing Written Test Questions for the Basicand Clinical Sciences. Page 58-9. www.nbme.com

Short Vignette

A 2-year-old boy has a 1-weekhistory of edema. Blood pressureis 100/60 mm Hg, and there isgeneralized edema and ascites.Serum concentrations are:creatinine 0.4 mg/dL, albumin 1.4g/dL, and cholesterol 569 mg/dL.Urinalysis shows 4+ protein andno blood. What is the most likelydiagnosis?

Long Vignette

A 2-year-old black child developedswelling of his eyes and ankles overthe past week. Blood pressure is100/60 mm Hg, pulse 110/min, andrespirations 28/min. In addition toswelling of his eyes and 2+ pittingedema of his ankles, he has abdominaldistension with a positive fluid wave.Serum concentrations are: creatinine0.4 mg/dL, albumin 1.4 g/dL, andcholesterol 569 mg/dL. Urinalysisshows 4+ protein and no blood.

What is the most likely renal abnormality in children withnephrotic syndrome and normal renal function?

Non Vignette

acute poststreptococcal glomerulonephritishemolytic-uremic syndrome

minimal change nephrotic syndrome

nephrotic syndrome due to focal and segmentalglomerulosclerosisSchönlein-Henoch purpura with nephritis

(A)(B)

*(C)

(D)

(E)

Short VignetteA 2-year-old boy has a 1-week history of edema. Blood pressure is

100/60 mm Hg, and there is generalized edema andascites. Serum concentrations are: creatinine 0.4 mg/dL,

albumin 1.4 g/dL, and cholesterol 569 mg/dL. Urinalysis shows4+ protein and no blood. What is the most likely diagnosis?

Long VignetteA 2-year-old black child developed swelling of his eyes andankles over the past week. Blood pressure is 100/60 mm

Hg, pulse 110/min, and respirations 28/min. In addition toswelling of his eyes and 2+ pitting edema of hisankles, he has abdominal distension with a positivefluid wave. Serum concentrations are: creatinine 0.4 mg/dL,albumin 1.4 g/dL, and cholesterol 569 mg/dL. Urinalysis shows4+ protein and no blood.

What is the most likely renal abnormality in children withnephrotic syndrome and normal renal function?

A 2-year-old boy has a 1-week history of edema. Blood pressureis 100/60 mm Hg, and there is generalized edema and ascites.Serum concentrations are: creatinine 0.4 mg/dL, albumin 1.4g/dL, and cholesterol 569 mg/dL. Urinalysis shows 4+ proteinand no blood. What is the most likely diagnosis?

A 2-year-old black child developed swelling of his eyes andankles over the past week. Blood pressure is 100/60 mm Hg,pulse 110/min, and respirations 28/min. In addition to swellingof his eyes and 2+ pitting edema of his ankles, he has abdominaldistension with a positive fluid wave. Serum concentrations are:creatinine 0.4 mg/dL, albumin 1.4 g/dL, and cholesterol 569mg/dL. Urinalysis shows 4+ protein and no blood.

A B *C* D E1 0 99 0 0

8 1 90 1 0

0 0 98 2 0

5 2 82 8 1

0 1 98 1 0

10 9 66 10 5

Overall P-Value

94

88

84

Case SM, Swanson DB. Constructing Written Test Questions for the Basic and Clinical Sciences, 1996. Page 58-9.

Does testing knowledge recall vs applicationof knowledge affect test scores?

Possessing Knowledgeversus

Possessing Competence

There is a bigdifference!

National examinations must testcompetence, not knowledge recall!

But whatcompetencies should

be tested?

Standar Kompetensi DokterKonsil Kedokteran Indonesia

(Standards of Medical Competencies)(Indonesian Medical Council)

2006

Describes the competencies expected ofstudent doctor graduating from an

Indonesian medical schools.

Areas of Competency(Standar Kompetensi Dokter 2006)

1. Effective communication

2. Clinical skills

3. Scientific basis of medical knowledge

4. Management of health problems

5. Management of information

6. Self awareness and self development

7. Ethics, morals, medico-legal aspectsand professionalism, and patient safety

Medical Council of Canada

Objectives for the QualifyingExamination

The ‘Objectives’ define thecompetencies expected of medicalgraduates entering supervised andindependent practice

http://www.mcc.ca/Objectives_Online/

Competency Frameworkfor Doctors in Canada

Summary …‘How is competence defined?’

The ability to use/applyknowledge and skills in thecontext of caring for patients.

The ‘competencies’ comprising‘competence’ should be definedby each health profession

Outline




Utility (U) of anAssessment Strategy

V = Validity

R = Reliability

E = Educational impact

A = Acceptability

C = Cost

U = f(V, R, E, A, C)

Cees van der Vleuten, 1990s

What are the desired featuresof national examinations forhealth professionals?

V = Validity

R = Reliability


A = Acceptability

C = Cost


V = ValidityR = Reliability


A = Acceptability

C = Cost

Validity – An Essential Featureof National Examinations

The question posed by validity is,“Does this examination permit usto make correct inferencesabout the competence ofexaminees?”


The two main factors influencingvalidity are:

Examination Content – what istested?

Examination Methods – how is thecontent tested?


The two factors influencingvalidity are:

Examination Content – what istested?


International Trends in Medicine –Testing Methods used in National

Examinations in Indonesia, Canada, theUSA and Australia

An MCQ ‘written’ test Questions are ‘case-based’ Tests are computer-based 150 or more questions Use A-type MCQ questions Allow 3 or more hours of testing time

An ‘Objective Structured Clinical Examination’(OSCE) Use standardized patients (SPs) Use 3 or more hours of testing time Use 10 to 20 ‘stations’, each from 8 to 25 minutes in

length

Is there evidence supportingthe ‘validity’ of theseexamination methods?

Norcini (ASME, 2003) – Of all assessmenttechniques, written test (MCQs) scores haveshown to be the best predictors of futureclinical performance and of clinical outcomes.

Tamblyn et al (JAMA, 2002,1998) – Canadianlicensing examination (clinical decision-making) scores show sustained relationshipswith effectiveness indices of preventive care

and acute and chronic disease management

The Relationship of ExaminationScores to Concurrent ClinicalPerformance for Practice-readydoctors (Canadian study, 2007)

Correlations between:

Clinical Performance (Mini-CEX scores)

and MCQ scores .335

Clinical Performance (Mini-CEX scores)

and OSCE scores .372

Is there evidence supportingthe ‘validity’ of theseexamination methods?

Yes, it isreassuring but not

strong!


The two factors influencingvalidity are:

Examination Content – whatis tested?


An Examination Blueprint

Sample Blueprint forIndonesia

Competency areas MCQ OSCE

Communication skills 10%

Clinical Skills 20% 70%

Scientific basis … 10%

Management of healthproblems

60%

Management ofinformation

10%

Self awareness &development

10%

Ethics, morals,professionalism

10%

Blueprint for MCQ Items Testing‘Management of Health Problems’

Clinical Tasks ChildHealth

Maternalhealth

Adulthealth

Mentalhealth

Populationhealth

Data gathering 5 3 13 5 4

Datainterpretationand synthesis

6 3 17 6 4

Drugmanagement

2 2 10 3 0

Non-drugmanagement

2 1 9 3 0

Georges Bordage

Clinical Decision-Making

Reduced (“empty mind”)

Compiled (“recall/recognition”)

Elaborated (“deductive thinker”)

Dispersed (“cluttered mind”)

(Bordage, G. Academic Medicine, 1994, 1999)

Clinical Decision-Making

Reduced (“empty mind”)

Compiled (“recall/recognition”)

Elaborated (“deductive thinker”)

Dispersed (“cluttered mind”)

(Bordage, G. Academic Medicine, 1994, 1999)

****************************************

Summary – ‘What to test?’

National examinations should:

Test content defined relative to anexamination blueprint

Test the application/use of knowledgeand skills (i.e., ‘competencies’)

XNot test recall of knowledge

XNot test thoroughness of datagathering

XTest clinical decision making, notclinical reasoning


V = Validity

R = Reliability


A = Acceptability

C = Cost

Reliability

Is a measure of the accuracy of testscores

For MCQ examinations, accuracy islargely a function of the number of itemson the test (i.e., sampling)

For OSCEs, accuracy is also a function ofthe consistency of examiners and SPs

Reliability is expressed as a reliabilitycoefficient ‘r’, where 0<r<1

‘Rule of thumb’: r > .80 for makingdecisions about individual examinees

Reliability as a function oftesting time

TestingTime inHours

1

2

4

8

MCQ

0.62

0.76

0.93

0.93

OSCE

0.43

0.60

0.76

0.86


V = Validity*

R = Reliability*

E = Educational Impact

A = Acceptability

C = Cost

Educational Impact

Students will learn, and schools willteach what is tested on nationalexaminations

With a view to ‘assessment drivinglearning’, is the national examination‘driving’ teaching and learning in theright direction? It will if it testscompetencies that are important tothe practice of the health profession inquestion.

Acceptability What strategies encourage acceptance of

national examinations by key stakeholders –i.e., the schools, the professional licensing andregistration bodies, the profession, and thegovernment?

The key strategy is INVOLVEMENT – indefining policies related to the role of theexamination in the profession, in defining anddeveloping examination content, in definingthe standards of acceptable performance(passing scores), in quality assurance, …

Cost

National examinations with acceptablemeasurement qualities are costly todevelop, administer, score and maintain

The major cost is peoples’ time

In the USA, Australia and Canada thecosts of the MCQ and OSCE nationalexaminations exceed $2000 per candidatefor each examination – a cost which issupported primarily by examinee fees

Cost Major cost items include:

developing test blueprints

Developing/peer review of examinationquestions/cases

developing supportive computer systems for testdevelopment, test administration, and test analysis

pilot testing and refining questions/cases

standard setting

recruiting/training/paying examiners

examination scoring

quality assurance reviews (using examinationstatistics – e.g., reliability of test scoresdiscrimination indices of test items)

maintaining test security

…

In summary …




Developing the Ideal Modelfor National HealthProfessional Examinations Is an MCQ examination and an OSCE an ideal model for a

national examination? My answer: No!

Why? Option lists in MCQs provide too much cuing to weaker

candidates, and

OSCEs lack reliability (accuracy) and hence validity.

What is the ideal model? Short answer questions in place of MCQs, and

Highly structured assessments of clinical skills in real clinicalsettings in place of OSCEs

Why not use the ideal model? Too expensive, and quality control is even more challenging!

Terima kasih

gordon page.pdf

Documents

defined competence

minimallevels of competence

testof competence

developnational examinations

thehealth professions

health care rationale

educational rationale

recall of knowledge