the “key features” approach to assess clinical...

The “Key Features” Approachto assess

Clinical Decision Making

Georges Bordage, MD, PhD, Professor

Michael G. DeGroote School of Medicine at McMaster University November, 2007

© G. Bordage, UIC, Chicago, 2007

Department of Medical EducationCollege of MedicineUniversity of Illinois at Chicago

2

Acknowledgements

Robert Lee Medical Council of Canada

Gordon Page University of British Columbia

Stephen Aaron University of Alberta

3

Workshop Objectives

1. Define “Key Features” (KFs)2. Use a systematic strategy to:

- Select problems- Define KFs- Develop KF cases, q. & scoring keys

3. Understand psychological & measurement basis of KFs approach

4

Agenda

1- What are KFs? Why change?2- Select problems & define KFs3- Develop KF test cases & quest.4- Select response format & scoring5- Why KF cases

Closure

5

In doubt… ask questions

6

What are Key Features?

Unique challenges,case-specific decisions,in the resolution of aclinical problem,for the level of candidates being tested

7

Key Features

Problem: Infant in severe, early respiratory distress

Examinee: Graduating medical students should:

KF-1 Consider 3 impending conditions: respiratory failure, dehydration, & congestive heart failure.

KF-2 Immediately evaluate & manage pt, including: ABG, nebulized salbutamol, O2, IV line, and portable chest x-ray; andavoid unnecessary investigation in acute phase.

Clinical scenarioThe triage nurse in the Emergency Department asks you to see a 9-month-old boy. The boy’s mother tells you that her son has had a cold for the past 4 days and…

Followed by 2 or 3 questions assessing only KFs:

1. What impending condition(s), if any, will you consider in this infant?

2. What orders or actions will you take, if any, in your immediate evaluation and management of this infant?

Paper & pencil (or OSCE, MCQs, Orals)

What’s a KF Case?

9

60s 80sProblem Decisionsolving making

PMPs KFsPatientManagement Key FeaturesProblems

Why change?

10

PMPs: Patient Management Problems

Christine McGuireCED-UIC 1960s

Paper & pencil testClinical scenario (CC) +

Sections (H&P, Lab. invest., Dx, Managt., F-up)

[L] Options || [R] Latent images – answers

11

Section D1. Alanine Aminotransferase (ALT)2. Alcohol level3. Aldolase, serum4. Alkaline phosphatase, serum5. Amylase, serum6. Arterial blood gases (ABG)7. Aspartate Aminotransferase (AST)8. Brain CT-scan9. Brain MRI10. Brain PET-scan11. Calcium, serum12. Carotid US-doppler13. Cerebral angiography14. Cerebro-spinal fluid examination 15. Complete Blood Count (CBC)16. C-Reactive Protein17. Creatine Phophokinase, serum18. Creatinine, serum19. Drug screening, serum20. Drug screening, urineEtc…

1. 2. 3. 4. 5. 6. 7. 8. No abnormalities noted 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 1.9 (0.7-1.5 mg/dL)19. Salicylate: 32mg/dL (20-25 mg/dL)20.

Latent images

12

PMPs: Patient Management Problems

3-hour or ½-day: ≈10 cases

Scoring: -2 -1 0 +1 +2

The more “good” things you did,the more THOROUGH you were,the higher your score

13

1984 : Cambridge Conference

PMPs

Dx: 5 ailments

14

5 ailments re: PMPs

Low content validity: ≈10 probl.

Low reliability (consistency) : ≈ .3

Problem solving = General skill

Unique format (latent image) : cueing

Over-rewarding thoroughness

15

Thoroughnessis a predictor of

“poor” performance

Elstein, Shulman &Sprafka, 1978

16

When in doubt, collectingmore data (EKG features)

- did not improve Dx accuracy- indicator of uncertainty, Dx error

Hatala et al, 1998

PMPs… rewarding wrong behavior

17

Problem solving in medicine

Not a general skill

Specific toeach case

18

Inter-case correlation = .1 - .3

Case Specificity

Each case presents unique challengesArthritis ≠ Anemia ≠ Crohns ≠ Diabetes

Key Features (KFs)

19

Consequences for assessment

• Assess only the unique challenges,the case-specific decisionsin the resolution of a clinical problem

…best discriminators

• Many short, focused problems • Better sampling… better reliability &

content validity

20

Purpose (object) of assessment

Knowledge recall (describe DT)

Clinical reasoning (how)

Clinical decisions, actions(recognize & manage DT)

H&P, Dx, Rx, Investigation, F-up

How to writea KF Case

22

3 separate stepsI. Select problemsII. Define KFs for each problemIII. Write test material

(cases, questions & scoring key)

IV. Pilot test cases

23

I- Select problems

Goal: Adequate &representative numberof clinical problemsfrom the domainfor… [graduating students]

24

Domain of clinical problems

Medical Council of CanadaObjectives for the Qualifying Exam

(3 rd ed., 2003)

120 primary clinical presentations (dyspnea)140 related clinical presentations260 clinical presentations

25

By alphabetical order

• Dysphagia/Difficulty swallowing 26-E

• Dyspnea 27-E

Acute dyspnea (minutes to hours) 27-1-E

Chronic dyspnea (weeks to months) 27-2-E

Pediatric dyspnea (resp. distress) 27-3-E

• Scrotal Mass• Scrotal Pain• Seizures (Epilepsy)

S

D

26

By disciplines

• Primary Care• Medicine• Obstetrics & gynecology• Population, ethics, legal, org.• Pediatrics• Psychiatry• Surgery

27

Each clinical presentation

• Rationale• Causal conditions• Key objectives• Objectives• Ethics• Applied sc. concepts

28

How many problems, pts?

• Inter-case correlation = .1 - .3• Desired reliability = 0.8

• Spearman-Brown Formula 40problems

29

Select problemsfrom a blueprint

Preg., neonat., infant 5%* 3Children (Peds) 16% 6Adolescents 16% 6Adults 47% 19Elderly (geriatrics) 16% 6

* Health Services Data 40

AGE GROUPS

30

Test Development Process

I- Chair randomly selects problems

II- Assigns a problem to a memberto define KFs

Discussion with committee

III- Member writes test case & quest.Discussion with committee

31

Selecting problems

Avoid broad problems…Ischemic disordersAlcohol abuseEpilepsy

Use more specific problemsStable and unstable anginaAlcohol withdrawal

32

- D.Dx with: SyncopyDrug withdrawalMetab/endo derangementFebrile seizures

- Emergency treatment

- Prevention of seizures…

Where do clerks go wrong?

SEIZURES - Epilepsy

33

Clinical situations

Undifferentiated complaintSingle, typical (atypical)Multiple, multi-systemUrgent, life-threateningPrevention,health promotion

34

Seizures (epilepsy)

RationaleCausal ConditionsKey Objectives ER Treatment ofObjectives Status EpilepticusEthicsApplied Sc. Concepts

35

Unique challenges, critical steps, decisions in the resolution of the problem

Steps, actions most likely to lead to errorMost difficult aspects of problem identification and management in practice

Focus only on Key Features

What Needs Examining?

36

Emergency Rx of Status Epilepticus:Adult – Life-threatening

Think about many cases at first, many possible actions…

Then focus on KFs…

II- Define KFs

37

Anatomy of a KF…

Initial clinical informationGiven an adult presenting with…

A clinical taskOrder investigation…If necessary, a qualifierOrder immediate investigation

38

Given a man brought to the E.R. with multiple seizures and without having regained consciousness, the graduating medical student should:

Key Featuresfor the graduating medical student

What Needs Examining?

39

1. Generate provisional Dx of status epilepticus

2. Begin initial management: Monitor cardio-respiratory fcts (ABC), NS, IV vitB, bolus hypertonic glucose, IV diazepam +/- phenytoin

3. Elicit Hx of possible causes: use of alcohol, medication, heroin or cocaine, diabetes

4. Order immediate exams: lytes, glucose, ca, ABG, drug screening, brain CT/MRI

What Needs Examining? KFs

40

Location, setting

At homeOffice, ambulatory clinicEmergency roomHospitalICU

With supportLimited supportNo support

41

How Valid?

Clerkship directors from acrossCanada confirmed :

- Existing KFs 92%- Generating KFs 94%

42

Practice… I & II

43

I. Select problems

Given a clinical presentation :Select a specific clinical situationfor the level of candidates being testedWhere do [clerks] go wrong?

44

MCC Objectives

Dyspnea…

45

II- Define KFs for the problem

Unique challenges, critical steps, decisions in the resolution of the problem

Steps, actions most likely to lead to errorMost difficult aspects of problem identification and management in practice

“Where do clerks go wrong?”

46

Is the challenge for clerk, mostly in:

Data gathering ?Data interpretation, Dx ?Under/over ordering investigation ?Selecting the Rx, management ?

Where do clerks most oftengo wrong ???

47

Test Development Process

I- Chair randomly selects problems

II- Assigns a problem to a memberto define KFs

Discussion with committee

III- Member writes test case & quest.Discussion with committee

48

III- Write test material

1) Prepare clinical scenariore: problem, situation, & KFs

2) Write test questionsre: KFs only

3) Choose a response format

4) Set scoring key

49

KFs dictate case scenario

Given man w/ suspected alcohol dependence brought to ER w/ multiple seizures w/o regaining consciousness…

Case scenario includes:CC, some Hx, initial physical

50

Sounds realData presented as in real life, uninterpreted:

“vomited blood” vs. “hematemesis”Leave out data if you want to ask about history, physical exam or investigationPut in info if you want to ask about managemtPut non-contributing data in the case, even if they don’t relate to the KFs, as in real life

Writing the Scenario

Mr. “X,” a 36-year-old man, is brought to the emergency room in your hospital by ambulance because he fell to a sidewalk unconscious while waiting for the bus. A witness immediately called an ambulance and reported to the ambulance crew that before falling to the ground, he seemed confused, agitated, and was arguing with some invisible person. After falling, he began to twitch for a short while, his face became blue, and then he began to have jerky movements all over his body for about a minute. He did not recover consciousness after the episode. During the 10-minute ambulance trip, he presented two other similar episodes, without recovering consciousness, and a third episode that you witnessed on arrival. His temperature is 37.8 C. He looks neglected & is unconscious. No relatives, friends accompanied Mr. “X.”

52


1) Prepare clinical scenariore: problem, situation, & KFs

2) Write test questionsre: KFs only

3) Choose a response format

4) Set scoring key

53

Generally1 question / KF

Asking the Question(s)

54






55

KF-1 Generate provisional Dx of status epilepticus

What’s the exam question?


56

Asking the Question

What is (are) your leading working diagnosis (es) at this point in time? You may list up to two.

1. ____________________________

2. ____________________________

Question 1 (KF-1)

57






58

Question 2 (KF-2)

What is your immediate management at this pointin time? List as many things as you feel are appropriate.1. ___________________________________2. ___________________________________3. ___________________________________4. ___________________________________5. ___________________________________6. ___________________________________7. ___________________________________8. ___________________________________9. ___________________________________10. ___________________________________

59

Question 3 (KF-3)

Ten minutes after arrival, Mr. “X” is still unconscious. The nurse found a telephone number in his wallet that you decide to call immediately. What questions will you ask the person answering the phone – assuming he/she knows the patient? You may select up to six questions. Select option 35 if you think that it is not appropriate to call at this point in time.

60

…Question 3

19. Lung infection20. Medication history21. Muscular disease22. Nausea23. Palpitation history24. Pet in household25. Previous similar problem26. Profession27. Sexual history28. Smoking history29. Social integration difficulties30. Surgery31. Travel history32. Viral infection33. Visual impairment34. Vomiting35. Not appropriate to call at this point

in time.

1. Abdominal pain2. Alcohol history3. Back pain history4. Benzodiazepine5. Cancer history6. Cocaine abuse7. Coronary bypass history 8. Diabetes history9. Diarrhea10. Dizziness11. Drug allergy12. Family history13. Food allergy14. Headache15. Hearing disability16. Heroin abuse17. Joint pain18. LSD abuse

61

Question 4 (KF-4)

It has been 15 minutes since Mr. X's arrival. What ancillary exams will you order at this point? You may select as many as you feel appropriate. Select option 35 if you think that ancillary exams are not needed at this point in time.

62

1. Alanine Aminotransferase (ALT)2. Alcohol level3. Aldolase, serum4. Alkaline phosphatase, serum5. Amylase, serum6. Arterial blood gases (ABG)7. Aspartate Aminotransferase (AST)

8. Brain CT-scan9. Brain MRI10. Brain PET-scan11. Calcium, serum12. Carotid US-doppler13. Cerebral angiography14. Cerebro-spinal fluid exam 15. Complete Blood Count (CBC)16. C-Reactive Protein17. Creatine Phophokinase, serum18. Creatinine, serum19. Drug screening, serum

20. Drug screening, urine21. Echovirus, serology22. EEG recording23. Electrolytes (Na, K, Cl)24. g-Glutamyl Transferase25. Glucose, serum26. Lactate Deshydrogenase, serum27. Lyme disease, serology28. Protein electrophoresis, plasma29. T4, Free 30. Temporal artery biopsy31. Thyroid-Stimulating Hormone32. Total protein, plasma33. Urea, serum34. VDRL (Venereal Disease Research

Laboratory), serum35. No tests needed at this point in time

Short menu…

63

Clinical pathology (labs)Anatomical pathology (incl. biopsy)EEGEKGImaging (x-rays…)MicrobiologyNo tests needed at this point in time

…Question 4 (computer-based)

64

Questions – KFs matrix

KF1 KF2 KF3 KF4

Q1

Q2

Q3

Q4

65

Qs/ case

Single-q. cases, more casesbut… reliability went downGeneralisability study:maximize reliabilitywith 2 -3 q. / case

1 question/ case, not enough>3 redundant, wasting testing time

Norman et al, Med Ed., 2006

66

Practice… 1 & 2

67

III. Write Test material

1. Clinical scenariobased on KFs to be tested

2. Questions: 1 q./ KF

68


1. Prepare clinical scenariore: problem, situation, & KFs

2. Write test questionsre: KFs only

3. Choose a response format

4. Set scoring key

69

Open response WIsvs. Selected response SMs

Nbr responses WI < SM (+14%; cueing)

Difficulty WI > SM (-18pts; 54 – 72)

Variance WI > SMDiscrimination WI > SMMarginal cand. WI > SM

SMs: H&P, Lab. & InvestigationWIs: Dx, Rx & Management

Discrimination as a fct of the number of options?

5-Option15-Option

30 40 50 60 70 80 90 1000

5

10

15

20

PERCENT CORRECT SCORE

PER

CEN

T O

F S

TU

DEN

TS

30

5

50 60 70 80 90 100

20

0

10

15

40

5 options15 options

What is the most likely renal abnormality in children with nephrotic syndrome and normal renal function?

A 2-year-old boy has a 1-week history of edema. Blood pressure is 100/60 mm Hg, and there is generalized edema and ascites. Serumconcentrations are: creatinine 0.4 mg/dL, albumin 1.4 g/dL, and cholesterol 569 mg/dL. Urinalysis shows 4+ protein and no blood. What is the most likely diagnosis?

A 2-year-old black child developed swelling of his eyes and ankles over the past week. Blood pressure is 100/60 mm Hg, pulse 110/min, and respirations 28/min. In addition to swelling of his eyes and 2+ pitting edema of his ankles, he has abdominal distension with a positive fluid wave. Serum concentrations are: creatinine 0.4 mg/dL, albumin 1.4 g/dL, and cholesterol 569 mg/dL. Urinalysis: 4+ protein and no blood.

A B *C* D EU: 1 0 99 0 0

L: 8 1 90 1 0

U: 0 0 98 2 0

L: 5 2 82 8 1

U: 0 1 98 1 0

L: 10 9 66 10 5

Overall difficulty

94

88

84Case SM, Swanson DB. Constructing Written Test Questions for the Basic and Clinical Sciences, 1996. Page 58-9.

Discrimination (U-L) as a fct of‘fidelity’ of the question stem

72


1. Prepare clinical scenariore: problem, situation, & KFs

2. Write test questionsre: KFs only

3. Choose a response format

4. Set scoring key

73

Only score answers that relate to the Key Features

Give zero for: - wrong answer- too many options or- harmful actions

(e.g., zero for choosing more than 5 options; or doing a catherization when uncalled for)

Scoring

74

KF-1

Score Keyed responses

1 Status epilepticus (SE)(Note: both elements are required)

0 Wrong answer orwrote more than 2 diagnoses.

75

KF-3 Initial Mgmt

1 ABC .20 .25NS .20 .17Vit B .20 .17Glucose .20 .17Diaz+Phen .20 .25

0 Wrong answer

12

3

Best reliability

76

Scoring KFs, cases

KFs: Partial credit (better reliability)

…different weights do not increase reliability

CASE (unit of measurement)

Average KF scores/problem (KF equal weight)

e.g., (1 + 0 + 1 + .66) / 4 = 0.67

77

Scoring test – Pass/fail

Test as a whole

Average problem scores (Prob. equal weight)

e.g., (0.67 + .75 + …)/ 32 = .78 = 78%

Passing scoreContent (criterion)-based approach(modified Angoff procedure: prob. that minimally

competent candidate will get answer correct; 30min./pr.)

78

Justification & References

Brief justification (1 paragraph)for problem & KFs

References:Sirven, JI & Waterhouse, E. Management of status epilepticus.Am Fam Physician 2003;68:469-76* EBM: Evidence based medicine

79

Test material

Clinical scenarioTest questionsResponse formatScoring keysBrief justification: problem & KFs

References

80

Reliability

PMPs (1/2-day; ~ 10 cases) : ~.30

KF exams (1/2-day; 32-36 cases): ~.65 -.71

Spearman - Brown Prof. Formula

.80 45-50 cases = 1 day

81

Practice… 3 & 4

82

Test material

3. Response format4. Scoring keys5. Brief justification:

problem & KFs6. References

KF Casetemplate

1. Title of Clinical Problem & ID number2. Author(s)3. Life Span Period4. Clinical Situation(s)5. Location, setting6. Patient’s age & gender if relevant7. List of KFs: Given… candidate should: …8. KF – question table9. Brief justification for problem & KFs10. References (EBM)11. Date of revisions12. Case Scenario13. Questions & scoring keys

85

Life Span Periods

[ ] Neonatal, infant, pregnancy[ ] Child (1-11)[ ] Adolescent (12-18)[ ] Adult (19-64)[ ] Elderly (≥65)

86

Clinical situations

[ ] Undifferentiated complaint[ ] Single, typical problem[ ] Multiple, multi-system problem[ ] Urgent, life-threatening problem[ ] Prevention, health promotion

87

Location, setting

[ ] At home[ ] Office, ambulatory clinic[ ] Emergency room[ ] Hospital[ ] ICU

[ ] With support[ ] Limited support[ ] No support

88

Common Adm. Pitfalls

Failure to follow, read instructions: http://www.mcc.ca/english/examinations/qualifying_e1.html

Be aware of scoringOne case at a timeNo reversal once case is submitted

Read, practice before exam day: http://www.mcc.ca/english/examinations/qualifying_e1_practice.html

Practice cases (6 available)

90

Common Adm. Pitfalls

Running out of time…• Built-in time management toolKeeping track…• Flags that count # of responses• Provide [N] values, calculator

…in general, candidates do well

92

1986-91Medical Council of CanadaQualifying (MD) exam – graduating clerks

5-yr R & D : Q4 Project – Page & Bordage

1992 : Replaced PMPs with KF cases2003 : 10-yr review

93

Why Use KF Cases ?

Reliability:Focused cases – broader samplingContent Validity:Assess most important, case-specific decisionsRepresentative sample of casesFidelity & discrimination: better ID weaker candidates

Construct validity: Predictive of complaints Tamblyn et al, JAMA, 2007

Defensible pass-fail decisions

94

Measurement Basics…

Adequate &representative samplingSimple, precise scoring focusing on key decisionsFormats fit various purposesDefensible pass-fail decisions

95

MCC Qualifying Exam

3 parts:I. Knowledge (MCQs)II. Clinical decision making

KF cases (CDM)III. Performance (OSCE)

96

Breadth of testing on MCC QE

Med

Peds

SurgeryObGyn

Psych

PMCH

KF Cases

Data Acq.

Comm..Pr Solv

III- Performance OSCE

II- Clin. Dec. Making KFs

I- Knowledge MCQ

97

Dissemination

1992… Medical schools across Canada1991 Collège des Médecins du Québec (SOI)Swiss National Examination Board1993 College Physicians & Surgeons of Pakistan1995 American College of Physicians (MKSAP)1996 Amer. C. Colon & Rectal Surgeons (CARSEP)

9 cases – 30 KFs; Crb α=.95 overall .93 CRS1997 Royal Australian College General Practice2002 Hatala & Norman, clerkships(k=15; Crb α=.49)

98

Thank you !Merci !

99

References

Page, G. & Bordage, G.The Medical Council of Canada's Key Feature Project: A More Valid Written Examination of Clinical Decision-making Skills. Ac. Med., 1995, 70: 104-110.Page, G., Bordage, G. & Allen, T. Developing Key-Feature Problems and Examinations to Assess Clinical Decision-making Skills. Ac. Med., 1995, 70: 194-201.Bordage, G. & Page, G. An Alternative to PMPs: The “Key Features” Concept. Further Developments in Assessing Clinical Competence, 2nd Ottawa Conference, 1987, 59-75.Norman, G., Bordage, G, Page, G. & Keane, D. How Specific is Case Specificity? Med. Educ. 2006;40:618-23.E. Farmer and G. Page “A practical guide to assessing clinical decision-making using the key features approach”, Med. Educ, 2005: 39, 1188-94.

100

Hatala R, Norman GR. Adapting the key feature exam for a clinical clerkship. Med.Edu. 36: 160-165, 2002 Tambyn, R, Abrahomowizc, M, Dauphinee, D et al.Physician scores on a national clinical skills examination as predictors of complaints to medical regulatoru authorities. JAMA. 2007; 298:993-1001. Farmer EA, Hinchy J. Assessing general practice clinical decision-making skills using the key features approach. Australian Fam Phys. 2005; 34:1059-61 Bordage G. Brailowsky, C., Carretier, H., Page, G. Content validation of key features on a national examination of clinical decision-making skills. Acad Med. 1995; 70:276-81Bordage G. Carretier, H., Bertrand, R., Page, G.Comparing times and performances of French- and English-speaking candidates taking a national examination of clinical decision-making skills. Acad Med. 1995; 70:359-65

References

101

© G. Bordage, UIC, Chicago, 2007

Georges Bordage, MD, PhD

Professor, Department of Medical EducationCollege of Medicine, U. Illinois at Chicago808 South Wood, Chicago, IL 60612-7309

[email protected]://www.uic.edu/com/mcme/mhpeweb/http://www.mhpe-online.org

the “key features” approach to assess clinical...

Documents