constructing flawless mcqs for assessment and learning.pdf
TRANSCRIPT
-
8/17/2019 Constructing flawless MCQs for assessment and learning.pdf
1/37
Constructing flawless MCQs
for assessment and learning
Jack W. Strandhoy, PhD
-
8/17/2019 Constructing flawless MCQs for assessment and learning.pdf
2/37
Students can escape from
bad teaching, but not frompoor assessment (Boyd, 1995)
-
8/17/2019 Constructing flawless MCQs for assessment and learning.pdf
3/37
Evaluation
Learning Objective Content
Integrated Components of Teaching
and Learning
-
8/17/2019 Constructing flawless MCQs for assessment and learning.pdf
4/37
How can we improve our MCQs?
•
Purpose: formative, summative
•
Focus: important concepts, common,
clear-cut, or potentially catastrophicclinical problems
• Format: question type, no technical flaws
•
Level: strive to test application ofknowledge, higher order thinking
-
8/17/2019 Constructing flawless MCQs for assessment and learning.pdf
5/37
From D. DaRosa
-
8/17/2019 Constructing flawless MCQs for assessment and learning.pdf
6/37
Fundamental knowledge item
a. Phenoxybenzamine
b. Phenylephrine
c. Physostigmine
d.
Pralidoxime *
e. Prazosin
Which of the following dephosphorylates AChE?
-
8/17/2019 Constructing flawless MCQs for assessment and learning.pdf
7/37
Applied basic science mechanism item
a. Phenoxybenzamine
b. Phenylephrine
c. Physostigmine
d.
Pralidoxime *
e. Prazosin
Reactivation of cholinesterase shortly after toxicexposure to an OP insecticide can be accomplishedwith:
-
8/17/2019 Constructing flawless MCQs for assessment and learning.pdf
8/37
Clinical task item
a. Phenoxybenzamine
b. Phenylephrine
c. Physostigmine
d.
Pralidoxime *
e. Prazosin
You are treating a farm worker in the ED who wasaccidentally poisoned with an OP. You treat parasym.symptoms with atropine. Which drug do you choose toreactivate cholinesterase?
-
8/17/2019 Constructing flawless MCQs for assessment and learning.pdf
9/37
A 13 year-old boy is brought to the ED 5 days
after a camping trip. He has a petechial rash
on his arms and legs that began on the wrists
and ankles and spread in the last two days.
He has a headache, malaise, fever and chills.
A skin biopsy and Ab titer are positive for a
rickettsial infection. He is prescribed anantibiotic that acts by which of the following
mechanisms?
A. Acts as a detergent on the rickettsial cell wall
B. Binds at the 30S ribosome to prevent attachment of tRNA
C.
Depolymerizes microtubules in the cytosol
D.
Inhibits DNA gyrase
E. Prevents ergosterol biosynthesis
Integrating information
Note: Info from several disciplines but question is focused on one topic.
-
8/17/2019 Constructing flawless MCQs for assessment and learning.pdf
10/37
Other formats
•
Video clips, sound files, radiographic
images, virtual microscopy
•
Much easier with computerized examsthan on paper
• Can test higher order thinking
•
Being incorporated into NBME exams
-
8/17/2019 Constructing flawless MCQs for assessment and learning.pdf
11/37
Type A one-best-answer items
•
Focus on important concepts; common,
clear-cut, or catastrophic clinical problem
•
Test application of knowledge rather thanfactual recall
• Question answer should be clear and
answerable from just the stem
•
Distractors should be homogeneous
• Avoid technical flaws
-
8/17/2019 Constructing flawless MCQs for assessment and learning.pdf
12/37
Questions with structural flaws
•
Which of the followingpairs has won the most
Abby awards?
• A. Jones & Smith
• B. Smith & Taylor
•
C. Smith & White
• D. White & Allen
-
8/17/2019 Constructing flawless MCQs for assessment and learning.pdf
13/37
Questions with structural flaws
•
Which of the followingpairs has won the most
Abby awards?
• A. Jones & Smith
• B. Smith & Taylor
•
C. Smith & White
• D. White & Allen
• Ans = C
-
8/17/2019 Constructing flawless MCQs for assessment and learning.pdf
14/37
Questions with structural flaws
•
Which of the followingpairs has won the most
Abby awards?
• A. Jones & Smith
• B. Smith & Taylor
•
C. Smith & White
• D. White & Allen
• Ans = C
•
Which of the followingdrug combinations is
most likely to produce
ototoxicity?
• A. tetracycline & HCTZ
•
B. gentamicin & furosemide• C. ampicillin & furosemide
• D. gentamicin & triamterene
• Ans = B
-
8/17/2019 Constructing flawless MCQs for assessment and learning.pdf
15/37
Questions with structural flaws
•
Grammatical clues – a and an
•
Longest answer is correct
•
Absolutes – always, never [usu. wrong]•
Vague, weasel words – usually, often,
frequently
•
Negatives, double negatives,except
or
not statements
-
8/17/2019 Constructing flawless MCQs for assessment and learning.pdf
16/37
Questions with structural flaws
•
Answers too long or complicated
•
Word repeats as clues
•
Inconsistent, awkward structure•
Terms like none of the above
-
8/17/2019 Constructing flawless MCQs for assessment and learning.pdf
17/37
Interpretation of imprecise terms
NBME, 2003
-
8/17/2019 Constructing flawless MCQs for assessment and learning.pdf
18/37
Questions with structural flaws
•
Stem should provide all of the information
necessary to answer the question
•
Flawed: – Which of the following is true about
pseudogout?
• Use longer stems and shorter options
rather than the other way around
-
8/17/2019 Constructing flawless MCQs for assessment and learning.pdf
19/37
Elementary School Quiz
•
True/False:
–
A shape that has 4 sides is called a square
–
An apple is a sphere – The numerical value of the term ! is
3.1415926585
-
8/17/2019 Constructing flawless MCQs for assessment and learning.pdf
20/37
Medical School Quiz
•
True/False:
–
The incidence of cystic fibrosis is 1:2000
–
Children with CF usually die in their teens – Males with CF are sterile
-
8/17/2019 Constructing flawless MCQs for assessment and learning.pdf
21/37
Be aware:
All of the following concerning H. pylori and peptic
ulcer dz (PUD)are correct EXCEPT: A.
Infection w/ this organism is seen in themajority of pts with PUD
B.
Infection enhances gastric acid secretion
C.
Only a minority of infected pts develop PUDD.
The organism secretes urease that breaksdown urea into NH3 and CO2, which forms thebasis of a diagnostic test for the presence ofthe organism.
= multiple T / F
-
8/17/2019 Constructing flawless MCQs for assessment and learning.pdf
22/37
Type A: One best answer
•
Which of the following most approximates
a sphere?
–
Apple –
Banana
–
Frisbee
–
Hula hoop
–
Pretzel
-
8/17/2019 Constructing flawless MCQs for assessment and learning.pdf
23/37
•
Which of the following most approximates
a sphere?
–
Apple –
Basketball
–
Egg
–
Charlie Browns head
–
Watermelon
-
8/17/2019 Constructing flawless MCQs for assessment and learning.pdf
24/37
Type R questions:
Extended Matching•
Theme: broad or narrow
•
Options list: homogeneous, alphabetical.
Words or short phrases•
Lead in statement: match how?
•
At least two stems (small vignettes). Make
similar in structure within a set
•
Make sure theres only 1 best answer and
at least 4 distractors/stem
-
8/17/2019 Constructing flawless MCQs for assessment and learning.pdf
25/37
A. gentamicin G. penicillin VB. hydrochlorothiazide H. prednisone
C.
lithium carbonate I. primaquineD. metronidazole J. vancomycin
E.
allopurinol F. morpine sulfate
For each pt. description, select the MOST likely drug that was administered.
A 9 yo boy is admitted with gram (+) sepsis. An antibiotic is quicklyinfused IV. The child complains of a flushed feeling and you note
redness around the face and neck.
A 35 yo woman psychiatry patient complains of excessive urination andthirst. Urine osmolality was 100 mOsm/Kg water.
A 35 yo woman with lupus nephritis is treated with chronic medication.She develops gastric irritation, altered fat deposition to her face and
back, and osteoporosis.
Type R questions
-
8/17/2019 Constructing flawless MCQs for assessment and learning.pdf
26/37
A. Acetaminophen P. Sulfasalazine
B. Amiodarone Q. Tetracycline
C. ACE inhibitors ….. R. Verapamil
A 56 yo man with recurrent ventricular arrhythmias
began taking an antiarrhythmic drug 5 mo ago. He nowhas progressive cough, dyspnea and low-grade fever.
ESR is increased. CXR shows diffuse interstitial
pneumonia. PFT show that diffusing capacity for CO is
decreased. {amiodarone}
A 62 yo man with COPD begins therapy with anantihypertensive drug. Two wks later, he has marked
worsening of dyspnea and clearly audible wheezing.
{propranolol}
NBME examples, Type R
For each pt, select the drug most likely to have caused adverse effect
-
8/17/2019 Constructing flawless MCQs for assessment and learning.pdf
27/37
Adjust the difficulty of type R
questions with the optionse.g., antibiotics vs cephalosporins
-
8/17/2019 Constructing flawless MCQs for assessment and learning.pdf
28/37
Available from NBME.org Constructing Written Test
Questions for the Basic and
Clinical Sciences 3rd ed.
Susan Case & David Swanson
181 pages
-
8/17/2019 Constructing flawless MCQs for assessment and learning.pdf
29/37
Template for clinical vignette
style questions•
Item stem (patient vignette): describe a patientwith a problem – Age, gender; Site of care (ED, etc.)
–
HPI: Presenting complaint; Duration – Pt. hx; family hx; Physical findings, studies, Dx, initial
treatments
•
Lead-in: e.g., Which of the following is the mostlikely diagnosis?
•
Options: List of diagnoses
•
Long stem, short options better than v/v
-
8/17/2019 Constructing flawless MCQs for assessment and learning.pdf
30/37
Additional Item Templates
•
Which of the following is the most likely
mechanism for (drug) in patients with (dz)?
•
A patient has (signs, symptoms). Which ofthe following (drugs) is best treatment?
Which drug should be avoided?
• A patient has (side effects). Which drug is
she most likely taking?
-
8/17/2019 Constructing flawless MCQs for assessment and learning.pdf
31/37
A 13 year-old boy is brought to the ED 5 days
after a camping trip. He has a petechial rashon his arms and legs that began on the wrists
and ankles and spread in the last two days.
He has a headache, malaise, fever and chills.
A skin biopsy and Ab titer are positive for a
rickettsial infection. He is prescribed an
antibiotic that acts by which of the following
mechanisms?
A. Acts as a detergent on the rickettsial cell wall
B. Binds at the 30S ribosome to prevent attachment of tRNA
C.
Depolymerizes microtubules in the cytosolD.
Inhibits DNA gyrase
E. Prevents ergosterol biosynthesis
Integrating information
Note: Info from several disciplines but question is focused on one topic.
-
8/17/2019 Constructing flawless MCQs for assessment and learning.pdf
32/37
Using Clinical Vignettes
No vignette:
The most likely renal abnormality in children withnephrotic syndrome and normal renal function is:
A. Acute poststreptococcal glomerulonephritis
B. Hemolytic-uremic syndrome
C. Minimal change nephrotic syndrome*D. Nephrotic syndrome due to focal and segmental
glomerulosclerosis
E. Schönlein-Henoch purpura with nephritis
A B C D E
1 0 99 0 0 TOP 10%
8 1 90 1 0 BOTTOM 10%
-
8/17/2019 Constructing flawless MCQs for assessment and learning.pdf
33/37
Clinical summary vignette:A 2-year-old boy has a 1-week history of edema.
Blood pressure is 100/60. There is generalized
edema and ascites. Serum creatinine is 0.4 mg/dL, albumin 1.4 g/dL, and cholesterol 569 mg/
dL Urinalysis shows 4+ protein and no blood.The most likely diagnosis is:
A B C D E
1 0 98 2 0 TOP 10%
5 2 82 8 1 BOTTOM 10%
-
8/17/2019 Constructing flawless MCQs for assessment and learning.pdf
34/37
Clinical observations vignette:
A 2-year-old boy has developed swelling of his ankles andaround his eyes for the past week. Blood pressure is
100/60, pulse 110/min, and respirations 28/min. In
addition to swelling around his eyes and 2+ pitting
edema of his ankles, he has abdominal distention with
a positive fluid wave. Serum creatinine is 0.4 mg/dL,albumin 1.4 g/dL, and cholesterol 569 mg/dL Urinalysis
shows 4+ protein and no blood. The most likely
diagnosis is:
A B C D E
0 1 98 1 0 TOP 10%
10 9 66 10 5 BOTTOM 10%
-
8/17/2019 Constructing flawless MCQs for assessment and learning.pdf
35/37
Research vignettes
•
Item stem – lab data, graphs, dose-
response curves
•
Lead in – – Most likely interpretation?
–
Next logical experiment?
–
Best statistical analysis of the data?
–
Best control experiment?
• Options -
-
8/17/2019 Constructing flawless MCQs for assessment and learning.pdf
36/37
Guidelines for good questions•
Focus on important concepts; dont wastetime testing trivial facts
•
Match exam question to objective incontent and application; encourage higher
order thinking
•
Screen for technical flaws, ambiguities
• Examine poor question stats for reasons
•
Explore multidisciplinary vignettes andquestions; require data analysis foranswers
-
8/17/2019 Constructing flawless MCQs for assessment and learning.pdf
37/37
Guidelines for good questions
•
You should be able to put your hand over
the answers, read the stem, answer it
based on the information, then choose the
answer from the list.
• Good questions should test more than just
knowledge – integration, synthesis,
judgement.