how can we reduce diagnostic error? - fmh€¦ · how can we reduce diagnostic error? ... accuracy...
TRANSCRIPT
+ HOW CAN WE REDUCE
DIAGNOSTIC ERROR?
GEOFF NORMAN, PhD
McMaster University
+ Strategies to reduce errors
GENERAL INSTRUCTIONS
Slowing down, be attentive, be systematic etc.
HEURISTIC - BASED STRATEGIES
“Cognitive Forcing Strategies” (Croskerry)
KNOWLEDGE- BASED STRATEGIES
During Diagnostic Process
Reflection (Mamede & Schmidt, Monteiro)
As a Learning Strategy
+
“…errors of intuitive judgment involve failures of both
systems: System 1, which generated the error, and System 2,
which failed to detect and correct it.”
D. Kahnemann, 2004
+
`“Most errors occur with [System] 1 and may to
some extent be expected whereas [System] 2
errors are infrequent and unexpected…”
P. Croskerry, 2009
+ GENERAL INSTRUCTIONS
“What can be done about biases? …The short answer is that
little can be achieved without a considerable investment of
effort… S1 is not readily educable…”
“The way to block errors that originate in System 1 is simple in
principle: recognize that you are in a conceptual minefield,
slow down, and ask for reinforcement from System 2.
[emphasis ours]“
(Thinking fast and slow, p.417 ).
+
Do errors result from rapid pattern recognition (System 1)
processes;
Does slower, more methodical problem-solving reduce
errors?
+ Norman, Brooks, Rosenthal, 1998
Accuracy and Time in Dermatology
100 slides in 20 categories
Students, clerks, residents, GPs, Dermatologist
Accuracy and Response Time
Accuracy by Educational
Level
0
10
20
30
40
50
60
70
80
90
100
Stud
ent
Clerk
Res
iden
tGP
Der
mto
logist
% c
orre
ct
Response time by
Educational Level
0
5
10
15
20
25
30
Stud
ent
Clerk
Res
iden
tGP
Der
mto
logist
Res
ponse
Tim
e
CorrectIncorrectD K
+ Sherbino, 2010
Accuracy and Time in Internal Medicine
75 Canadian PGY2 internal medicine
20 written I.M. cases
“Proceed as rapidly as you can but try not to make any
mistakes”
+ Rapid Instructions
You are to make your diagnosis and type it
in as quickly but as accurately as possible.
Case information will appear on one
screen, and you click on a button to go to
the diagnosis screen. You may spend as
much time as you wish reading the case
information, but remember that you only
have 30 minutes to complete all the cases.
R = -.55
ACCURACY VS. READING TIME
+
Accuracy is associated with
shorter time
Longer times reflect
uncertainty
+ Norman et al., 2011
Controlled trial
Rapid reasoning (2010) vs. Systematic approach (2011)
20 cases
PGY2 residents
96 Rapid (2010)
108 Systematic (2011)
+ Systematic Instructions
Be careful and thorough. Try not to skip
anything. Consider all the data. Take as long
as you want on that screen, but when you click
on button to go to diagnosis screen, you can’t
go back to the information. There is a counter
in the upper right corner that tells you how
many cases you’ve done. You are to consider
all the data and then make your diagnosis and
type it in.
+ Comparison of Rapid and
Systematic Instructions
+ Comparison of Rapid and
Systematic Instructions
+ Heuristic Based Strategies
+ Cognitive Forcing Strategies
“Becoming alert to the influence of bias requires maintaining
keen vigilance and mindfulness of one’s own thinking. When a
bias is identified by a decision-maker, a deliberate decoupling
from the intuitive mode is required so that corrective
“mindware” can be engaged from the analytical mode.”
Croskerry, 2013
+ CAN CLINICIANS IDENTIFY WHEN
A BIAS IS PRESENT?
Zwaan et al, 2016
+ Bias Definitions
Anchoring The tendency to perceptually lock onto
salient features in the patient’s initial
presentation too early in the diagnostic
process, and failing to adjust this initial
impression in the light of later information.
Availability Heuristic The disposition to judge things as being more
likely or frequently occurring, if they readily
come to mind. Thus recent experience with a
disease may inflate the likelihood of its being
diagnosed. Conversely, if a disease has not
been seen for a long time (i.e. is less available)
it may be underdiagnosed.
Base Rate Neglect The tendency to ignore the true prevalence of
a disease, either inflating or reducing its base
rate, and distorting Bayesian reasoning
Confirmation Bias The tendency to look for confirming data to
support a diagnosis rather than look for
disconfirming evidence to refute it, despite
the latter often being more persuasive and
definitive
Premature Closure The tendency to apply premature closure to
the decision-making process, accepting a
diagnosis before it has been fully verified.
The consequences of the bias are reflected in
the maxim - ”when the diagnosis is made, the
thinking stops.”
Representativeness Bias The tendency to look for prototypical
manifestations of disease. Restraining
decision-making along pattern-recognition
lines leads to atypical variants being missed.
+ Subjects
Members of Society to Improve Diagnosis in Medicine
(mailing list)
Practicing physicians
Initial approach and consent (n = 75)
Web based administration (n =37)
+ Methods
Created 50/50 cases
Two approximately equally likely diagnoses
Experimental manipulation of test positive / negative
Measurement
Was a diagnostic error committed?
Which of the following biases were present?
+ PE or Pneumonia
History of Present Illness A 43-year old woman was brought to the emergency
department by her husband at 0200 in the morning because of shortness of breath.
The dyspnea occurred suddenly at 1100 pm …The patient complained of nausea …
She has had no recent surgery.
Past Medical History
…The ECG demonstrates non-specific ST depression in V3-V6.
A Chest X-ray was ordered to diagnose pneumonia.
The chest x-ray demonstrated an opacity …The patient was prescribed a course of
antibiotics, …and instructed to follow-up …
Two days later, the patient was seen in the clinic for follow-up with
her primary care physician. She reported marked improvement in
her chest pain and shortness of breath, as well as resolution of
her fevers and chills. She was instructed to complete her course
of antibiotics.
+ PE or Pneumonia
History of Present Illness A 43-year old woman was brought to the emergency
department by her husband at 0200 in the morning because of shortness of breath.
The dyspnea occurred suddenly at 1100 pm …The patient complained of nausea …
She has had no recent surgery.
Past Medical History ….The ECG demonstrates non-specific ST depression in V3-V6.
A Chest X-ray was ordered to diagnose pneumonia.The chest x-ray demonstrated an
opacity …The patient was prescribed a course of antibiotics, …and instructed to
follow-up …
Two days later, the patient was seen in the clinic for follow-up with
her primary care physician. She reported continued chest pain
and shortness of breath, and several episodes of hemoptysis. A
CT Pulmonary Angiogram was ordered to diagnose a
pulmonary embolism. This demonstrated a pulmonary
embolism in the segmental pulmonary artery of her left lower
lobe. A heparin drip was started and the patient was admitted to
the hospital.
DIAGNOSIS A DIAGNOSIS B
Pneumonia Pulmonary embolism
Acute MI Type A aortic dissection
Tubo-ovarian abscess Appendicitis
Subarachnoid hemorrhage Meningitis
Kidney stone Type B aortic dissection
Pyelonephritis AAA
Pancreatitis Cholecystitis
Cellulitis DVT
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%P
erc
en
t o
f R
es
po
ns
es
Specific Bias with Consistent and Inconsistent Outcome
Consistent Inconsistent
Consistent Inconsistent Relative
Increase
KAPPA*
Anchoring 35% 70% 100% 0.0
Availability 25% 55% 120% .025
Confirmation 35% 62% 77% .024
Base Rate 11% 28% 154% .063
Premature Closure 39% 88% 125% .046
Representativeness 26% 45% 73% .044
+ COGNITIVE DEBIASING –
Effect on Errors
Smith and Slack (2015)
Sherbino et al., (2012, 2013)
+
(Smith and Slack, 2015)
19 Fam Med residents
Debiasing workshop
Evaluation by preceptor with actual
patients before and after
+ Accuracy and Ability to Recognize
Bias
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
PRE POST
Accuracy
Recognition
+
(Sherbino et al., 2013)
198 students
Intervention 145
Control 46
Instruction on two biases
Search satisficing
Locate the second lesion on X ray / ECG
Availability
Identify the rare diagnosis
Test on 6 cases
Near transfer, far transfer, “False positive”
+ Search Satisficing
0%
20%
40%
60%
80%
100%
120%
CFS Control
PRIMARY
SECONDARY
+ Availability
0%
20%
40%
60%
80%
100%
120%
CFS Control
Common Inc
Uncommon Corr
+ SUMMARY
Cognitive debiasing strategies:
No apparent agreement on presence of absence of specific
biases when cases are not preselected to illustrate a bias
Students can learn to identify biases in contrived situations (e.g.
Reilly, 2013; Ogdie, 2012)
No effect of cognitive debiasing training on diagnostic errors
+
“ If you have not heard about myasthenia
gravis, you cannot cognitively debias your
way into that diagnosis. You can spend all
day in system 2 and collect more and more
information, but if you do not have a well-
developed illness script that contains
atypical manifestations of heart failure, you
will never recognise it. In the realm of expert
performance, knowledge is king.”
Gurpreet Dhaliwal, BMJ Qual Saf, 2016
+ KNOWEDGE-BASED STRATEGIES
During the Diagnostic Process
Reflection
- Structured
- Self-initiated
As a Learning Strategy
+ Structured Reflection
(Mamede & Schmidt)
With case description in front of them:
Write down most likely diagnosis
Write down alternative diagnoses
List findings
Supporting
Against
Not present
Rank diagnoses in order of likelihood
+
(Psych Res 2010)
34 PGY2 residents
50 medical students
6 complex, 6 easy cases
Analytic reflective reasoning vs.
Write down first thing that comes to mind
+ Reflection vs. “First thing”
Residents
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
Simple Complex
Reflection
First thing
+ Reflection vs. “First thing”
Students
0%
10%
20%
30%
40%
50%
60%
70%
Simple Complex
Reflection
First thing
+
Mamede and Schmidt, 2012
46 Year 4 med students
4 learning cases (2/2) , 6 test cases (2/2/2)
Immediate test / Delayed (1 week)
+ Reflection vs. “First thing”
Immediate and Delayed
0%
10%
20%
30%
40%
50%
60%
70%
Immediate Delayed
Reflection
First thing
+ Do clinicians know when to reflect?
+ Self-initiated Reflection (Monteiro, 2013)
47 residents
27 PGY1, 15 PGY2, 23 PGY3
16 cases
First Impression (Fast) then Reflection
After first pass through cases, review case again and either confirm or
revise diagnosis
+
Pass 1, 746 diagnoses / 316 correct
Pass 2, 62 diagnoses (8%) revised
+ Effect of Revision on individual
diagnoses
0
0.2
0.4
0.6
0.8
1
1.2
1.4
Pass 1 Pass 2
Revised
Not Revised
+ Effect of Revision overall
0
5
10
15
20
25
30
35
40
45
50
Before revision After revision
+ Summary
Structured reflection appears to have positive effect, of
about 10%.
However, results are apparently affected by context, expertise,
case difficulty.
“Reflection” strategy very time consuming
Spontaneous reflection has small overall effect
8% revise diagnosis
Minimal overall increase (1.3%)
+ Instruction in Clinical Reasoning
Whole case, low fidelity (Durning)
Mixed Practice (Hatala, 1999)
Explicit Direction for S1 and S2 (Norman,
Brooks, Colle, 2000)
+ Durning, 2011
133 Year 2 med students
Instruction on 3 cases using:
Written case
DVD video case
Standardized Patient
Test using:
OSCE
Written knowledge quiz
Video quiz
+ Effect of Paper vs. DVD vs Std Pt
Instruction on Performance
0
10
20
30
40
50
60
70
80
OSCE Written Video
Paper
DVD
StdPt
+ Mixed vs. Blocked Practice
In the face of ambiguous features (which are subject to
reinterpretation), and multiple categories, students must learn
the features which discriminate one category from another, not
those which support a particular category
+ Mixed vs. Blocked Practice
(Hatala, 2000)
ECG Diagnosis -- 3 categories
6 examples / category
Blocked
Review, then 6 examples/category
Mixed
Review, 2/category, 12 (4 x 3) practice
TEST
6 new ECGs
Accuracy -- %
0
5
10
15
20
25
30
35
40
45
50
Mixed Blocked
+ Explicit Instruction to Use S1 and S2 Norman, Brooks Colle, 2000
32 Undergrad Psychology students
11 disorders, rules + examples
Test -- 10 new ECGs
+ Instructions
Think of the first thing that comes to mind, then consider
features
vs.
Gather all the data then arrive at diagnosis
Diagnostic Accuracy
System 1 + 2 System 2
Diagnostic Accuracy
Resident
Clerk
System 1 + 2 System 2
+ CONCLUSION (1)
WORKPLACE-BASED (REAL TIME) STRATEGIES
Error reduction based on cognitive biases is ineffective
Experts cannot agree on specific bias
Instruction based on identification of bias does not reduce error
Error reduction based on reorganizing knowledge has small effect, but requires self-recognition of error
+ CONCLUSION (2)
INSTRUCTIONAL STRATEGIES
A number of strategies to improve teaching
of reasoning
Simple cases
Mixed practice
Combined reasoning (S1 and S2)