kids, cats and concepts: toward a grand unified theory of
DESCRIPTION
TRANSCRIPT
Kids, Cats and Concepts: Toward a Grand Unified Theory of Thinking
Geoff Norman
The Goal
To link research in three domains:Dual processing models of thinking Exemplar and prototype models of categorization / concept formation
Expertise and clinical reasoning
to a greater understanding of human information processing
How I got there
Distant Studies of clinical problem-solving
Intermediate Role of experience in clinical reasoning
Recent Diagnostic errors and “dual processing”
A Difficult Diagnostic Task
An Easy Diagnostic Task
The rule is insufficient for the classification task
But we can do the task quickly, accurately, and effortlessly
HOW?
The Role of Similarity
DUAL PROCESSINGTwo basic strategies
System 1Based on holistic similarity to prior examples“Exemplar theory” (more later)
System 2Based on underlying conceptual characteristics “Causal models”
System 2 thinking
Playing by the rules
Analytic View of Expertise
“The matters that set experts apart from beginners are symbolic, inferential, and rooted in experiential knowledge…Experts build up a repertory of working rules of thumb or “heuristics” that, combined with book knowledge, make them expert practitioners.”
E. Feigenbaum. The fifth generation: artificial intelligence and Japan's computer challenge to the world. 1983
System 1 thinking
I’ve seen it before and here it comes again
Successful categorizationFrom 2-D abstract representation
without analysis of featureswithout language
Successful generalizationTo other 2 D abstraction in atypical orientation
The Non- Analytic View
“We must be prepared to abandon the traditional view that runs from Plato to Piaget and Chomsky that a beginner starts with specific cases and… abstracts and interiorizes more and more sophisticated rules.It might turn out that skill acquisition moves in just the opposite direction; from abstract rules to particular cases.”
H.L. Dreyfus, 2002
OutlineDual processing Concept formation and categorizationExpertise and Clinical Reasoning
Applications -- ImplicationsLevels of ProcessingPerceptionTransferAging and reasoningIntelligence
Three Literatures
Concept formation(categorization)
Clinical Reasoning
Dual Processing(Thinking)
Medin,Brooks
Norman,Schmidt
Stanovich,Evans,Kahnemann
Dual Processing
System 1Rapid, unconscious, based on concrete similarity, “just” pattern recognition
System 2Slow, logical, conceptual, energy-intensive,
CHARACTERISTICSSystem 1 System 2
UnconsciousImplicitAutomaticEffortlessRapidHolistic,Old (evolution)Contextualized
ConsciousExplicitControlledEffortfulSlowAnalyticNew (evolution)Abstract
Neuroanatomy of System 1,2
System 1“right inferior prefrontal cortex”
Evans, 2008
“Involves hippocampus”
Smith & DeCoster, 2000
System 2“ventral medial prefrontal cortex”
Neurophysiology of System 1,2
Glucose dose (vs. Placebo)
Shift of processing strategy toward System 2 (more energy demand) with glucose load
(Attraction effect - 17% vs. 47%)
(Masicampo & Baumeister, 2008)
Mental representations
SYSTEM 1Abstract concepts
Feature list, probability, causal mechanism, process
SYSTEM 2???????
Categorization / Concept Formation
Exemplar Theory - Medin, Brooks
Categories consist of a collection of prior instances
identification of category membership based on availability of similar instances
Retrieval process is “non-analytic (unaware), hence can result from objectively irrelevant features
Retrieval process is not deliberate, not available to introspection
(Like System 1)
Dual Processing in Medicine
From Process to Knowledge(Analytical and Experiential)
The beginnings - clinical reasoning as
a process“Hypothetico-deductive method”
(Elstein, Shulman, Sprafka, 1977)
Expert (and novice) clinicians generate multiple diagnostic hypotheses early in the encounter then gather data to confirm (usually) these hypotheses
Does hypothesis predict accurate
solution?
0102030405060708090
100
Correct on chart
Present Absent
Correct hypothesis?
Barrows, Neufeld, Norman, 1981
Where do hypotheses come from?
“Medical experts differed from novices in that they generated better hypotheses……
and we don’t know why!”
A. ElsteinDx Error Conference
May 31, 2008
Expert Physicians and Dual Processing
To what extent does the:
formal knowledge of medical school
vs. experiential knowledge of practice
contribute to expertise
Schmidt & Norman, 1991
Basic ScienceMechanisms
Basic ScienceMechanisms
Basic ScienceMechanisms
ExamplesClinical Rules
Clinical Rules
Novice Intermediate Expert
System 2
System 1
Who do you pick?
Dr. JW completed the specialty exam last year and stood 14th in the country.
Dr. WS completed the specialty exam 6 years ago. At the time, she was in the top 1/3 of all candidates.
The Conundrum
Why do we prefer the candidate with apparently less “competence” but much more experience?
What did she get from 10 years of experience?
10 years of experiences
(System 1 knowledge)
BUT
Every measure of formal (System 2) knowledge decays right after graduation
Day and Norcini, 1988
Years since Graduation
420
440
460
480
500
520
540
<20 21-24 25-29 30-34 35-39
Evidence of System 1 in Diagnostic
Reasoning
Visual Diagnosis and Response Time
STUDY100 slides in 20 categories
Students, clerks, residents, GPs, Dermatologist
Accuracy and Response Time
Accuracy by Educational Level
0
10
20
30
40
50
60
70
80
90
100
StudentClerk
Resident
GP
Dermtologist
% correct
Response time by Educational Level
0
5
10
15
20
25
30
StudentClerk
Resident
GP
Dermtologist
Response Time
CorrectIncorrectD K
Evidence of Exemplars
Effect of Similarity (Allen, Brooks, Norman, 1992)
24 medical students, 6 conditions
Learn Rules Practice rules
Train Set A Train Set B(6 x 4) x 5 (6 x 4) x 5
Test (9 / 30)
Accuracy by Bias Condition
0
10
20
30
4050
60
70
80
90
Bias Corr Bias Incorr
Correct
Incorrect
Other
Is it just visual similarity?
If it’s “non-analytic” does it apply to objectively irrelevant features?
ECG InterpretationHatala et al, 1999
Medical students/ Fam Med residentsPRACTICE (4/4 + 7 filler)
middle aged banker with chest pain OR elderly woman with chest pain
Anterior M I
TEST ( 4 critical + 3 filler)Middle aged banker with chest pain
Left Bundle Branch Block
RESULTSPercent of Diagnoses by
Condition
0
10
20
30
40
50
Correct PriorDiagnosis
Percent mentioning
Bias
No bias
CONCLUSIONS - Medical Diagnosis and Dual
Processing Experiential knowledge is a major contributor to diagnostic expertise
Categories and concepts are based on our specific experience with the world
These specific experiences are accessed and used without awareness
When do experts use system 2?
Analytic reasoning and Diagnosis
Invoked for confirmation in all Dx encounters
Analytic knowledge of many forms:Illness scriptsSymptom-disease probabilitiesSemantic axesFeature lists (e.g. DSM 4)
Where Do Clinicians Use Basic Science?
Most use basic science rarely?Observational studies (Schmidt, Patel)
Some use basic science some of the time Difficult problems in nephrology
Some use physiology ALL the timeIntensivists, anesthesiology
Most use it rarely(Patel, Schmidt)
Clinicians rarely use basic science explanation in routine practice.
While they may possess the knowledge, it remains “encapsulated” until mobilized for specific goals (to solve specific problems) (Schmidt, HG)
Some Use it with Difficult Cases
(Norman, Brooks, Trott, Smith)
When experts are confronted with difficult cases, do they revert to causal reasoning?
Experimental Design
R1 --GP R2 -- IM Nephroln=4 n=4 n=4
Clinical Cases k = 8
Explain and Diagnose
Diagnostic Accuracy
0
0.2
0.4
0.6
0.8
1
R1-FM R2-IM Nephrol
Causal Explanations
0
0.5
1
1.5
2
2.5
R1-FM R2-IM Nephrol
No of Diagnoses / Investigations
0
1
2
3
4
5
6
R1-FM R2-IM Nephrol
Conclusions - Use of Basic Science
In difficult diagnostic situations, clinicians use causal physiological knowledge and analytic reasoning
Expertise associated with more coherent explanations, better diagnosis
Dual Processing and Experience
With increasing experience, do people rely more or less on System 1 -- Non-analytic reasoning?
Studies of Relative Experts
(Moruzi, Brooks, Norman, 2003)Dermatologists/ GPs / residents36 slides (typical / atypical)
Condition AVerbal description of slide (verbal)
then photo (visual + verbal)
Condition B Photo only (visual)
Diagnostic Accuracy
0
10
20
30
40
50
60
70
80
90
Verbal Verbal+Visual Visual
Resident
Diagnostic Accuracy
01020
30405060
708090
Verbal Verbal+Visual Visual
G.P.
Dermatol
Resident
Hatala et al.
ECG DiagnosisPrior match / unmatch history
Postgraduate residents and med students
RESULTSPercent of Diagnoses by
Condition
05
101520253035404550
Correct Prior
Diagnosis
Percent mentioning
BiasNo bias
Medical Students
Dual Processing and Instruction
Role of Instruction in reasoning
Since NA (System 1) reasoning occurs at all levels, is effective, is “automatic”
You can’t: tell student to not do it tell student to beware of biases tell student to think of better diagnoses
Does a coordinated strategy
improve accuracy? Norman, Brooks, Colle (ECG)
Schmidt and Mamede (Gen Medicine)
Ark & Eva, (ECG)
Norman, Brooks Colle, 2000
Contrast instructions to: Think of the first thing that comes to mind, then consider features
vs. Gather all the data then arrive at diagnosis
32 Undergrad Psychology students 11 disorders, rules + examples Test -- 10 new ECG’s
Diagnostic Accuracy
0
10
20
30
40
50
60
70
Pattern + Rules Rules
Schmidt & Mamede, 2005
42 I.M. residents 16 written cases --- simple / complex Within subject/case design Instructions:
“First thing that comes to mind”vs.
“Hypotheses, findings for/against, differential, ….”
Diagnostic Accuracy
0
10
20
30
40
50
60
70
80
Exemplars Rules
Simple
Complex
ECG Diagnosis - Ark & Eva
48 undergrad psychology students 8 ECG diagnoses (A/A’, B/B’, C/C’,D/D’)
Instructions• Compare and contrast vs. Sequential• Combined Analytical/Non-analytical vs.
usual approach
Test20 ECG’s (10 old, 10 new)Immediate / 1 week later
Effect of Examples and Instructions
on New Cases after One Week
Ark & Eva, 2005
3035404550556065707580
Compare No Compare
CombinedNo Instruction
Conclusions - Dual Processing and
Diagnosis Evidence that clinicians access both kinds of knowledge/ use both processes
Evidence that with increasing experience, greater reliance on system 1
Evidence that students benefit from explicit instruction to use both
Dual Processing and Thinking
DP and levels of processing
DP and perception
DP and transfer
DP and aging
Dual Processing and Levels of Processing
Are conceptual structures and deep processing an underpinning to development of System 1 (non-analytic) knowledge?
(Dreyfus)
Role of Basic Science in Novice Reasoning
(Woods, Brooks, Norman, 2003)4 neurology / muscular diseases
36 medical students Basic Science or Symptom/Disease probability
Measurement
Diagnostic Test15 cases, 4-6 features
Administered at 0, 7 days
Score on Dx Test
30
35
40
45
50
55
Immediate 1 Week
Feature ListBasic Sci
Score on Dx Test
30
35
40
45
50
55
Immediate 1 Week
Feature ListBasic Sci
Score on Dx Test
30
35
40
45
50
55
Immediate 1 Week
Feature ListBasic Sci
Dual Processing and Perception
Word Superiority Effect
Higher - level concepts (words) in memory facilitate recognition of elements of words and pseudo-words:
R I N KB I N KN R I K
- possibly because of rapid (top-down) then slow (bottom up) processing
Influence on Feature Interpretation
Diagnostic hypotheses arise from pattern recognition processes based on similarity to prior examples
In situations of feature ambiguity, hypotheses may influence what is seen
Influence of Diagnosis on Feature Perception (LeBlanc
et al)20 residents, 20 final year students
8 photos of classical signs from clinical diagnosis textbooks
Correct history and diagnosisvs.
Incorrect history and diagnosis
RESULTSDiagnostic Accuracy by
Bias
0
10
2030
40
50
6070
80
90
Correct Alternate
StudentResident
RESULTSNumber of Features of Correct Diagnosis by
Condition
00.050.1
0.150.2
0.250.3
0.350.4
0.450.5
Correct Alternate
Diagnosis
No. of Features
StudentResident
RESULTSNumber of Features of Alternate Diagnosis by
Condition
0
0.05
0.1
0.15
0.2
0.25
Correct Alternate
Diagnosis
No. of Features
StudentResident
Dual Processing and Transfer
Although medical (and other) study is directed at conceptual learning, use of conceptual knowledge to solve problems (transfer) is rare and difficult.
WHY????
Spontaneous Transfer
8 high performing undergrad (Health Sciences) students.
3 principles (Laplace, Poiseuille, Starling) 12 test cases
Score 0= wrong answer, 1= right answer, wrong explanation 2 = right answer, right but poor explanation 3= right answer, good explanation
Laplace’s Law
In a cylindrical vessel, the wall tension is proportional to the radius and pressure exerted by the vessel contents. This can be expressed as T = PR where T is wall tension, P is pressure exerted by the contents, and R is the radius of the vessel.
A 72 year old female has been diagnosed with an aneurysm (dilatation) of the aorta. The doctor tells her that if it grows to 5 cm in diameter she will need surgery to prevent bleeding. Explain why the increasing diameter is a problem.
Average Score
15.2/36 = 42%
“…during early learning, the principle is only understood in terms of the earlier example… the principle and example are bound together. Even if learners are given the principle or formula, they would use the details of the earlier problem in figuring out how to apply that principle to the current problem” (Ross, 1987)
Why are the examples so seductive?
System 1Fast, unconscious, contextualized, concrete
System 2Slow, logical, abstract
Transfer amounts to overriding System 1 to utilize abstract, conceptual information
DP and Age
Evidence from psychology that with increasing age, we rely more on System 1 thinking
Eva & Cunnington, 2006
15 family docs, 7 < 60 yr., 8>60 yr.
8 cases: 2 diagnoses, 4 conditions
Generated, Provided, Privileged, Extreme--------------->>>>>> weight on second diagnosis
Diff (Dx 1 - Dx2)
-30
-20
-10
0
10
20
30
40
50
Gen Prov Priv Extreme
YoungOld
Some Last Words
(from other people)
{the expert} does not solve problems. He does not even think. He just does what normally works, and, of course, it normally works…. The expert is simply not following any rules! He is… discriminating thousands of special cases.
H Dreyfus
“In general, to preserve expertise we must foster intuition at all levels of decision-making, otherwise wisdom will become an endangered species of knowledge.”
H. Dreyfus
First and Last Word on Expertise
“It is a profoundly erroneous truism, repeated by all copy-books and by eminent people making speeches, that we should cultivate the habit of thinking about what we are doing. The precise opposite is the case. Civilization advances by extending the number of operations which we can perform without thinking about them. Operations of thought are like cavalry charges in a battle -- they are strictly limited in number, they require fresh horses, and must only be made at decisive moments.”
A.N. Whitehead, 1911 (in J Bargh, 1999)
Bibliography
Dreyfus HL From Socrates to expert systems: The limits and dangers of calculative rationality. http://socrates.berkeley.edu
Evans J St BT. In two minds: dual - process accounts of reasoning. Trends in Cognitive Science 2003; 7: 454-459
Evans J StBT. Dual processing accounts of reasoning, judgment and social cognition. Ann Rev Psychol 2008;59: 255-78.