1
Decision Making with Incomplete Information:Measures, Mediators, and Decision Support
Marc C. CanellasAdvisor: Dr. Karen M. Feigh
Ph.D. Thesis Proposal, School of Aerospace Engineering
Dr. Amy Pritchett, School of Aerospace Engineering, Georgia TechDr. Brian German, School of Aerospace Engineering, Georgia TechDr. Stephen Cross, School of Industrial and Systems Engineering, Georgia TechDr. Juan Rogers, School of Public Policy, Georgia Tech
Sept. 3, 2015
• What is heuristic decision making?• Decision algorithms that only use a subset of “necessary” information
• Why is heuristic decision making important?• Heuristics are used by people in time-stressed, high stakes environments (e.g. aviation,
medical, emergency response, and military domains)• Can established decision support research help support heuristic decision makers?• Previous decision support research has almost exclusively focused on supporting
normative decision making strategies which do not apply to heuristic decision making environments.
• What recent methods for heuristic decision support have been developed?• Structuring the decision making process and environment
How should systems be designed to support heuristic decision makers?
Sept. 3, 2015
• What is an unstudied area of heuristic decision support with potential?• Heuristic decision support for structuring information (information acquisition)
2
3
Issues with normative decision making
Sept. 3, 20151Orasanu and Connolly, 1993; 2Green and Mehr, 1997;3Tversky and Kahneman, 1974; 4Katsikopoulos and Fasolo,
2006; 5Elwyn et al., 2001; 6This result is the output of the naturalistic decision making and fast-and-frugal heuristics research programs
Descriptive problem: • People often do not make decisions by gathering and processing all
information (due to time, cost, experience, etc.)1
• Necessary information often not available2
• People often cannot provide reliable assessments of probabilities, attributes weights, and value3
• Lack of transparency and understanding of underlying methods4,5
Normative decision making is not a good basis for building decision support tools for these environments4.
• People are often very well-adapted to their environment and use simple rules that enable them to perform just-as-good if not, better than normative methods6
4
Normative vs. Heuristic
Sept. 3, 20151Nelson, 2005; 2Nelson, 2008; 3Meder and Nelson, 2012; 4Katsikopoulos and Fasolo, 2006; 5Todd,
Gigerenzer, and the ABC Research Group, 2012
Normative Heuristic Benefits of Heuristic Decision Making
Decision Making
Strategies
Gather and process all information about the
environment
Simple rules that process information deemed “necessary”
• Faster to perform4
• Easier to communicate4
• More accurate and robust in many environments5Information
AcquisitionQuantify the value of each
potential piece of information1,2,3 ?
Heuristic information acquisition methods have the potential tobenefit heuristic decision making performance.
Simple rules for determining which
information to acquire
5Sept. 3, 2015
Objectives and MethodDevelop methods of heuristic information acquisition
6
Research Objectives and Method
Sept. 3, 2015
1. Define measures of incomplete information
2. Develop methods of heuristic information acquisition
3: Examine the effect of distributions of incomplete information
4: Compare heuristic information acquisition methods to normative methods
5: Examine the effect of distributions of incomplete information on human decision making
6: Implement the heuristic information acquisition methods in a decision support tool
Simulation: Artificial Environment
Simulation: Empirical Environment
Simulation: Empirical Environment
Simulation: Empirical Environment
Human-Subjects Study Human-Subjects Study
Q: What measures define distributions of incomplete information and affect decision making accuracy?
Q1: Can information be removed or added in specific ways to increase accuracy of decision making strategies?
Q2: What aspects of the environment affect when the methods work?
Q: What aspects of the environment mediate the effect of the measures of distributions of incomplete information on the accuracy of decision making strategies?
Q: Do the heuristic information R&G methods perform similarly to established normative information acquisition methods?
Q: What are the effects of distributions of incomplete information on human decision making?
Q: How can the heuristic information R&G methods be implemented in a decision support tool?
Complete1,2,3 Complete4 In-progress Future work Future work Future Work
7Sept. 3, 2015
Task 0: Construct a Decision Making Simulation Engine Enables comprehensive study of all combinations of environments and incomplete information
8
Example Decision Task: UAV Path Selection
Sept. 3, 2015
Three attributes of the paths: time, danger, and control.
Preferences• Minimum time is preferred• Low danger is preferred to high danger• Autopilot control is preferred to manual control
Decision Task
BASE
TARGET
B30 min
Autopilot Option Time to Target Danger Control
A 10 High Manual
B 30 Low Autopilot
A10 minManual
9
Decision Making Simulation Engine
Sept. 3, 2015
Decision Task Decision Making Strategies
Option Time to Target Danger Control
A 10 High ManualB 30 Low Autopilot
Incomplete information
Environment
Option Time to Target Danger Control
A K ? KB ? K K
Option Time to Target Danger Control
A 10 ? ManualB ? Low Autopilot
ChoiceEnvironment
Incomplete information
Decision Task
The state of the world
Missing information about theoptions’ attributes
Option-attribute informationpresented to the decision maker
Process information to choose a course of action
10
Weighted Additive (WADD)1 Equal Weighting (EW)1 Tallying2 Take-Two3,4 Take-the-Best (TTB)5
Linear model with weights
Linear model without weights
Linear modelcounting positive
attributesTwo-attribute
reasoning by rankingOne-attribute
reasoning by ranking
Multi-attribute decision making6;Novice decision
makers7
When weights cannot be determined or
agreed upon8
Criminal sentencing decisions9;
Medical decision making10
People often search for a second,
confirming attribute3,4
Consumer choice11; Expert decision
makers7
Decision Making Strategies
Sept. 3, 2015Operationalizations: 1Payne et al., 1990; 2Gigerenzer and Gaissmaier, 2011; 3Karelaia, 2006; 4Dieckmann and Rieskamp, 2007; 5Gigerenzer and Goldstein, 1996; 6Park, 2004; 7Garcia-Retamero and Dhami, 2009; 8Dawes, 1979; 9von Helversen
and Rieskamp, 2009; 10Kattah et al., 2009; 11Kohli and Jedidi
Analytic:gather as much information as
possible to use in a linear model
Heuristic:Use only a subset of “necessary”
information
11
Decision Making Simulation Engine
Sept. 3, 2015
Decision Task Decision Making Strategies
Option Time to Target Danger Control
A 10 High ManualB 30 Low Autopilot
Incomplete information
Environment
Option Time to Target Danger Control
A K ? KB ? K K
Option Time to Target Danger Control
A 10 ? ManualB ? Low Autopilot
ChoiceEnvironment
Incomplete information
Decision Task
The state of the world
Missing information about theoptions’ attributes
Option-attribute informationpresented to the decision maker
Process information to choose a course of action
WADD
EW
Tallying
Take Two
TTB
12Sept. 3, 2015
Task 1. Define measures of incomplete information
13
Task 1: Identify important measures of distributions of incomplete information
Sept. 3, 2015
Define important measures of distributions of incomplete information.
Identify potential methods for decision support based on incomplete information.
14
Current measures of incomplete information are not sufficient
Sept. 3, 2015
Option Time to Target Danger ControlA ? ? ?B 30 Low Autopilot
Option Time to Target Danger ControlA 10 ? ?B 30 Low ?
Equivalent total information
3 pieces(50%)
Total information is not sufficient to understand the full effects of incomplete information.
1 2
15
Measuring distributions of incomplete information
Sept. 3, 20151Martignon and Hoffrage, 2002; 2Garcia-Retamero and Rieskamp, 2008; 3Canellas et al., 2014; 4Canellas
et al., 2015; 5Canellas and Feigh, 2014
Measure Definition MeasuresTotal Information (TI)1,2 Count (percent) of attribute scores
known- The amount of information known to the
decision makerInformation Imbalance (II)3,4 Difference in total information for
each option.- Approximates the number of known-
unknown attribute comparisons typically used by heuristics
- Situations where one option is well-known to the decision maker
Complete Attribute Pairs (CAP)5 Count (percent) of attributes in which attribute scores are known for both options.
- The number of known-known attribute comparisons
16
Measuring distributions of incomplete information
Sept. 3, 2015
Option Time to Target Danger Control
A ? ? ?B 30 Low Autopilot
Option Time to Target Danger Control
A 10 ? ?B 30 Low ?
Total Information 3 (50%) Know half the information
Information Imbalance 3 (100%)
Know all the information about
one option and nothing about the
other
Complete Attribute Pairs 0 (0%)
No attributes have information for both
options.
Total Information 3 (50%) Know half the information
Information Imbalance 1 (33%)Know slightly more
information about one option than the other
Complete Attribute Pairs 1 (33%)
For one attribute, there is information about
both options.
17
Task 1. Decision Making Simulation Engine (artificial environment)
Sept. 3, 2015 Simulation set-up used in: Canellas et al., 2014; Canellas and Feigh, 2014; Canellas et al., 2015
Decision Task Decision Making StrategiesOption A1 A2 A3 A4
O1 40 20 80 40O2 40 60 40 60
Incomplete information
EnvironmentChoice
Environment (48)
Incomplete Information (28)
Decision Task (88: 16 million)
Artificial: randomly generated attribute scores and weights.
Option A1 A2 A3 A4
O1 ? ? K K
O2 K K K K
Option A1 A2 A3 A4
O1 ? ? 80 40
O2 40 60 40 60
Accuracy is determined by a linear model that is not representative and biased toward WADD.
WADD
EW
Tallying
TTB
Two analytic strategies and two heuristic strategies.
Accuracy: percent of decision tasks that the decision making strategy selects the correct option.
Decision Making Strategies
Choice
18
Task 1. Effect of incomplete information on decision making accuracy
Sept. 3, 2015
0% 25% 50% 75% 100%50%
60%
70%
80%
90%
100%
Information Imbalance
0% 20% 40% 60% 80% 100%50%
60%
70%
80%
90%
100%
Total Information
Accuracy
0% 25% 50% 75% 100%50%
60%
70%
80%
90%
100%
Complete Attribute Pairs
Decision Support Recommendation:
Provide decision makers with more information.
Decision Support Recommendation:
For heuristic decision makers, provide equal information about both options.For analytic decision makers, there is no effect.
Decision Support Recommendation:
Provide decision makers with attribute information for both options.
19
4 883 84 822 79 77 771 75 72 72 720 64 65 67 68
1 2 3 4 5 6 7 8
4 1003 88 912 78 81 851 71 73 77 810 62 68 73 78
1 2 3 4 5 6 7 8
Tradeoff: Total Information – Complete Attribute Pairs
Sept. 3, 2015
Total Information
Complete Attribute
Pairs
Complete Attribute
Pairs
Take-the-Best (TTB)Weighted-Additive (WADD)
Total InformationCells are empty when that combination cannot occur in a 2-option, 4-attribute task.
• Increasing total information:• Always increases accuracy
• Increasing complete attribute pairs while keeping total information constant:• Does not increase accuracy
• Increasing total information:• Decreases accuracy (if CAP ≥ 1)
• Increasing complete attribute pairs while keeping total information constant:• Increases accuracy
20
4 883 84 822 79 77 771 75 72 72 720 64 65 67 68
1 2 3 4 5 6 7 8
Task 1: Identify important measures of distributions of incomplete information
Sept. 3, 2015
Define important measures of distributions of incomplete information.
Within levels of total information, changing information imbalance or complete attribute pairs has a statistically significant effect on heuristic decision making performance. 0% 25% 50% 75% 100%
50%
60%
70%
80%
90%
100%
Information Imbalance Identify potential methods for decision support based on incomplete information.
Identified trade-offs between total information and the two measures of incomplete information (information imbalance and complete attribute pairs)
Complete Attribute
Pairs
Take-the-Best (TTB)
Total Information
21
Research Objectives and Method
Sept. 3, 2015 1Canellas et al., 2014; 2Canellas and Feigh, 2014; 3Canellas et al., 2015;
1. Define measures of incomplete information
2. Develop methods of heuristic information acquisition
3: Examine the effect of distributions of incomplete information
4: Compare heuristic information acquisition methods to normative methods
5: Examine the effect of distributions of incomplete information on human decision making
6: Implement the heuristic information acquisition methods in a decision support tool
Simulation: Artificial Environment
Simulation: Empirical Environment
Simulation: Empirical Environment
Simulation: Empirical Environment
Human-Subjects Study Human-Subjects Study
1. Added new metrics: information imbalance and complete attribute pairs.
2. Identified trade-offs with total information.
Q1: Can information be removed or added in specific ways to increase accuracy of decision making strategies?
Q2: What aspects of the environment affect when the methods work?
Q: What aspects of the environment mediate the effect of the measures of distributions of incomplete information on the accuracy of decision making strategies?
Q: Do the heuristic information R&G methods perform similarly to established normative information acquisition methods?
Q: What are the effects of distributions of incomplete information on human decision making?
Q: How can the heuristic information R&G methods be implemented in a decision support tool?
Complete1,2,3 Complete4 In-progress Future work Future work Future Work
22Sept. 3, 2015
Task 2. Develop methods of heuristic information acquisition
23
Task 2. Develop methods of heuristic information acquisition
Sept. 3, 2015
Q1: Can information be removed or added in specific ways to increase accuracy of decision making strategies?
Q2: What aspects of the environment affect when the methods work?
24
Choice
Decision Making Strategies
Task 2. Decision Making Simulation Engine (empirical environments)
Sept. 3, 2015 Environments from 1Czerlinski et al., 1999, used in numerous simulations (e.g. 2Hogarth and Karelaia, 2006; and 3Katsikopoulos, 2013); 4Hogarth and Karelaia, 2007
Decision Task (36 million)
Incomplete Information
Environment
Artificial: randomly generated attribute scores and weights.
Accuracy is determined by a linear model that is not representative.
WADD
EW
Tallying
TTB
Two analytic strategies and two heuristic strategies.
Empirical: 15 standard benchmarking simulation environments that represent “real-world” non-linear environments1,2,3
Measure environmental parameters4 that have been shown to affect decision making accuracy.
Measure: total information, information imbalance, and complete attribute pairs
Examine 3, 4, and 5 attribute decision tasks.
Take Two
Two analytic strategies and three heuristic strategies.
25
4 Rules of Information Restriction and Guidance
Sept. 3, 2015
Restriction(Remove 1 piece of information)
Guidance(Add 1 piece of information)
Complete Attribute Pairs Maintain the same number of complete
attribute pairsIncrease the number of complete
attribute pairs
Information Imbalance Reduce information imbalance Reduce information imbalance
26
86.6 A1 A2 A3 A4O1 K K KO2 K K
89.674.6 83.267.9 77.6
Example: Reduce information while maintaining the number of complete attribute pairs
Sept. 3, 2015
86.6 A1 A2 A3 A4
O1 K K K ?
O2 K K ? ?
86.6 A1 A2 A3 A4
O1 -12.0 -3.4 +3.0 ?
O2 -18.7 -9.0 ? ?
Removing one piece of information in a way that keeps the same number of complete attribute pairs can increase accuracy.
Total information: 5Complete attribute pairs: 2
Total information: 4Complete attribute pairs: or 21
Total information: 4Complete attribute pairs: or 21
Initial decision task with incomplete information (initial accuracy: 86.6%)
Resulting accuracy of TTB(heuristic)
Change in accuracy of TTB(heuristic)
2
27Sept. 3, 2015
If the decision maker is using Tallying, each instance of removing information via II-R increases the decision maker’s accuracy by 2.5%.
Results Summary: Reduce information while maintaining the number of complete attribute pairs
WADD EW Tallying Take Two TTB
AverageAccuracyChange
Complete Attribute Pairs – Restriction
73.0 A1 A2 A3 A4
O1 K K +3.0 ?
O2 K K ? ?
Total information: 4Complete attribute pairs: 2
Change in accuracy of TTB(heuristic)
Analytic Heuristic
28Sept. 3, 2015
StrategyComplete
Attribute Pairs- Guidance
Complete Attribute Pairs
- Restriction
InformationImbalance- Guidance
InformationImbalance
- RestrictionWADD +3.9 +3.2 +1.4 -0.5
EW +4.1 +3.3 +2.0 -0.1
Tallying +4.6 +2.0 +9.8 +2.5
Take Two +5.0 +2.1 +7.3 +1.4
TTB +4.8 +2.7 +6.3 +1.4
If the decision maker is using Tallying, each instance of removing information via II-R increases the decision maker’s accuracy by 2.5%.
Task 2 Summary: Average accuracy change of all 4 information restriction and guidance methods
By reducing information or adding information in specific ways, the accuracy of decision making strategies can be increased – heuristic decision making strategies in particular.
Each value is average accuracy change that results from using the rule once when applicable(averaged across all 15 environments and most of the 36 million runs)
Analytic
Heuristic
29
Task 2. Develop methods of heuristic information acquisition
Sept. 3, 2015
Q1: Can information be removed or added in specific ways to increase accuracy of decision making strategies?• For heuristic strategies, the average accuracy change is generally positive when
using the methods.• For analytic strategies, the average accuracy change generally positive for the
complete attribute pairs methods.
Q2: What aspects of the environment affect when the methods work?The methods enable decision making strategies to achieve close to their full information accuracy. Therefore, in environments where the strategies perform well, the information restriction and guidance methods perform well.
30Sept. 3, 2015
Current and Future Work
31
Research Objectives and Method
Sept. 3, 2015 1Canellas et al., 2014; 2Canellas and Feigh, 2014; 3Canellas et al., 2015; 4Canellas and Feigh, 2015
1. Define measures of incomplete information
2. Develop methods of heuristic information acquisition
3: Examine the effect of distributions of incomplete information
4: Compare heuristic information acquisition methods to normative methods
5: Examine the effect of distributions of incomplete information on human decision making
6: Implement the heuristic information acquisition methods in a decision support tool
Simulation: Artificial Environment
Simulation: Empirical Environment
Simulation: Empirical Environment
Simulation: Empirical Environment
Human-Subjects Study Human-Subjects Study
1. Defined information imbalance and complete attribute pairs.
2. Identified trade-offs with total information.
1. The methods work generally for heuristic strategies and the CAP-methods work for analytic strategies.
2. Work best in environments conducive to decision strategies performance
Q: What aspects of the environment mediate the effect of the measures of distributions of incomplete information on the accuracy of decision making strategies?
Q: Do the heuristic information R&G methods perform similarly to established normative information acquisition methods?
Q: What are the effects of distributions of incomplete information on human decision making?
Q: How can the heuristic information R&G methods be implemented in a decision support tool?
Complete1,2,3 Complete4 In-progress In-progress Future work Future Work
32
Task 3: Examine the effect of distributions of incomplete information
Sept. 3, 2015
Using the more comprehensive simulation environment developed for Task 2:Q1. Do the effects of complete attribute pairs and information imbalance generalize
to empirical simulation environments?Q2. Do any of the environmental parameters mediate the effect of incomplete
information?
0% 25% 50% 75% 100%50%
60%
70%
80%
Information Imbalance0% 25% 50% 75% 100%
50%
60%
70%
80%
Information Imbalance
Accuracy
Artificial Environment Empirical Environment
33
Task 4: Compare normative and heuristic information acquisition methods
Sept. 3, 2015 1Meder and Nelson, 2012; 2Canellas and Feigh, 2015; Table contains hypothetical results
NormativeInformation Acquisition
HeuristicInformation Acquisition
Quantify the value of each potential piece of
information1
Adapt the normative methods to my decision task, then use the simulation engine to study:
Q1. How often and in what cases do the methods agree or disagree as to what information should be acquired or not?
Q2. How do the methods differentially affect performance of decision making strategies?
Simple rules for determining which
information to acquire
Heuristic information
restriction and guidance methods2
A1 A2 A3
O1 K K ?O2 K ? ?
?→K
?→K
WADD TTB
Normative +2% +3%
Heuristic +5% +9%
34
Research Objectives and Method
Sept. 3, 2015 1Canellas et al., 2014; 2Canellas and Feigh, 2014; 3Canellas et al., 2015; 4Canellas and Feigh, 2015
1. Define measures of incomplete information
2. Develop methods of heuristic information acquisition
3: Examine the effect of distributions of incomplete information
4: Compare heuristic information acquisition methods to normative methods
5: Examine the effect of distributions of incomplete information on human decision making
6: Implement the heuristic information acquisition methods in a decision support tool
Simulation: Artificial Environment
Simulation: Empirical Environment
Simulation: Empirical Environment
Simulation: Empirical Environment
Human-Subjects Study Human-Subjects Study
1. Defined information imbalance and complete attribute pairs.
2. Identified trade-offs with total information.
1. The methods work generally for heuristic strategies and the CAP-methods work for analytic strategies.
2. Work best in environments conducive to decision strategies performance
Q: What aspects of the environment mediate the effect of the measures of distributions of incomplete information on the accuracy of decision making strategies?
Q: Do the heuristic information R&G methods perform similarly to established normative information acquisition methods?
Q: What are the effects of distributions of incomplete information on human decision making?
Q: How can the heuristic information R&G methods be implemented in a decision support tool?
Complete1,2,3 Complete4 In-progress In-progress Future work Future Work
35
Task 5: Examine human decision making with incomplete information
Sept. 3, 2015 1Rieskamp and Hoffrage, 2008; 2Monti et al., 2012
Simulated Decision Making Strategies Choice
Environment
Incomplete Information
Decision Task with Incomplete
Information
Human Decision Maker
Q. How do distributions of incomplete information on decision making accuracy extend to human subjects?
Method: • Present decision tasks with incomplete information
• Use varying time pressure to incentivize the use of heuristic or analytic decision making strategies1
• Design decision tasks with incomplete information such that decision strategies can be identified (process- and outcome-oriented methods)1,2
Option Time to Target Danger Control
A 10 ? ?
B 30 Low ?
36
Task 6: Implement the heuristic information acquisition methods in a decision support tool
Sept. 3, 2015
BASE
TARGET
B30 min
Autopilot
A10 minManual
Objective: Identify methods for implementing heuristic acquisition methods
Method: • Instead of tables of decision tasks being
presented to decision makers (Task 5), use context-relevant displays of information
• The planned task domain is a planning tool for controlling and operating a swarm of UAVs
37
Research Method
Sept. 3, 2015 1Canellas et al., 2014; 2Canellas and Feigh, 2014; 3Canellas et al., 2015; 4Canellas and Feigh, 2015
1. Define measures of incomplete information
2. Develop methods of heuristic information acquisition
3: Examine the effect of distributions of incomplete information
4: Compare heuristic information acquisition methods to normative methods
5: Examine the effect of distributions of incomplete information on human decision making
6: Implement the heuristic information acquisition methods in a decision support tool
Simulation: Artificial Environment
Simulation: Empirical Environment
Simulation: Empirical Environment
Simulation: Empirical Environment
Human-Subjects Study Human-Subjects Study
1. Defined information imbalance and complete attribute pairs.
2. Identified trade-offs with total information.
1. The methods work generally for heuristic strategies and the CAP-methods work for analytic strategies.
2. Work best in environments conducive to decision strategies performance
Q: What aspects of the environment mediate the effect of the measures of distributions of incomplete information on the accuracy of decision making strategies?
Q: Do the heuristic information R&G methods perform similarly to established normative information acquisition methods?
Q: What are the effects of distributions of incomplete information on human decision making?
Q: How can the heuristic information R&G methods be implemented in a decision support tool?
Complete1,2,3 Complete4 In-progress In-progress Future work Future Work
38Sept. 3, 2015
Conclusions
39
Motivation
• Heuristics are often used when there is time pressure, high information acquisition costs, information overload, or ill-structured environments.• A majority of previous decision
support research does not apply to heuristic decision making• There is potential for heuristic
information acquisition to support decision making performance
Sept. 3, 2015
40
Contributions
Sept. 3, 2015
1. Define measures of incomplete information
2. Develop methods of heuristic information
restriction and guidance (R&G)
3: Examine the effect of distributions of
incomplete information
4: Compare heuristic information R&G
methods to normative methods
5: Examine the effect of distributions of
incomplete information on human decision
making
6: Implement the heuristic information
R&G methods in a decision support tool
A new understanding of heuristic decision making with incomplete information
Completed In-Progress
How to implement this knowledge in a decision support tool.
Future
41Sept. 3, 2015
Acknowledgements• Dr. Karen Feigh, Dr. Zarrin Chua, and Rachel Haga for contributing to the research• Members of the German Research Group and the Cognitive Engineering Center
for their guidance and support• The NSF Graduate Research Fellowship Program which funds my research• The Office of Naval Research which funds the larger project which motivated this
research