bsagent penn
DESCRIPTION
Expressing Preferences in a Principal-Agent Task: A Comparison of Choice, Matching and RatingTRANSCRIPT
Expressing Preferences in a Principal-Agent Task:
A Comparison of Choice, Matching and Rating
Joel Huber
Dan Ariely
Greg Fischer
Research Overview
C h o ice am on g trip les R atin g in d ivid u a l O p tion s
E s tim ate P artworts
A ssess B iases in E s tim ated vs . Tru e
M atch in g P a irs
A g en ts learn va lu es
Targ e t P artworth s
Choice Learning Task
Relative importance of shift from Poor -----> Fair -----> Good
Total cost ($900-$600-$300)
Ski slope quality (C-70, B-80, A-90)
Likelihood of Excellent Snow (50%,70%,90%)
Travel time (6-4-2 hours)
Night Life (poor, fair, good)
Training exercise: Which one would you choose?
A BTotal Cost ($900-$600-$300) $300 $900
Ski slope quality (C-70, B-80, A-90) 70 90
Theoretical Framework Effort -- Accuracy Tradeoff
• Tasks focus attention on certain features
• Tasks direct the framing of the judgment
• Tasks requiring greater effort result in greater simplification
How Biases are Manifested
• Attribute focus:– More weight on more important attributes
• Level focus:– Negativity: penalize alternatives with low
values on an attribute– Utility Dependence: value difference between
levels drives simplification
Assume the following target partworths
0
60
Worst Middle Best
Attribute Weight 45%
35%
20%
Price
Slope quality
Snow Probability
Low -endWeight
.80
.50
.20
0
60
Worst Middle Best
Attribute Weight 45%
35%
20%
Price
Slope quality
Snow Probability
Low-endWeight
.80
.50
.20
Target--------------Attribute Focusing
0
60
Worst Middle Best
Shift in Attribute Weight +22%
-14%
-25%
Price
Slope quality
Snow Probability
Shift in Low -endWeight
0%
0%
0%
0
60
Worst Middle Best
Attribute Weight 45%
35%
20%
Price
Slope quality
Snow Probability
Low-endWeight
.80
.50
.20
Target--------------Negativity
0
60
Worst Middle Best
Shift inAttribute Weight 0
0
0
Price
Slope quality
Snow Probability
Shift inLow-endWeight
+13%
+40%
+150%
0
60
Worst Middle Best
Attribute Weight 45%
35%
20%
Price
Slope quality
Snow Probability
Low-endWeight
.80
.50
.20
Target--------------Utility Dependence
0
60
Worst Middle Best
Shift inAttribute Weight 0
0
0
Price
Slope quality
Snow Probability
Shift in Low-endWeight
+10%
0
-50%
Target Partworths for First Study
Relative importance of shift from Poor -----> Fair -----> Good
Total cost ($900-$600-$300)
Ski slope quality (C-70, B-80, A-90)
Likelihood of Excellent Snow (50%,70%,90%)
Travel time (6-4-2 hours)
Night Life (poor, fair, good)
Example of a Choice
Which item would you choose? A B C
Total cost ($900-$600-$300) $900 $600 $300 Ski Slope Quality (C-70,B-80,A-90) 90 80 70
Likelihood of Excellent Snow (50%,70%,90%) 90% 70% 50% Travel time (6-4-2 hours) 4-hrs 3-hrs 2-hrs
Night Life (poor, fair, good) poor good fair
Relative importance of shift from Poor -----> Fair -----> Good
Total cost ($900-$600-$300)
Ski slope quality (C-70, B-80, A-90)
Likelihood of Excellent Snow (50%,70%,90%)
Travel time (6-4-2 hours)
Night Life (poor, fair, good)
Example of a Rating Relative importance of shift from Poor -----> Fair -----> Good
Total cost ($900-$600-$300)
Ski slope quality (C-70, B-80, A-90)
Likelihood of Excellent Snow (50%,70%,90%)
Travel time (6-4-2 hours)
Night Life (poor, fair, good)
Rate the overall value of this ski trip
Total cost ($900-$600-$300) $300Ski Slope Quality (C-70,B-80,A-90) 70
Likelihood of Excellent Snow (50%,70%,90%) 50%Travel time (6-4-2 hours) 2-hrs
Night Life (poor, fair, good) fair
Worst Average Best 1 2 3 4 5 6 7 8 9
Example of Matching Relative importance of shift from Poor -----> Fair -----> Good
Total cost ($900-$600-$300)
Ski slope quality (C-70, B-80, A-90)
Likelihood of Excellent Snow (50%,70%,90%)
Travel time (6-4-2 hours)
Night Life (poor, fair, good)
Indicate the cost that would make these two trips equally valuableA B
Total cost ($900-$600-$300) ? $300Ski Slope Quality (C-70,B-80,A-90) 90 70
Likelihood of Excellent Snow (50%,70%,90%) 90% 50%Travel time (6-4-2 hours) 4-hrs 2-hrs
Night Life (poor, fair, good) poor fair
Hypotheses--Choice
• Greatest quantity of information--least precise outcome required
• Attribute focus:– Prominence effect: increases focus on the most
important attributes
• Level focus:– Negativity: screen out low values– Utility dependence: collapse small differences
Hypotheses--Rating
• Simplest task--but requires implicit anchor
• Attribute focus:– Greater weight to more important attributes due
to motive to ignore less important attributes
• Level focus:– Negativity: due to loss aversion and moderate
reference level
Hypotheses--Matching
• Most complex task--pair differences put in metric of matching variable
• Attribute focus:– Greater weight to the matching variable: due to
compatibility with response scale and anchoring and incomplete adjustment
• Level focus:– Focus on differences makes nonlinear
responses difficult
Linear Partworths Study
• Sample: 80 MBA’s given bonus for how well they match management’s values
• Training: 7 choice and 9 matching tasks with only two attributes varying
• Tasks: 18 choice, matching and rating judgments with five attributes varying
• Four labeling conditions
Four Labeling Conditions
Attr. 1 Attr. 2 Attr. 3 Attr. 4 Attr. 5
Weight 36% 28% 16% 11% 9%
Condition
1. Total Cost Slope Quality Snow Probability Travel Time Night Life
2. Slope Quality Total Cost
3. Lift Wait Time Slope Quality
4. Slope Quality Lift Wait Time
Target Partworths
TRUE PARTWORTHS
0
50
100
150
200
250
1 2 3
Attribute Weight 36%
29%
16%
11%
9%
Low -end Weight .50
.50
.50
.50
.50
True ------------------ChoicePartworths Partworths
CHOICE
0
50
100
150
200
250
1 2 3
Shift in Attribute Weight -3%
-3%
+19%
+26%
-43%
Shift in Low -endWeight +22%
+46%
+9%
+53%
+66%
TRUE PARTWORTHS
0
50
100
150
200
250
1 2 3
Attribute Weight 36%
29%
16%
11%
9%
Low -end Weight .50
.50
.50
.50
.50
True -----------------MatchingPartworths Partworths
TRUE PARTWORTHS
0
50
100
150
200
250
1 2 3
Attribute Weight 36%
29%
16%
11%
9%
Low -end Weight .50
.50
.50
.50
.50
MATCHING
0
50
100
150
200
250
1 2 3
Shift in Attribute Weight +46%
-21%
-13%
-22%
-65%
Shift in Low -endWeight 0%
+7%
+17%
0%
+1%
Linear Tradeoff Study--Results(Standard deviation)
CHOICE RATINGS MATCHING
Attribute Focus:-3% -9% 46%
(14.3) (14.8) (10.3)
Level Focus40% 18% 3%
(27.0) (8.6) (4.7)
Decision Time19 11 26
(3.0) (1.9) (5.2)
Attitudes (0-100)Realistic 67 61 53Confident 63 57 43
Easy 58 49 39Interesting 57 56 55
Percent Overweighting of the Top Attribute
Percent Overweighting Least Liked Levels
Time per judgment in seconds
Linear Tradeoff Study--Summary
• Labeling has minimal impact--respondents are able to overcome priors
• Attribute focusing--No evidence in choice and ratings, matching puts 46% extra weight on matching variable
• Level focusing--Choice evokes strong, ratings moderate, and matching no negativity
Nonlinear Study--Motivation
• Will results hold under task of greater complexity?
• Will negativity hold in context of increasing returns?
• Does matching encourage linearity?
Nonlinear Study--Design
• One labeling condition
• 60 MBA’s
• Two balanced nonlinear targets
Two Nonlinear TargetsRelative importance of shift from Poor -----> Fair -----> Good
Total cost ($900-$600-$300)
Ski slope quality (C-70, B-80, A-90)
Likelihood of Excellent Snow (50%,70%,90%)
Travel time (6-4-2 hours)
Night Life (poor, fair, good)
Relative importance of shift from Poor -----> Fair -----> Good
Total cost ($900-$600-$300)
Ski slope quality (C-70, B-80, A-90)
Likelihood of Excellent Snow (50%,70%,90%)
Travel time (6-4-2 hours)
Night Life (poor, fair, good)
True ------------------ChoicePartworths Partworths
TRUE PARTWORTHS
0
50
100
150
200
250
1 2 3
Attribute Weight 36%
29%
16%
11%
9%
Low -end Weight .50
.50
.50
.50
.50
CHOICE
0
50
100
150
200
250
1 2 3
Shift in Attribute Weight -9%
-2%
+45%
-11%
-23%
Shift in Low -end Weight +20%
+27%
+4%
+15%
+84%
True -----------------RatingsPartworths Partworths
TRUE PARTWORTHS
0
50
100
150
200
250
1 2 3
Attribute Weight 36%
29%
16%
11%
9%
Low -end Weight .50
.50
.50
.50
.50
RATINGS
0
50
100
150
200
250
1 2 3
Shift in Attribute Weight -21%
-2%
+23%
+27%
+17%
Shift in Low -end Weight +8%
+24%
+26%
+19%
+29%
True -----------------MatchingPartworths Partworths
TRUE PARTWORTHS
0
50
100
150
200
250
1 2 3
Attribute Weight 36%
29%
16%
11%
9%
Low -end Weight .50
.50
.50
.50
.50
MATCHING
0
50
100
150
200
250
1 2 3
Shift in Attribute Weight +23%
-9%
-5%
-25%
-24%
Shift in Low -end Weght 0%
-4%
-7%
-11%
0%
Attribute Focusing—Linear Study
0%
10%
20%
30%
40%
50%
0% 10% 20% 30% 40% 50%
Matching
Choice
Ratings
Target Importance Weights
Expressed Importance
Weights
Attribute Focusing—Nonlinear Study
0%
10%
20%
30%
40%
50%
0% 10% 20% 30% 40% 50%
Matching
Choice
Ratings
Target Importance Weights
Expressed Importance
Weights
Attribute Focusing—Nonlinear Study
0%
10%
20%
30%
40%
50%
0% 10% 20% 30% 40% 50%
Matching
Choice
Ratings
Target Importance Weights
Expressed Importance
Weights Slope = .6
Results of the Nonlinear Study
CHOICE RATINGS MATCHINGCurvature Condition
Attribute Focus:
o-++- -9% -24% 24%
o+--+ -9% -18% 23%
Level Focus
o-++- 28% 25% 1%
o+--+ 22% 24% -8%
Decision Time
o-++- 52 18 56
o+--+ 45 22 50
Attitudes (0-100)Realistic 74 51 52Confident 59 51 51
Easy 50 47 37Interesting 54 52 59
Percent Overweighting of the Top Attribute
Percent Overweighting Least Liked Levels
Time per judgment in seconds
Conclusions: Agent’s Ability to Express Values Depends on the
Task• Choice: strong negativity bias, no evidence
of prominence
• Ratings: quick but imprecise, ignores increasing returns
• Matching: most work, but most precise except for matching variable which overvalued
Conclusions: Judgments as agent differ from own judgments
• Agents take more time, use more attributes
• Prominence in choice is replaced with equal weighting bias
• Matching works well in agent task, not for own values
• The agent task enables us to identify biases, avoiding the relative task statements
Implications
• Own decision may be improved by self elicitation and getting feedback on their expression in various tasks
• Matching should not be used for evaluating the matching variable
• Negativity and loss aversion are pervasive biases in both ratings and choice