bsagent penn

Expressing Preferences in a Principal-Agent Task:

A Comparison of Choice, Matching and Rating

Joel Huber

Dan Ariely

Greg Fischer

Research Overview

C h o ice am on g trip les R atin g in d ivid u a l O p tion s

E s tim ate P artworts

A ssess B iases in E s tim ated vs . Tru e

M atch in g P a irs

A g en ts learn va lu es

Targ e t P artworth s

Choice Learning Task

Relative importance of shift from Poor -----> Fair -----> Good

Total cost ($900-$600-$300)

Ski slope quality (C-70, B-80, A-90)

Likelihood of Excellent Snow (50%,70%,90%)

Travel time (6-4-2 hours)

Night Life (poor, fair, good)

Training exercise: Which one would you choose?

A BTotal Cost ($900-$600-$300) $300 $900

Ski slope quality (C-70, B-80, A-90) 70 90

Theoretical Framework Effort -- Accuracy Tradeoff

• Tasks focus attention on certain features

• Tasks direct the framing of the judgment

• Tasks requiring greater effort result in greater simplification

How Biases are Manifested

• Attribute focus:– More weight on more important attributes

• Level focus:– Negativity: penalize alternatives with low

values on an attribute– Utility Dependence: value difference between

levels drives simplification

Assume the following target partworths

0

60

Worst Middle Best

Attribute Weight 45%

35%

20%

Price

Slope quality

Snow Probability

Low -endWeight

.80

.50

.20

0

60

Worst Middle Best

Attribute Weight 45%

35%

20%

Price

Slope quality

Snow Probability

Low-endWeight

.80

.50

.20

Target--------------Attribute Focusing

0

60

Worst Middle Best

Shift in Attribute Weight +22%

-14%

-25%

Price

Slope quality

Snow Probability

Shift in Low -endWeight

0%

0%

0%

0

60

Worst Middle Best

Attribute Weight 45%

35%

20%

Price

Slope quality

Snow Probability

Low-endWeight

.80

.50

.20

Target--------------Negativity

0

60

Worst Middle Best

Shift inAttribute Weight 0

0

0

Price

Slope quality

Snow Probability

Shift inLow-endWeight

+13%

+40%

+150%

0

60

Worst Middle Best

Attribute Weight 45%

35%

20%

Price

Slope quality

Snow Probability

Low-endWeight

.80

.50

.20

Target--------------Utility Dependence

0

60

Worst Middle Best

Shift inAttribute Weight 0

0

0

Price

Slope quality

Snow Probability

Shift in Low-endWeight

+10%

0

-50%

Target Partworths for First Study

Relative importance of shift from Poor -----> Fair -----> Good

Total cost ($900-$600-$300)

Ski slope quality (C-70, B-80, A-90)

Likelihood of Excellent Snow (50%,70%,90%)

Travel time (6-4-2 hours)

Night Life (poor, fair, good)

Example of a Choice

Which item would you choose? A B C

Total cost ($900-$600-$300) $900 $600 $300 Ski Slope Quality (C-70,B-80,A-90) 90 80 70

Likelihood of Excellent Snow (50%,70%,90%) 90% 70% 50% Travel time (6-4-2 hours) 4-hrs 3-hrs 2-hrs

Night Life (poor, fair, good) poor good fair

Relative importance of shift from Poor -----> Fair -----> Good

Total cost ($900-$600-$300)

Ski slope quality (C-70, B-80, A-90)

Likelihood of Excellent Snow (50%,70%,90%)

Travel time (6-4-2 hours)

Night Life (poor, fair, good)

Example of a Rating Relative importance of shift from Poor -----> Fair -----> Good

Total cost ($900-$600-$300)

Ski slope quality (C-70, B-80, A-90)

Likelihood of Excellent Snow (50%,70%,90%)

Travel time (6-4-2 hours)

Night Life (poor, fair, good)

Rate the overall value of this ski trip

Total cost ($900-$600-$300) $300Ski Slope Quality (C-70,B-80,A-90) 70

Likelihood of Excellent Snow (50%,70%,90%) 50%Travel time (6-4-2 hours) 2-hrs

Night Life (poor, fair, good) fair

Worst Average Best 1 2 3 4 5 6 7 8 9

Example of Matching Relative importance of shift from Poor -----> Fair -----> Good

Total cost ($900-$600-$300)

Ski slope quality (C-70, B-80, A-90)

Likelihood of Excellent Snow (50%,70%,90%)

Travel time (6-4-2 hours)

Night Life (poor, fair, good)

Indicate the cost that would make these two trips equally valuableA B

Total cost ($900-$600-$300) ? $300Ski Slope Quality (C-70,B-80,A-90) 90 70

Likelihood of Excellent Snow (50%,70%,90%) 90% 50%Travel time (6-4-2 hours) 4-hrs 2-hrs

Night Life (poor, fair, good) poor fair

Hypotheses--Choice

• Greatest quantity of information--least precise outcome required

• Attribute focus:– Prominence effect: increases focus on the most

important attributes

• Level focus:– Negativity: screen out low values– Utility dependence: collapse small differences

Hypotheses--Rating

• Simplest task--but requires implicit anchor

• Attribute focus:– Greater weight to more important attributes due

to motive to ignore less important attributes

• Level focus:– Negativity: due to loss aversion and moderate

reference level

Hypotheses--Matching

• Most complex task--pair differences put in metric of matching variable

• Attribute focus:– Greater weight to the matching variable: due to

compatibility with response scale and anchoring and incomplete adjustment

• Level focus:– Focus on differences makes nonlinear

responses difficult

Linear Partworths Study

• Sample: 80 MBA’s given bonus for how well they match management’s values

• Training: 7 choice and 9 matching tasks with only two attributes varying

• Tasks: 18 choice, matching and rating judgments with five attributes varying

• Four labeling conditions

Four Labeling Conditions

Attr. 1 Attr. 2 Attr. 3 Attr. 4 Attr. 5

Weight 36% 28% 16% 11% 9%

Condition

1. Total Cost Slope Quality Snow Probability Travel Time Night Life

2. Slope Quality Total Cost

3. Lift Wait Time Slope Quality

4. Slope Quality Lift Wait Time

Target Partworths

TRUE PARTWORTHS

0

50

100

150

200

250

1 2 3

Attribute Weight 36%

29%

16%

11%

9%

Low -end Weight .50

.50

.50

.50

.50

True ------------------ChoicePartworths Partworths

CHOICE

0

50

100

150

200

250

1 2 3

Shift in Attribute Weight -3%

-3%

+19%

+26%

-43%

Shift in Low -endWeight +22%

+46%

+9%

+53%

+66%

TRUE PARTWORTHS

0

50

100

150

200

250

1 2 3

Attribute Weight 36%

29%

16%

11%

9%

Low -end Weight .50

.50

.50

.50

.50

True -----------------MatchingPartworths Partworths

TRUE PARTWORTHS

0

50

100

150

200

250

1 2 3

Attribute Weight 36%

29%

16%

11%

9%

Low -end Weight .50

.50

.50

.50

.50

MATCHING

0

50

100

150

200

250

1 2 3

Shift in Attribute Weight +46%

-21%

-13%

-22%

-65%

Shift in Low -endWeight 0%

+7%

+17%

0%

+1%

Linear Tradeoff Study--Results(Standard deviation)

CHOICE RATINGS MATCHING

Attribute Focus:-3% -9% 46%

(14.3) (14.8) (10.3)

Level Focus40% 18% 3%

(27.0) (8.6) (4.7)

Decision Time19 11 26

(3.0) (1.9) (5.2)

Attitudes (0-100)Realistic 67 61 53Confident 63 57 43

Easy 58 49 39Interesting 57 56 55

Percent Overweighting of the Top Attribute

Percent Overweighting Least Liked Levels

Time per judgment in seconds

Linear Tradeoff Study--Summary

• Labeling has minimal impact--respondents are able to overcome priors

• Attribute focusing--No evidence in choice and ratings, matching puts 46% extra weight on matching variable

• Level focusing--Choice evokes strong, ratings moderate, and matching no negativity

Nonlinear Study--Motivation

• Will results hold under task of greater complexity?

• Will negativity hold in context of increasing returns?

• Does matching encourage linearity?

Nonlinear Study--Design

• One labeling condition

• 60 MBA’s

• Two balanced nonlinear targets

Two Nonlinear TargetsRelative importance of shift from Poor -----> Fair -----> Good

Total cost ($900-$600-$300)

Ski slope quality (C-70, B-80, A-90)

Likelihood of Excellent Snow (50%,70%,90%)

Travel time (6-4-2 hours)

Night Life (poor, fair, good)

Relative importance of shift from Poor -----> Fair -----> Good

Total cost ($900-$600-$300)

Ski slope quality (C-70, B-80, A-90)

Likelihood of Excellent Snow (50%,70%,90%)

Travel time (6-4-2 hours)

Night Life (poor, fair, good)

True ------------------ChoicePartworths Partworths

TRUE PARTWORTHS

0

50

100

150

200

250

1 2 3

Attribute Weight 36%

29%

16%

11%

9%

Low -end Weight .50

.50

.50

.50

.50

CHOICE

0

50

100

150

200

250

1 2 3

Shift in Attribute Weight -9%

-2%

+45%

-11%

-23%

Shift in Low -end Weight +20%

+27%

+4%

+15%

+84%

True -----------------RatingsPartworths Partworths

TRUE PARTWORTHS

0

50

100

150

200

250

1 2 3

Attribute Weight 36%

29%

16%

11%

9%

Low -end Weight .50

.50

.50

.50

.50

RATINGS

0

50

100

150

200

250

1 2 3

Shift in Attribute Weight -21%

-2%

+23%

+27%

+17%

Shift in Low -end Weight +8%

+24%

+26%

+19%

+29%

True -----------------MatchingPartworths Partworths

TRUE PARTWORTHS

0

50

100

150

200

250

1 2 3

Attribute Weight 36%

29%

16%

11%

9%

Low -end Weight .50

.50

.50

.50

.50

MATCHING

0

50

100

150

200

250

1 2 3

Shift in Attribute Weight +23%

-9%

-5%

-25%

-24%

Shift in Low -end Weght 0%

-4%

-7%

-11%

0%

Attribute Focusing—Linear Study

0%

10%

20%

30%

40%

50%

0% 10% 20% 30% 40% 50%

Matching

Choice

Ratings

Target Importance Weights

Expressed Importance

Weights

Attribute Focusing—Nonlinear Study

0%

10%

20%

30%

40%

50%

0% 10% 20% 30% 40% 50%

Matching

Choice

Ratings

Target Importance Weights

Expressed Importance

Weights

Attribute Focusing—Nonlinear Study

0%

10%

20%

30%

40%

50%

0% 10% 20% 30% 40% 50%

Matching

Choice

Ratings

Target Importance Weights

Expressed Importance

Weights Slope = .6

Results of the Nonlinear Study

CHOICE RATINGS MATCHINGCurvature Condition

Attribute Focus:

o-++- -9% -24% 24%

o+--+ -9% -18% 23%

Level Focus

o-++- 28% 25% 1%

o+--+ 22% 24% -8%

Decision Time

o-++- 52 18 56

o+--+ 45 22 50

Attitudes (0-100)Realistic 74 51 52Confident 59 51 51

Easy 50 47 37Interesting 54 52 59

Percent Overweighting of the Top Attribute

Percent Overweighting Least Liked Levels

Time per judgment in seconds

Conclusions: Agent’s Ability to Express Values Depends on the

Task• Choice: strong negativity bias, no evidence

of prominence

• Ratings: quick but imprecise, ignores increasing returns

• Matching: most work, but most precise except for matching variable which overvalued

Conclusions: Judgments as agent differ from own judgments

• Agents take more time, use more attributes

• Prominence in choice is replaced with equal weighting bias

• Matching works well in agent task, not for own values

• The agent task enables us to identify biases, avoiding the relative task statements

Implications

• Own decision may be improved by self elicitation and getting feedback on their expression in various tasks

• Matching should not be used for evaluating the matching variable

• Negativity and loss aversion are pervasive biases in both ratings and choice