pricing and incentives i kse 801 uichin lee. financial incentives and the performance of crowds...
TRANSCRIPT
Pricing and Incentives I
KSE 801Uichin Lee
Financial Incentives and the “Performance of Crowds”
Winter Mason and Duncan J. WattsKDD-HCOMP ‘09, June 28, 2009
Paris, France
Motivation of Crowdsourcing: Monetary incentive vs. Performance
• Motivation of crowdsourcing and peer production– Intrinsic motivation: enjoyment, desire to help out– Social rewards (Flickr) – Monetary incentives (M-Turk)
• “Rational choice” in economic theory– Rational workers will choose to improve their performance in response
to a scheme that rewards such improvements with financial gain• Example: an autoglass factory where workers install windshields
on a production line and switched from time-rate wage (pay per hour) to piece-rate wage (pay per unit)– Over the course of a year and half, individual productivity has
increased 20%, concluding that performance based pay scheme is a powerful tool for eliciting improved performance
Motivation of Crowdsourcing: Monetary incentive vs. Performance
• Yet, under certain circumstance the provision of financial incentives can undermine “intrinsic motivation” (e.g., enjoyment, altruism), possibly leading to poorer outcome
• Workers may ignore rational incentives to work longer when they have accomplished pre-set targets (e.g., taxi driver examples)
• Recent experiments demonstrate that financial incentives undermine actual performance (called chocking effect); e.g., hampering innovations
Experiments
• Study 1: Image ordering– Image ordering: sorting a set of images taken from a traffic camera at
2-second intervals in chronological order– To understand the impact of monetary compensation on
performance (quantity and quality)– Differentiate between quantity of work (output) and quality of work
(accuracy)– Use image ordering tasks where output can vary widely and accuracy
can be measured objectively
• Study 2: Word puzzles– To further investigate the absence of an effect on accuracy (will see
what it means later)
Study 1: Image ordering
At the beginning of a task, you will be presented with a list of images taken from traffic cameras. An example list is shown below
Your goal is to reorder the list chronologically from left to right and top to bottom. The sorted list is shown below.
Drag and drop for ordering
Study 1: Image ordering• All participants (total 611 workers) were paid an initial fee of $0.10 to
complete the introductory survey and training set
• Then, participants were given information about (1) how much they would be paid (pay level) and (2) what difficulty level they will be given– Pay level:
• No pay• Low pay: $0.01 per image set• Medium pay: $0.05 per image set• High pay: $0.10 per image set
– Difficulty level:• Easy (two images per set)• Medium (three images per set)• Hard (four images per set)
Study 1: Image ordering – Results
• Across all difficulty levels participants chose to complete more tasks on average when the pay was higher– F(3, 607) = 15.73, p<0.001
• Across all payment levels, the number of completed tasks decreased with increasing difficulty
Num
ber o
f tas
ks c
ompl
eted
Pay per task
Hard: 4 images
Medium: 3 images
Easy: 2images
Study 1: Image ordering – Results • Accuracy measure:
– A: Proportion of image sets correctly sorted
– B: Spearman’s rank correction between the correct order and the sorted order• Perfect match: ρ = 1• Perfect mismatch: ρ = 0
• One-way ANOVA test confirms that increasing wage did not improve accuracy– A: F(3, 607) = 0.66, p<0.001– B: F(3, 607) = 0.82, p<0.001
Accu
racy
(pro
porti
on o
f im
ages
se
ts c
orre
ctly
labe
led)
Pay per task
Hard: 4 images
Medium: 3 images
Easy: 2 images
Discussion
• Another test: single wage of $0.01, yet alerting participants that accuracy is measured, with the following 4 variants1) Only informed that accuracy would be measured2) Shown feedback on their accuracy after every fourth image set3) Told explicitly that their pay would be contingent on their
performance4) (2) and (3) are used in combination
• With this test, they found that quantity and quality results were indistinguishable, suggesting that workers were treating their pay as performance dependent
Discussion: Anchoring effect?
• Workers in all conditions generally felt that the expected wage for the work they performed is greater than what they received – They felt they are paid less than
expected– X2=243.61, p<0.001
• Thus, workers were no more motivated to perform better no matter how much they were actually paid
Perc
eive
d Ta
sk V
alue
$0.10$0.01$0.05
$0.20
$0.10
Actual Task Value$0.05
Study 2: Word Puzzle• Want to further investigate absence of an effect on accuracy
* Shown a list of 15 possible words (not all of the words listed are in the puzzle)* Select a word: click the first and last letter (if correct, it will turn red)
* Two wage models: quota vs. piece rate* Quota: every puzzle successfully completed* Piece: every word they found* Pay levels: low, medium, high, (no pay) -- Puzzle: $0.01, $0.05, $0.10 -- Word: $0.01, $0.02, $0.03
Study 2: Word Puzzle – Results
• Payment results in higher # of completions
Incentive Framing
Num
ber o
f Puz
zles
Com
plet
ed
Study 2: Word Puzzle – Results
• # of puzzles completed did not differ by payment strategies --- no significant impact on quantity!
Pay per Puzzle
Num
ber o
f Puz
zles
Com
plet
ed
Pay per Word
Num
ber o
f Puz
zles
Com
plet
ed
F(2, 108) = 0.71, not significant F(2, 124) = 1.82, not significant
66
Possible Reason: Intrinsic motivation of enjoyment
Study 2: Word Puzzle – Results Ac
cura
cy
(fra
ction
of w
orld
s fo
und
per p
uzzl
e)
Cost
per
Wor
d
No Contingent Pay Pay per Puzzle Pay per Word
Accuracy
Cost per word
High accuracy per puzzle
means low cost per word
Low accuracy per puzzle, but workers find as many
words as they can Intrinsic
motivation (enjoyment)
Discussion• Quota requires more efforts than piece rate• Quota elicits more work: (1) greater marginal difficulties of
finding next words to complete a puzzle, (2) implicit goal setting – harder puzzles become more salient goals
Perc
eive
d Ta
sk V
alue
Perc
eive
d Ta
sk V
alue
Actual Task Value (Per Puzzle)Actual Task Value (Per Word)
Anchoring effects
Discussion• When there is no expectation of financial reward, effort is
motivated by other kinds of rewards (e.g., social); but when monetary compensation is expected, as in the AMT framework, the anticipated financial value of the effort will be the driving mechanism.
• Although paid workers generally did more work than unpaid workers, how they were paid had a larger impact on their output and accuracy than how much they were paid.
• Moreover, paying workers a low rate led to them to perceive their work as less valuable than not paying them at all (previous slide)
Summary
• Investigated the relationship between financial incentives and performance in the novel setting of online peer production systems
• Main findings: – Increased payments increased the quantity of work
performed, but not its quality– The particular design of the compensation scheme (a quota
scheme vs. a piece rate, for example) can have a significant effect on quality even to the point where better work can be accomplished for less pay (or even without any payment)