bayesian truth-serum

18
Introduction to Bayesian Truth Serum (BTS) presented by Fuming Shih

Upload: fuming-shih

Post on 11-May-2015

334 views

Category:

Technology


0 download

TRANSCRIPT

Page 1: Bayesian truth-serum

Introduction to Bayesian Truth Serum (BTS)

presented by Fuming Shih

Page 2: Bayesian truth-serum

Ask for true opinion?

• Will you buy Samsung Galaxy S3 when it comes out? (Yes/No)

• Will you vote in the next presidential election?– (definitely/probably/probably not/definitely not)

• Have you had more than 20 sexual partners over the past year (Yes/No)

Page 3: Bayesian truth-serum

What is BTS?

• Survey scoring method that provides truth-telling incentives for respondents answering multiple-choice questions

• Respondents to supply not only their own answers, but also percentage estimates of others’ answers.

• The formula then assigns high scores to answers that are surprisingly common

A Bayesian Truth Serum for Subjective Data by Drazen Prelec Science 15 October 2004: Vol. 306 no. 5695 pp. 462-466

Page 4: Bayesian truth-serum

BTS simplified

• “The premise behind this approach is the following. If people truly hold a particular belief, they are more likely to think that others agree or have had similar experiences.”

• you are your best estimator– or your estimation reveals you– posterior probability

Youthe unknown world (distribution of different opinions)

your estimation

Page 5: Bayesian truth-serum

Example Survey

Page 6: Bayesian truth-serum

How it works

reference: http://internetconferences.net/ipsi/files/CroatiaPrelecVIPSI.pdf

Page 7: Bayesian truth-serum

Calculate BTS Score

reference: http://internetconferences.net/ipsi/files/CroatiaPrelecVIPSI.pdf

Page 8: Bayesian truth-serum

The Information score: measures surprisingly common

ex. log(0.15/0.05)

reference: http://internetconferences.net/ipsi/files/CroatiaPrelecVIPSI.pdf

Page 9: Bayesian truth-serum

prediction score measures prediction accuracy

equals zero for a perfect prediction

reference: http://internetconferences.net/ipsi/files/CroatiaPrelecVIPSI.pdf

Page 10: Bayesian truth-serum

Conclusion First

• The best strategy for the respondent is to tell the truth

Your preference “wins” to the extent that itis more popular than collectively estimated

reference: http://internetconferences.net/ipsi/files/CroatiaPrelecVIPSI.pdf

Page 11: Bayesian truth-serum

The intuitive argument for m=2

reference: http://internetconferences.net/ipsi/files/CroatiaPrelecVIPSI.pdf

Page 12: Bayesian truth-serum

and I happen to like Red

reference: http://internetconferences.net/ipsi/files/CroatiaPrelecVIPSI.pdf

Page 13: Bayesian truth-serum

This is my best estimate of the Red share (e.g., 50%)

reference: http://internetconferences.net/ipsi/files/CroatiaPrelecVIPSI.pdf

Page 14: Bayesian truth-serum

Bayesian reasoning implies that someone wholikes White will estimate a smaller share for Red

reference: http://internetconferences.net/ipsi/files/CroatiaPrelecVIPSI.pdf

Page 15: Bayesian truth-serum

The average predicted share for Red will fallsomewhere between these two estimates

reference: http://internetconferences.net/ipsi/files/CroatiaPrelecVIPSI.pdf

Page 16: Bayesian truth-serum

Hence, if I like Red I should believe thatthe share for Red will be underestimated

or ‘surprisingly popular’

My prediction of theaverage Red shareestimate

reference: http://internetconferences.net/ipsi/files/CroatiaPrelecVIPSI.pdf

Page 17: Bayesian truth-serum

The argument holds even if I know that mypreferences are unusual

reference: http://internetconferences.net/ipsi/files/CroatiaPrelecVIPSI.pdf

Page 18: Bayesian truth-serum

Application?

• Honest signals subjective preferences– BTS draws more truth opinions from the users– reality mining captures the objective ground truths

• Are there relations between these two?– I feel stressful when multiple people around me– I feel depressed when I am alone

• A improvement on psychological-social probe– developing an opinion probe on funf-framework– capture preferences and context at the same time