Problem SettingSocial Feedback Mechanisms
Empirical EvidenceConclusions
How Opinions are Received by Online Communities:A Case Study on Amazon.com Helpfulness Votes
Cristian Danescu-Niculescu-Mizil1
Gueorgi Kossinets2
Jon Kleinberg1
Lillian Lee1
1Department of Computer Science, Cornell University2Google Inc. (formerly at the Department of Sociology, Cornell University)
04-23-2009 WWW2009
C. Danescu-Niculescu-Mizil, G. Kossinets, J. Kleinberg, L. Lee How Opinions are Received by Online Communities
Problem SettingSocial Feedback Mechanisms
Empirical EvidenceConclusions
Opinion vs. Meta-Opinion
C. Danescu-Niculescu-Mizil, G. Kossinets, J. Kleinberg, L. Lee How Opinions are Received by Online Communities
Problem SettingSocial Feedback Mechanisms
Empirical EvidenceConclusions
Opinion vs. Meta-Opinion
C. Danescu-Niculescu-Mizil, G. Kossinets, J. Kleinberg, L. Lee How Opinions are Received by Online Communities
Problem SettingSocial Feedback Mechanisms
Empirical EvidenceConclusions
Opinion vs. Meta-Opinion
C. Danescu-Niculescu-Mizil, G. Kossinets, J. Kleinberg, L. Lee How Opinions are Received by Online Communities
Problem SettingSocial Feedback Mechanisms
Empirical EvidenceConclusions
Opinion vs. Meta-Opinion
C. Danescu-Niculescu-Mizil, G. Kossinets, J. Kleinberg, L. Lee How Opinions are Received by Online Communities
Problem SettingSocial Feedback Mechanisms
Empirical EvidenceConclusions
Opinion vs. Meta-Opinion
C. Danescu-Niculescu-Mizil, G. Kossinets, J. Kleinberg, L. Lee How Opinions are Received by Online Communities
Problem SettingSocial Feedback Mechanisms
Empirical EvidenceConclusions
Opinion vs. Meta-Opinion
C. Danescu-Niculescu-Mizil, G. Kossinets, J. Kleinberg, L. Lee How Opinions are Received by Online Communities
Problem SettingSocial Feedback Mechanisms
Empirical EvidenceConclusions
Opinion vs. Meta-Opinion
Opinion:“What did Gary think of the book?”
[survey by Pang and Lee, 2008; Wu and Huberman, 2008]
Meta-Opinion:“What did the users think about Gary’s opinion of the book?”
[Ghose et al., 2008; Kim et al., 2006; Liu et al., 2007]Widespread in everyday life (e.g., political polls, market research)
C. Danescu-Niculescu-Mizil, G. Kossinets, J. Kleinberg, L. Lee How Opinions are Received by Online Communities
Problem SettingSocial Feedback Mechanisms
Empirical EvidenceConclusions
Opinion vs. Meta-Opinion
Opinion:“What did B think of A?”
[survey by Pang and Lee, 2008; Wu and Huberman, 2008]
Meta-Opinion:“What did C think about B’s opinion of A?”
[Ghose et al., 2008; Kim et al., 2006; Liu et al., 2007]
Widespread in everyday life (e.g., political polls, market research)
C. Danescu-Niculescu-Mizil, G. Kossinets, J. Kleinberg, L. Lee How Opinions are Received by Online Communities
Problem SettingSocial Feedback Mechanisms
Empirical EvidenceConclusions
Opinion vs. Meta-Opinion
Meta-Opinion:“What did C think about B’s opinion of A?”
Widespread in everyday life (e.g., political polls, marketresearch) and in online communities (e.g., Amazon.com,Facebook).
In this work we analyze and model the factors influencingmeta-opinions.
C. Danescu-Niculescu-Mizil, G. Kossinets, J. Kleinberg, L. Lee How Opinions are Received by Online Communities
Problem SettingSocial Feedback Mechanisms
Empirical EvidenceConclusions
Amazon.com as a Testbed for Meta-Opinion Analysis
Data4,000,000 reviews700,000 products
Factors
TextualNon-Textual
C. Danescu-Niculescu-Mizil, G. Kossinets, J. Kleinberg, L. Lee How Opinions are Received by Online Communities
Problem SettingSocial Feedback Mechanisms
Empirical EvidenceConclusions
Amazon.com as a Testbed for Meta-Opinion Analysis
Data4,000,000 reviews700,000 products
Factors
TextualNon-Textual
C. Danescu-Niculescu-Mizil, G. Kossinets, J. Kleinberg, L. Lee How Opinions are Received by Online Communities
Problem SettingSocial Feedback Mechanisms
Empirical EvidenceConclusions
Amazon.com as a Testbed for Meta-Opinion Analysis
Data4,000,000 reviews700,000 products
Factors
TextualNon-Textual
C. Danescu-Niculescu-Mizil, G. Kossinets, J. Kleinberg, L. Lee How Opinions are Received by Online Communities
Problem SettingSocial Feedback Mechanisms
Empirical EvidenceConclusions
Amazon.com as a Testbed for Meta-Opinion Analysis
Data4,000,000 reviews700,000 products
FactorsTextual
Non-Textual
C. Danescu-Niculescu-Mizil, G. Kossinets, J. Kleinberg, L. Lee How Opinions are Received by Online Communities
Problem SettingSocial Feedback Mechanisms
Empirical EvidenceConclusions
Amazon.com as a Testbed for Meta-Opinion Analysis
Data4,000,000 reviews700,000 products
FactorsTextualNon-Textual
C. Danescu-Niculescu-Mizil, G. Kossinets, J. Kleinberg, L. Lee How Opinions are Received by Online Communities
Problem SettingSocial Feedback Mechanisms
Empirical EvidenceConclusions
Amazon.com as a Testbed for Meta-Opinion Analysis
Data4,000,000 reviews700,000 products
FactorsTextualNon-Textual
C. Danescu-Niculescu-Mizil, G. Kossinets, J. Kleinberg, L. Lee How Opinions are Received by Online Communities
Problem SettingSocial Feedback Mechanisms
Empirical EvidenceConclusions
Thought Exercise
A product has a average star rating of 3:
Suppose your aim is to write a helpful review for that product.You can only alter the star rating of the review.Which would be your star rating choice?
C. Danescu-Niculescu-Mizil, G. Kossinets, J. Kleinberg, L. Lee How Opinions are Received by Online Communities
Problem SettingSocial Feedback Mechanisms
Empirical EvidenceConclusions
Social Psychology Hypotheses
Hypotheses:ConformityBrilliant-but-cruelIndividual-bias
C. Danescu-Niculescu-Mizil, G. Kossinets, J. Kleinberg, L. Lee How Opinions are Received by Online Communities
Problem SettingSocial Feedback Mechanisms
Empirical EvidenceConclusions
Social Psychology Hypotheses
Hypotheses:ConformityBrilliant-but-cruelIndividual-bias
Conformity HypothesisSocial psychology of conformity[Bond et al., Psych. Bulletin, ’96]
→ A review is evaluated as more helpfulwhen its star rating is closer to the averagestar rating for the product.
C. Danescu-Niculescu-Mizil, G. Kossinets, J. Kleinberg, L. Lee How Opinions are Received by Online Communities
Problem SettingSocial Feedback Mechanisms
Empirical EvidenceConclusions
Social Psychology Hypotheses
Hypotheses:ConformityBrilliant-but-cruelIndividual-bias
Briliant-but-cruel Hypothesis[Amabile, J. Exp. Social Psych. ’83]:“negative reviewers are perceived as moreintelligent, competent, and expert thanpositive reviewers."
→ A review is evaluated as more helpfulwhen its star rating is below to the averagestar rating for the product.
C. Danescu-Niculescu-Mizil, G. Kossinets, J. Kleinberg, L. Lee How Opinions are Received by Online Communities
Problem SettingSocial Feedback Mechanisms
Empirical EvidenceConclusions
Social Psychology Hypotheses
Hypotheses:ConformityBrilliant-but-cruelIndividual-bias
Individual-bias HypothesisWhen a user considers a review, he or shewill rate it more highly if it expresses anopinion that he or she agrees with.
→ A review is evaluated as more helpfulwhen its star rating reflects the evaluators’personal opinion about the product.
C. Danescu-Niculescu-Mizil, G. Kossinets, J. Kleinberg, L. Lee How Opinions are Received by Online Communities
Problem SettingSocial Feedback Mechanisms
Empirical EvidenceConclusions
Deviation from the mean
Hypotheses:ConformityBrilliant-but-cruelIndividual-bias
absolute deviation = |star rating − avg. star rating|
1←0←
1←
C. Danescu-Niculescu-Mizil, G. Kossinets, J. Kleinberg, L. Lee How Opinions are Received by Online Communities
Problem SettingSocial Feedback Mechanisms
Empirical EvidenceConclusions
Deviation from the mean
Hypotheses:ConformityBrilliant-but-cruelIndividual-bias
Further away from the average
Mor
e he
lpfu
l
C. Danescu-Niculescu-Mizil, G. Kossinets, J. Kleinberg, L. Lee How Opinions are Received by Online Communities
Problem SettingSocial Feedback Mechanisms
Empirical EvidenceConclusions
Deviation from the mean
Hypotheses:ConformityBrilliant-but-cruelIndividual-bias
Further away from the average
Mor
e he
lpfu
l
C. Danescu-Niculescu-Mizil, G. Kossinets, J. Kleinberg, L. Lee How Opinions are Received by Online Communities
Problem SettingSocial Feedback Mechanisms
Empirical EvidenceConclusions
Deviation from the mean
Hypotheses:ConformityBrilliant-but-cruelIndividual-bias
signed deviation = star rating − avg. star rating
−1←
0←
1←
C. Danescu-Niculescu-Mizil, G. Kossinets, J. Kleinberg, L. Lee How Opinions are Received by Online Communities
Problem SettingSocial Feedback Mechanisms
Empirical EvidenceConclusions
Deviation from the mean
Hypotheses:ConformityBrilliant-but-cruelIndividual-bias
Above averageBelow average
Mor
e he
lpfu
l
C. Danescu-Niculescu-Mizil, G. Kossinets, J. Kleinberg, L. Lee How Opinions are Received by Online Communities
Problem SettingSocial Feedback Mechanisms
Empirical EvidenceConclusions
Deviation from the mean
Hypotheses:ConformityBrilliant-but-cruelIndividual-bias
Above averageBelow average
Mor
e he
lpfu
l
C. Danescu-Niculescu-Mizil, G. Kossinets, J. Kleinberg, L. Lee How Opinions are Received by Online Communities
Problem SettingSocial Feedback Mechanisms
Empirical EvidenceConclusions
Deviation from the mean
Hypotheses:ConformityBrilliant-but-cruelIndividual-bias
Above averageBelow average
Mor
e he
lpfu
l
C. Danescu-Niculescu-Mizil, G. Kossinets, J. Kleinberg, L. Lee How Opinions are Received by Online Communities
Problem SettingSocial Feedback Mechanisms
Empirical EvidenceConclusions
Individual-bias Hypothesis
Hypotheses:ConformityBrilliant-but-cruelIndividual-bias
Individual-bias Hypothesis→ A review is considered more helpful whenits star rating reflects the evaluators’ personalopinion about the product.
C. Danescu-Niculescu-Mizil, G. Kossinets, J. Kleinberg, L. Lee How Opinions are Received by Online Communities
Problem SettingSocial Feedback Mechanisms
Empirical EvidenceConclusions
Individual-bias vs. Conformity
Hypotheses:ConformityBrilliant-but-cruelIndividual-bias
Recall: Conformity Hypothesis→ A review is considered more helpful whenits star rating reflects the reviewers’ averageopinion about that product.
Individual-bias Hypothesis→ A review is considered more helpful whenits star rating reflects the evaluators’ personalopinion about the product.
C. Danescu-Niculescu-Mizil, G. Kossinets, J. Kleinberg, L. Lee How Opinions are Received by Online Communities
Problem SettingSocial Feedback Mechanisms
Empirical EvidenceConclusions
Individual-bias vs. Conformity
Hypotheses:ConformityBrilliant-but-cruelIndividual-bias
Recall: Conformity Hypothesis→ A review is considered more helpful whenits star rating reflects the reviewers’ averageopinion about that product.
Individual-bias Hypothesis→ A review is considered more helpful whenits star rating reflects the evaluators’ personalopinion about the product.
Individual-bias = Conformity ?Undistinguishable if everybody agrees.
C. Danescu-Niculescu-Mizil, G. Kossinets, J. Kleinberg, L. Lee How Opinions are Received by Online Communities
Problem SettingSocial Feedback Mechanisms
Empirical EvidenceConclusions
Variance
Hypotheses:ConformityBrilliant-but-cruelIndividual-bias
Individual-bias = Conformity ?Undistinguishable if everybody agrees.
C. Danescu-Niculescu-Mizil, G. Kossinets, J. Kleinberg, L. Lee How Opinions are Received by Online Communities
Problem SettingSocial Feedback Mechanisms
Empirical EvidenceConclusions
Variance
→ “Go with the average."
→ “Go slightly above the avg."
→ “Avoid the average."
C. Danescu-Niculescu-Mizil, G. Kossinets, J. Kleinberg, L. Lee How Opinions are Received by Online Communities
Problem SettingSocial Feedback Mechanisms
Empirical EvidenceConclusions
Variance
→ “Go with the average."
→ “Go slightly above the avg."
→ “Avoid the average."
C. Danescu-Niculescu-Mizil, G. Kossinets, J. Kleinberg, L. Lee How Opinions are Received by Online Communities
Problem SettingSocial Feedback Mechanisms
Empirical EvidenceConclusions
Variance
→ “Go with the average."
→ “Go slightly above the avg."
→ “Avoid the average."
C. Danescu-Niculescu-Mizil, G. Kossinets, J. Kleinberg, L. Lee How Opinions are Received by Online Communities
Problem SettingSocial Feedback Mechanisms
Empirical EvidenceConclusions
Variance
→ “Go with the average."
→ “Go slightly above the avg."
→ “Avoid the average."
C. Danescu-Niculescu-Mizil, G. Kossinets, J. Kleinberg, L. Lee How Opinions are Received by Online Communities
Problem SettingSocial Feedback Mechanisms
Empirical EvidenceConclusions
Mixture of Opinion Model
Empirical observation: Model generated:
C. Danescu-Niculescu-Mizil, G. Kossinets, J. Kleinberg, L. Lee How Opinions are Received by Online Communities
Problem SettingSocial Feedback Mechanisms
Empirical EvidenceConclusions
What about the actual text?
Hypotheses:ConformityBrilliant-but-cruelIndividual-biasText-only
C. Danescu-Niculescu-Mizil, G. Kossinets, J. Kleinberg, L. Lee How Opinions are Received by Online Communities
Problem SettingSocial Feedback Mechanisms
Empirical EvidenceConclusions
What about the actual text?
Hypotheses:ConformityBrilliant-but-cruelIndividual-biasText-only
Text-only HypothesisHelpfulness is evaluated purely on thetextual content of the review.
The non-textual factors are simplycorrelates of textual quality.
C. Danescu-Niculescu-Mizil, G. Kossinets, J. Kleinberg, L. Lee How Opinions are Received by Online Communities
Problem SettingSocial Feedback Mechanisms
Empirical EvidenceConclusions
What about the actual text?
Hypotheses:ConformityBrilliant-but-cruelIndividual-biasText-only
Controlling for Text
Manual re-evaluation of thetext helpfulness
C. Danescu-Niculescu-Mizil, G. Kossinets, J. Kleinberg, L. Lee How Opinions are Received by Online Communities
Problem SettingSocial Feedback Mechanisms
Empirical EvidenceConclusions
What about the actual text?
Hypotheses:ConformityBrilliant-but-cruelIndividual-biasText-only
Controlling for Text
Manual re-evaluation of thetext helpfulness
human effortsubjectivity
C. Danescu-Niculescu-Mizil, G. Kossinets, J. Kleinberg, L. Lee How Opinions are Received by Online Communities
Problem SettingSocial Feedback Mechanisms
Empirical EvidenceConclusions
What about the actual text?
Hypotheses:ConformityBrilliant-but-cruelIndividual-biasText-only
Controlling for Text
Manual re-evaluation of thetext helpfulnessUse Machine Learning to evaluatetext helpfulness
C. Danescu-Niculescu-Mizil, G. Kossinets, J. Kleinberg, L. Lee How Opinions are Received by Online Communities
Problem SettingSocial Feedback Mechanisms
Empirical EvidenceConclusions
What about the actual text?
Hypotheses:ConformityBrilliant-but-cruelIndividual-biasText-only
Controlling for Text
Manual re-evaluation of thetext helpfulnessUse Machine Learning to evaluatetext helpfulness
C. Danescu-Niculescu-Mizil, G. Kossinets, J. Kleinberg, L. Lee How Opinions are Received by Online Communities
Problem SettingSocial Feedback Mechanisms
Empirical EvidenceConclusions
What about the actual text?
Hypotheses:ConformityBrilliant-but-cruelIndividual-biasText-only
Controlling for Text
Manual re-evaluation of thetext helpfulnessUse Machine Learning to evaluatetext helpfulness
How to interpret the mismatchesbetween predicted and actualhelpfulness?ML alg. errors and non-textualinfluence are indistinguishable.
C. Danescu-Niculescu-Mizil, G. Kossinets, J. Kleinberg, L. Lee How Opinions are Received by Online Communities
Problem SettingSocial Feedback Mechanisms
Empirical EvidenceConclusions
What about the actual text?
Hypotheses:ConformityBrilliant-but-cruelIndividual-biasText-only
Controlling for Text
Manual re-evaluation of thetext helpfulnessUse Machine Learning to evaluatetext helpfulness
C. Danescu-Niculescu-Mizil, G. Kossinets, J. Kleinberg, L. Lee How Opinions are Received by Online Communities
Problem SettingSocial Feedback Mechanisms
Empirical EvidenceConclusions
What about the actual text?
Hypotheses:ConformityBrilliant-but-cruelIndividual-biasText-only
Controlling for Text
Manual re-evaluation of thetext helpfulnessUse Machine Learning to evaluatetext helpfulnessSeed reviews in different non-textualcontexts
C. Danescu-Niculescu-Mizil, G. Kossinets, J. Kleinberg, L. Lee How Opinions are Received by Online Communities
Problem SettingSocial Feedback Mechanisms
Empirical EvidenceConclusions
What about the actual text?
Hypotheses:ConformityBrilliant-but-cruelIndividual-biasText-only
Controlling for Text
Manual re-evaluation of thetext helpfulnessUse Machine Learning to evaluatetext helpfulnessSeed reviews in different non-textualcontexts
wait for a few years...submit to WWW 2011
C. Danescu-Niculescu-Mizil, G. Kossinets, J. Kleinberg, L. Lee How Opinions are Received by Online Communities
Problem SettingSocial Feedback Mechanisms
Empirical EvidenceConclusions
C. Danescu-Niculescu-Mizil, G. Kossinets, J. Kleinberg, L. Lee How Opinions are Received by Online Communities
Problem SettingSocial Feedback Mechanisms
Empirical EvidenceConclusions
Plagiarism!
Hypotheses:ConformityBrilliant-but-cruelIndividual-biasText-only
C. Danescu-Niculescu-Mizil, G. Kossinets, J. Kleinberg, L. Lee How Opinions are Received by Online Communities
Problem SettingSocial Feedback Mechanisms
Empirical EvidenceConclusions
Plagiarism!
Hypotheses:ConformityBrilliant-but-cruelIndividual-biasText-only
Controling for TextAbout 1% of the reviews are plagiarized![David and Pinch, 2006]
(MH statistical significance test)
C. Danescu-Niculescu-Mizil, G. Kossinets, J. Kleinberg, L. Lee How Opinions are Received by Online Communities
Problem SettingSocial Feedback Mechanisms
Empirical EvidenceConclusions
Plagiarism!
Hypotheses:ConformityBrilliant-but-cruelIndividual-biasText-only
Controling for TextAbout 1% of the reviews are plagiarized![David and Pinch, 2006]
: the plagiarized reviews with deviation 0are significantly more helpful than those withdeviation 3(MH statistical significance test)
C. Danescu-Niculescu-Mizil, G. Kossinets, J. Kleinberg, L. Lee How Opinions are Received by Online Communities
Problem SettingSocial Feedback Mechanisms
Empirical EvidenceConclusions
Conclusions
We have analyzed how the helpfulness of a review depends onthe other star ratings of the respective product.
This dependence contrasts with theories from social psychology.
We have discovered the important effects of variance onmeta-opinions.
We proposed a simple mathematical model that can account forthis apparently complex dependence.
By taking advantage of plagiarism we proved that helpfulnessevaluation does not only depends on the text, but also dependsdirectly on non-textual factors.
Thank you!
C. Danescu-Niculescu-Mizil, G. Kossinets, J. Kleinberg, L. Lee How Opinions are Received by Online Communities