Amazon review utility estimator

Overview

Goal: To determine the “usefulness” of Amazon.com reviews

Using Mallet classifiers Several custom features If accurate, this system could be applied

beyond Amazon, including other product reviews or even Slashdot/Digg comments.

Reviews

Used Amazon ECS: Collected large number of reviews over 4 categories: Textbooks, Digital Cameras, Music, DVD

Textbooks: 24,419 reviews with over 5 votes

Digital Cameras: 22,566 Music: 43,328 DVD: 132,208

Regression?

All of the length features seem to have a trend when grouped in buckets

DVD data

Avg Total Avg Word Avg Para

0-25% 133.96 5.58 1.84

26-50% 197.04 5.66 2.33

51-75% 248.72 5.68 2.79

76-100% 281.66 5.72 2.84

Regression R2 ~ .3

Rating

# of words

Regression

Rating

Avg Sentence Length

Features Bag of words Average: length, sentence length, word

length % of words that are stop words # of spelling errors # of paragraphs Pronouns, articles, Proper nouns etc. Punctuation History

Stuff We Learned Some good reviews are hard to find

“e-toys has this for 19.99” rated helpful by 17/21 people.

And some people are just stupid “and there you have it. That's the secret. ” 77%... “On DVD, I'll buy this NOW! Not on VHS...Jezus...” 78%...

We attempted manually classifying ~100 reviewsIn 4 buckets around 30% accuracyIn 2 buckets around 55% ....abstract.cs.washington.edu/~kylej1/quiz.php

Cont. Trade off between Precision and Recall:

Many features increase precision but hurt recallThe range of good reviews is very broad

Word Count / Sentence Length / % stopwords have biggest impactPrecision +5%, Recall -8%

Diminishing returns..

Cont.

Precision in the High 80s with the right combination of featuresRecall suffers, drops to between 40-50%

Experimenting with multiple classifiers in series.To boost recall without destroying precisionSimilar to Boosting.

Future When should computer override customer

rating?Amazon has huge # of “Labeled” data…but the

labels are sometimes poorReview Quality is very subjectiveWeight based on # of total votes?

○ Some concerns with this

Bias detectionPositive or Negative impact?

End

Questions?