evaluating similarity measures: a large-scale study in the orkut social network ellen spertus...
TRANSCRIPT
![Page 1: Evaluating Similarity Measures: A Large-Scale Study in the orkut Social Network Ellen Spertus spertus@google.com](https://reader036.vdocuments.site/reader036/viewer/2022062417/55181754550346ac318b46ae/html5/thumbnails/1.jpg)
Evaluating Similarity Measures: A Large-Scale Study in the orkut
Social Network
Ellen Spertus
![Page 2: Evaluating Similarity Measures: A Large-Scale Study in the orkut Social Network Ellen Spertus spertus@google.com](https://reader036.vdocuments.site/reader036/viewer/2022062417/55181754550346ac318b46ae/html5/thumbnails/2.jpg)
Recommender systems
• What are they?
• Example: Amazon
![Page 3: Evaluating Similarity Measures: A Large-Scale Study in the orkut Social Network Ellen Spertus spertus@google.com](https://reader036.vdocuments.site/reader036/viewer/2022062417/55181754550346ac318b46ae/html5/thumbnails/3.jpg)
Controversial recommenders
“What to do when your TiVo thinks you’re gay”, Wall Street Journal, Nov. 26, 2002
http://tinyurl.com/2qyepg
![Page 4: Evaluating Similarity Measures: A Large-Scale Study in the orkut Social Network Ellen Spertus spertus@google.com](https://reader036.vdocuments.site/reader036/viewer/2022062417/55181754550346ac318b46ae/html5/thumbnails/4.jpg)
Controversial recommenders
“What to do when your TiVo thinks you’re gay”, Wall Street Journal, Nov. 26, 2002
http://tinyurl.com/2qyepg
![Page 5: Evaluating Similarity Measures: A Large-Scale Study in the orkut Social Network Ellen Spertus spertus@google.com](https://reader036.vdocuments.site/reader036/viewer/2022062417/55181754550346ac318b46ae/html5/thumbnails/5.jpg)
Controversial recommenders
“What to do when your TiVo thinks you’re gay”, Wall Street Journal, Nov. 26, 2002
http://tinyurl.com/2qyepg
![Page 6: Evaluating Similarity Measures: A Large-Scale Study in the orkut Social Network Ellen Spertus spertus@google.com](https://reader036.vdocuments.site/reader036/viewer/2022062417/55181754550346ac318b46ae/html5/thumbnails/6.jpg)
Controversial recommenders
Wal-Mart DVD recommendations
http://tinyurl.com/2gp2hm
![Page 7: Evaluating Similarity Measures: A Large-Scale Study in the orkut Social Network Ellen Spertus spertus@google.com](https://reader036.vdocuments.site/reader036/viewer/2022062417/55181754550346ac318b46ae/html5/thumbnails/7.jpg)
Controversial recommenders
Wal-Mart DVD recommendations
http://tinyurl.com/2gp2hm
![Page 8: Evaluating Similarity Measures: A Large-Scale Study in the orkut Social Network Ellen Spertus spertus@google.com](https://reader036.vdocuments.site/reader036/viewer/2022062417/55181754550346ac318b46ae/html5/thumbnails/8.jpg)
Controversial recommenders
Wal-Mart DVD recommendations
http://tinyurl.com/2gp2hm
![Page 9: Evaluating Similarity Measures: A Large-Scale Study in the orkut Social Network Ellen Spertus spertus@google.com](https://reader036.vdocuments.site/reader036/viewer/2022062417/55181754550346ac318b46ae/html5/thumbnails/9.jpg)
Google’s mission
To organize the world's information and make it universally accessible and useful.
![Page 10: Evaluating Similarity Measures: A Large-Scale Study in the orkut Social Network Ellen Spertus spertus@google.com](https://reader036.vdocuments.site/reader036/viewer/2022062417/55181754550346ac318b46ae/html5/thumbnails/10.jpg)
communities
0
500000
1000000
1500000
2000000
2500000
3000000
3500000
1/28
/200
4
2/28
/200
4
3/28
/200
4
4/28
/200
4
5/28
/200
4
6/28
/200
4
7/28
/200
4
8/28
/200
4
9/28
/200
4
10/2
8/20
04
11/2
8/20
04
Members
Communities
![Page 11: Evaluating Similarity Measures: A Large-Scale Study in the orkut Social Network Ellen Spertus spertus@google.com](https://reader036.vdocuments.site/reader036/viewer/2022062417/55181754550346ac318b46ae/html5/thumbnails/11.jpg)
Community recommender
• Goal: Per-community ranked recommendations
• How to determine?
![Page 12: Evaluating Similarity Measures: A Large-Scale Study in the orkut Social Network Ellen Spertus spertus@google.com](https://reader036.vdocuments.site/reader036/viewer/2022062417/55181754550346ac318b46ae/html5/thumbnails/12.jpg)
Community recommender
• Goal: Per-community ranked recommendations
• How to determine?– Implicit collaborative
filtering– Look for common membership
between pairs of communities
![Page 13: Evaluating Similarity Measures: A Large-Scale Study in the orkut Social Network Ellen Spertus spertus@google.com](https://reader036.vdocuments.site/reader036/viewer/2022062417/55181754550346ac318b46ae/html5/thumbnails/13.jpg)
Terminology
• Consider each community to be a set of members– B: base community (e.g., “Pizza”)– R: related community (e.g., “Cheese”)
• Similarity measure– Based on overlap |B∩R|
![Page 14: Evaluating Similarity Measures: A Large-Scale Study in the orkut Social Network Ellen Spertus spertus@google.com](https://reader036.vdocuments.site/reader036/viewer/2022062417/55181754550346ac318b46ae/html5/thumbnails/14.jpg)
Example: Pizza
![Page 15: Evaluating Similarity Measures: A Large-Scale Study in the orkut Social Network Ellen Spertus spertus@google.com](https://reader036.vdocuments.site/reader036/viewer/2022062417/55181754550346ac318b46ae/html5/thumbnails/15.jpg)
Example: Pizza
![Page 16: Evaluating Similarity Measures: A Large-Scale Study in the orkut Social Network Ellen Spertus spertus@google.com](https://reader036.vdocuments.site/reader036/viewer/2022062417/55181754550346ac318b46ae/html5/thumbnails/16.jpg)
Terminology
• Consider each community to be a set of members– B: base community (e.g., “Wine”)– R: related community (e.g., “Linux”)
• Similarity measure– Based on overlap |B∩R|– Also depends on |B| and |R|– Possibly asymmetric
![Page 17: Evaluating Similarity Measures: A Large-Scale Study in the orkut Social Network Ellen Spertus spertus@google.com](https://reader036.vdocuments.site/reader036/viewer/2022062417/55181754550346ac318b46ae/html5/thumbnails/17.jpg)
Example of asymmetry
Stanford (2756) Stanford Class of
2006 (52)5
![Page 18: Evaluating Similarity Measures: A Large-Scale Study in the orkut Social Network Ellen Spertus spertus@google.com](https://reader036.vdocuments.site/reader036/viewer/2022062417/55181754550346ac318b46ae/html5/thumbnails/18.jpg)
Similarity measures
• L1 normalization
• L2 normalization
• Pointwise mutual information– Positive correlations– Positive and negative correlations
• Salton tf-idf
• Log-odds
![Page 19: Evaluating Similarity Measures: A Large-Scale Study in the orkut Social Network Ellen Spertus spertus@google.com](https://reader036.vdocuments.site/reader036/viewer/2022062417/55181754550346ac318b46ae/html5/thumbnails/19.jpg)
L1 normalization
• Vector notation
• Set notation
![Page 20: Evaluating Similarity Measures: A Large-Scale Study in the orkut Social Network Ellen Spertus spertus@google.com](https://reader036.vdocuments.site/reader036/viewer/2022062417/55181754550346ac318b46ae/html5/thumbnails/20.jpg)
L2 normalization
• Vector notation
• Set notation
![Page 21: Evaluating Similarity Measures: A Large-Scale Study in the orkut Social Network Ellen Spertus spertus@google.com](https://reader036.vdocuments.site/reader036/viewer/2022062417/55181754550346ac318b46ae/html5/thumbnails/21.jpg)
Mutual information: positive correlation
• Formally,
• Informally, how well membership in the base community predicts membership in the related community
b b r + - r - +
![Page 22: Evaluating Similarity Measures: A Large-Scale Study in the orkut Social Network Ellen Spertus spertus@google.com](https://reader036.vdocuments.site/reader036/viewer/2022062417/55181754550346ac318b46ae/html5/thumbnails/22.jpg)
Mutual information: positive and negative correlation
b b r + - r - +
![Page 23: Evaluating Similarity Measures: A Large-Scale Study in the orkut Social Network Ellen Spertus spertus@google.com](https://reader036.vdocuments.site/reader036/viewer/2022062417/55181754550346ac318b46ae/html5/thumbnails/23.jpg)
Salton tf-idf
![Page 24: Evaluating Similarity Measures: A Large-Scale Study in the orkut Social Network Ellen Spertus spertus@google.com](https://reader036.vdocuments.site/reader036/viewer/2022062417/55181754550346ac318b46ae/html5/thumbnails/24.jpg)
LogOdds0
• Formally,
• Informally, how much likelier a member of B is to belong to R than a non-member of B is.
![Page 25: Evaluating Similarity Measures: A Large-Scale Study in the orkut Social Network Ellen Spertus spertus@google.com](https://reader036.vdocuments.site/reader036/viewer/2022062417/55181754550346ac318b46ae/html5/thumbnails/25.jpg)
LogOdds0
• Formally,
• Informally, how much likelier a member of B is to belong to R than a non-member of B is.
• This yielded the same rankings as L1.
![Page 26: Evaluating Similarity Measures: A Large-Scale Study in the orkut Social Network Ellen Spertus spertus@google.com](https://reader036.vdocuments.site/reader036/viewer/2022062417/55181754550346ac318b46ae/html5/thumbnails/26.jpg)
LogOdds
![Page 27: Evaluating Similarity Measures: A Large-Scale Study in the orkut Social Network Ellen Spertus spertus@google.com](https://reader036.vdocuments.site/reader036/viewer/2022062417/55181754550346ac318b46ae/html5/thumbnails/27.jpg)
Predictions?
• Were there significant differences among the measures?– Top-ranked recommendations– User preference
• Which measure was “best”?
• Was there a partial or total ordering of measures?
![Page 28: Evaluating Similarity Measures: A Large-Scale Study in the orkut Social Network Ellen Spertus spertus@google.com](https://reader036.vdocuments.site/reader036/viewer/2022062417/55181754550346ac318b46ae/html5/thumbnails/28.jpg)
Recommendations for “I love wine” (2400)
![Page 29: Evaluating Similarity Measures: A Large-Scale Study in the orkut Social Network Ellen Spertus spertus@google.com](https://reader036.vdocuments.site/reader036/viewer/2022062417/55181754550346ac318b46ae/html5/thumbnails/29.jpg)
Experiment
• Precomputed top 12 recommendations for each base community for each similarity measure
• When a user views a community page– Hash the community and user ID to– Select an ordered pair of measures to– Interleave, filtering out duplicates
• Track clicks of new users
![Page 30: Evaluating Similarity Measures: A Large-Scale Study in the orkut Social Network Ellen Spertus spertus@google.com](https://reader036.vdocuments.site/reader036/viewer/2022062417/55181754550346ac318b46ae/html5/thumbnails/30.jpg)
Click interpretation
base community related community M n j Member ? + non-member ?? ??
![Page 31: Evaluating Similarity Measures: A Large-Scale Study in the orkut Social Network Ellen Spertus spertus@google.com](https://reader036.vdocuments.site/reader036/viewer/2022062417/55181754550346ac318b46ae/html5/thumbnails/31.jpg)
Click interpretation
base community related community M n j Member ? + non-member ?? ??
![Page 32: Evaluating Similarity Measures: A Large-Scale Study in the orkut Social Network Ellen Spertus spertus@google.com](https://reader036.vdocuments.site/reader036/viewer/2022062417/55181754550346ac318b46ae/html5/thumbnails/32.jpg)
Overall click rate (July 1-18)
Total recommendation pages generated: 4,106,050
![Page 33: Evaluating Similarity Measures: A Large-Scale Study in the orkut Social Network Ellen Spertus spertus@google.com](https://reader036.vdocuments.site/reader036/viewer/2022062417/55181754550346ac318b46ae/html5/thumbnails/33.jpg)
Overall click rate (July 1-18)
![Page 34: Evaluating Similarity Measures: A Large-Scale Study in the orkut Social Network Ellen Spertus spertus@google.com](https://reader036.vdocuments.site/reader036/viewer/2022062417/55181754550346ac318b46ae/html5/thumbnails/34.jpg)
Overall click rate (July 1-18)
![Page 35: Evaluating Similarity Measures: A Large-Scale Study in the orkut Social Network Ellen Spertus spertus@google.com](https://reader036.vdocuments.site/reader036/viewer/2022062417/55181754550346ac318b46ae/html5/thumbnails/35.jpg)
Analysis
For each pair of similarity measures Ma and Mb and each click C, either:
• Ma recommended C more highly than Mb
• Ma and Mb recommended C equally
• Mb recommended C more highly than Ma
![Page 36: Evaluating Similarity Measures: A Large-Scale Study in the orkut Social Network Ellen Spertus spertus@google.com](https://reader036.vdocuments.site/reader036/viewer/2022062417/55181754550346ac318b46ae/html5/thumbnails/36.jpg)
Results
• Clicks leading to joinsL2 » MI1 » MI2 » IDF › L1 » LogOdds
• All clicks L2 » L1 » MI1 » MI2 › IDF» LogOdds
![Page 37: Evaluating Similarity Measures: A Large-Scale Study in the orkut Social Network Ellen Spertus spertus@google.com](https://reader036.vdocuments.site/reader036/viewer/2022062417/55181754550346ac318b46ae/html5/thumbnails/37.jpg)
Positional effects
• Original experiment– Ordered recommendations by rank
• Second experiment– Generated recommendations using L2– Pseudo-randomly ordered recommendations,
tracking clicks by placement– Tracked 1.3 M clicks between
September 22-October 21
![Page 38: Evaluating Similarity Measures: A Large-Scale Study in the orkut Social Network Ellen Spertus spertus@google.com](https://reader036.vdocuments.site/reader036/viewer/2022062417/55181754550346ac318b46ae/html5/thumbnails/38.jpg)
Results: single row (n=28108)
Namorado Para o Bulldog
![Page 39: Evaluating Similarity Measures: A Large-Scale Study in the orkut Social Network Ellen Spertus spertus@google.com](https://reader036.vdocuments.site/reader036/viewer/2022062417/55181754550346ac318b46ae/html5/thumbnails/39.jpg)
Results: single row (n=28,108)
p=.12 (not significant)
1.00 1.01 .98
![Page 40: Evaluating Similarity Measures: A Large-Scale Study in the orkut Social Network Ellen Spertus spertus@google.com](https://reader036.vdocuments.site/reader036/viewer/2022062417/55181754550346ac318b46ae/html5/thumbnails/40.jpg)
Results: two rows (n=24,459)
![Page 41: Evaluating Similarity Measures: A Large-Scale Study in the orkut Social Network Ellen Spertus spertus@google.com](https://reader036.vdocuments.site/reader036/viewer/2022062417/55181754550346ac318b46ae/html5/thumbnails/41.jpg)
Results: two rows (n=24,459)
p < .001
1.04 1.05 1.08 .97 .94 .92
![Page 42: Evaluating Similarity Measures: A Large-Scale Study in the orkut Social Network Ellen Spertus spertus@google.com](https://reader036.vdocuments.site/reader036/viewer/2022062417/55181754550346ac318b46ae/html5/thumbnails/42.jpg)
Results: 3 rows (n=1,226,659)
![Page 43: Evaluating Similarity Measures: A Large-Scale Study in the orkut Social Network Ellen Spertus spertus@google.com](https://reader036.vdocuments.site/reader036/viewer/2022062417/55181754550346ac318b46ae/html5/thumbnails/43.jpg)
Results: 3 rows (n=1,226,659)
1.11 1.06 1.04 1.01 .97 .99 1.01 .94 .87
p < .001
![Page 44: Evaluating Similarity Measures: A Large-Scale Study in the orkut Social Network Ellen Spertus spertus@google.com](https://reader036.vdocuments.site/reader036/viewer/2022062417/55181754550346ac318b46ae/html5/thumbnails/44.jpg)
Users’ reactions
• Hundreds of requests per day to add recommendations
• Angry requests from community creators– General– Specific
![Page 45: Evaluating Similarity Measures: A Large-Scale Study in the orkut Social Network Ellen Spertus spertus@google.com](https://reader036.vdocuments.site/reader036/viewer/2022062417/55181754550346ac318b46ae/html5/thumbnails/45.jpg)
Amusing recommendations
C++
![Page 46: Evaluating Similarity Measures: A Large-Scale Study in the orkut Social Network Ellen Spertus spertus@google.com](https://reader036.vdocuments.site/reader036/viewer/2022062417/55181754550346ac318b46ae/html5/thumbnails/46.jpg)
Amusing recommendations
C++ What’s she trying to say?
For every time a woman has confused you…
![Page 47: Evaluating Similarity Measures: A Large-Scale Study in the orkut Social Network Ellen Spertus spertus@google.com](https://reader036.vdocuments.site/reader036/viewer/2022062417/55181754550346ac318b46ae/html5/thumbnails/47.jpg)
Amusing recommendations
Chocolate
![Page 48: Evaluating Similarity Measures: A Large-Scale Study in the orkut Social Network Ellen Spertus spertus@google.com](https://reader036.vdocuments.site/reader036/viewer/2022062417/55181754550346ac318b46ae/html5/thumbnails/48.jpg)
Amusing recommendations
Chocolate PMS
![Page 49: Evaluating Similarity Measures: A Large-Scale Study in the orkut Social Network Ellen Spertus spertus@google.com](https://reader036.vdocuments.site/reader036/viewer/2022062417/55181754550346ac318b46ae/html5/thumbnails/49.jpg)
Allowing community owners to set recommendations
![Page 50: Evaluating Similarity Measures: A Large-Scale Study in the orkut Social Network Ellen Spertus spertus@google.com](https://reader036.vdocuments.site/reader036/viewer/2022062417/55181754550346ac318b46ae/html5/thumbnails/50.jpg)
Allowing community owners to set recommendations
![Page 51: Evaluating Similarity Measures: A Large-Scale Study in the orkut Social Network Ellen Spertus spertus@google.com](https://reader036.vdocuments.site/reader036/viewer/2022062417/55181754550346ac318b46ae/html5/thumbnails/51.jpg)
Manual recommendations
• Eight days after release– 50,876 community owners– Added 267,623 recommendations– Deleted 59,599 recommendations– Affecting 73,230 base communities and– 111,936 related communities
• Open question: How do they compare with automatic recommendations?
![Page 52: Evaluating Similarity Measures: A Large-Scale Study in the orkut Social Network Ellen Spertus spertus@google.com](https://reader036.vdocuments.site/reader036/viewer/2022062417/55181754550346ac318b46ae/html5/thumbnails/52.jpg)
Future research 1
Determining similar users based on common communities– Is it useful?– Will the measures make the same total order?
(9 users)
![Page 53: Evaluating Similarity Measures: A Large-Scale Study in the orkut Social Network Ellen Spertus spertus@google.com](https://reader036.vdocuments.site/reader036/viewer/2022062417/55181754550346ac318b46ae/html5/thumbnails/53.jpg)
Other types of information
• Distance in social network
• Demographic– Country– Age– Etc.
![Page 54: Evaluating Similarity Measures: A Large-Scale Study in the orkut Social Network Ellen Spertus spertus@google.com](https://reader036.vdocuments.site/reader036/viewer/2022062417/55181754550346ac318b46ae/html5/thumbnails/54.jpg)
Future research 2
Per-user community recommendations– Using social network information– Using profile information (e.g., country)
![Page 55: Evaluating Similarity Measures: A Large-Scale Study in the orkut Social Network Ellen Spertus spertus@google.com](https://reader036.vdocuments.site/reader036/viewer/2022062417/55181754550346ac318b46ae/html5/thumbnails/55.jpg)
Future research 2
Per-user community recommendations– Using social network information– Using profile information (e.g., country)
![Page 56: Evaluating Similarity Measures: A Large-Scale Study in the orkut Social Network Ellen Spertus spertus@google.com](https://reader036.vdocuments.site/reader036/viewer/2022062417/55181754550346ac318b46ae/html5/thumbnails/56.jpg)
Future research 2
Per-user community recommendations– Using social network information– Using profile information (e.g., country)
![Page 57: Evaluating Similarity Measures: A Large-Scale Study in the orkut Social Network Ellen Spertus spertus@google.com](https://reader036.vdocuments.site/reader036/viewer/2022062417/55181754550346ac318b46ae/html5/thumbnails/57.jpg)
Future research 3
Do we get the same ordering for other domains?
L2 » MI1 » MI2 » IDF › L1 » LogOdds
![Page 58: Evaluating Similarity Measures: A Large-Scale Study in the orkut Social Network Ellen Spertus spertus@google.com](https://reader036.vdocuments.site/reader036/viewer/2022062417/55181754550346ac318b46ae/html5/thumbnails/58.jpg)
Acknowledgments
• Mehran Sahami
• Orkut Buyukkokten
• orkut team
![Page 59: Evaluating Similarity Measures: A Large-Scale Study in the orkut Social Network Ellen Spertus spertus@google.com](https://reader036.vdocuments.site/reader036/viewer/2022062417/55181754550346ac318b46ae/html5/thumbnails/59.jpg)
Bonus material
![Page 60: Evaluating Similarity Measures: A Large-Scale Study in the orkut Social Network Ellen Spertus spertus@google.com](https://reader036.vdocuments.site/reader036/viewer/2022062417/55181754550346ac318b46ae/html5/thumbnails/60.jpg)
Self-rated beauty
• “beauty contest winners”
• “very attractive”
• “attractive”
• “average”
• “mirror-cracking material”
![Page 61: Evaluating Similarity Measures: A Large-Scale Study in the orkut Social Network Ellen Spertus spertus@google.com](https://reader036.vdocuments.site/reader036/viewer/2022062417/55181754550346ac318b46ae/html5/thumbnails/61.jpg)
Self-rated beauty: men
• “beauty contest winners” 8%
• “very attractive” 18%
• “attractive” 39%
• “average” 24%
• “mirror-cracking material”11%
![Page 62: Evaluating Similarity Measures: A Large-Scale Study in the orkut Social Network Ellen Spertus spertus@google.com](https://reader036.vdocuments.site/reader036/viewer/2022062417/55181754550346ac318b46ae/html5/thumbnails/62.jpg)
Self-rated beauty: women
• “beauty contest winners” 8%
• “very attractive” 16%
• “attractive” 39%
• “average” 27%
• “mirror-cracking material”9%
![Page 63: Evaluating Similarity Measures: A Large-Scale Study in the orkut Social Network Ellen Spertus spertus@google.com](https://reader036.vdocuments.site/reader036/viewer/2022062417/55181754550346ac318b46ae/html5/thumbnails/63.jpg)
Self-rated beauty by country
• Most beautiful– men:– women:
• Least beautiful– men:– women:
![Page 64: Evaluating Similarity Measures: A Large-Scale Study in the orkut Social Network Ellen Spertus spertus@google.com](https://reader036.vdocuments.site/reader036/viewer/2022062417/55181754550346ac318b46ae/html5/thumbnails/64.jpg)
Self-rated beauty by country
• Most beautiful– men: Syrian– women: Barbadian
• Least beautiful– men: Gambian– women: Ascension Islanders
![Page 65: Evaluating Similarity Measures: A Large-Scale Study in the orkut Social Network Ellen Spertus spertus@google.com](https://reader036.vdocuments.site/reader036/viewer/2022062417/55181754550346ac318b46ae/html5/thumbnails/65.jpg)
Ratings by others
• Karma– trustiness– sexiness– coolness
• How do these correlate with age?
![Page 66: Evaluating Similarity Measures: A Large-Scale Study in the orkut Social Network Ellen Spertus spertus@google.com](https://reader036.vdocuments.site/reader036/viewer/2022062417/55181754550346ac318b46ae/html5/thumbnails/66.jpg)
Ratings by others
![Page 67: Evaluating Similarity Measures: A Large-Scale Study in the orkut Social Network Ellen Spertus spertus@google.com](https://reader036.vdocuments.site/reader036/viewer/2022062417/55181754550346ac318b46ae/html5/thumbnails/67.jpg)
Friend counts
![Page 68: Evaluating Similarity Measures: A Large-Scale Study in the orkut Social Network Ellen Spertus spertus@google.com](https://reader036.vdocuments.site/reader036/viewer/2022062417/55181754550346ac318b46ae/html5/thumbnails/68.jpg)
Self-rated best body part