modeling information seeking behavior in social media
DESCRIPTION
Modeling Information Seeking Behavior in Social Media. Eugene Agichtein Emory University. Yandong Liu (2 n d year Phd ). Intelligent Information Access Lab (IRLab). Text and data mining Modeling information seeking behavior Web search and social media search - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Modeling Information Seeking Behavior in Social Media](https://reader035.vdocuments.site/reader035/viewer/2022070421/56816359550346895dd415e0/html5/thumbnails/1.jpg)
Modeling Information Seeking Behavior in Social Media
Eugene AgichteinEmory University
![Page 2: Modeling Information Seeking Behavior in Social Media](https://reader035.vdocuments.site/reader035/viewer/2022070421/56816359550346895dd415e0/html5/thumbnails/2.jpg)
2
Intelligent Information Access Lab (IRLab)
Qi Guo (2nd year Phd)
Yandong Liu (2nd year Phd)
Ablimit Aji (1st year PhD)
• Text and data mining• Modeling information seeking behavior• Web search and social media search• Tools for medical informatics and public health
Supported by:
External collaborators:- Beth Buffalo (Neurology)- Charlie Clarke (Waterloo)- Ernie Garcia (Radiology)- Phil Wolff (Psychology)- Hongyuan Zha (GaTech)
![Page 3: Modeling Information Seeking Behavior in Social Media](https://reader035.vdocuments.site/reader035/viewer/2022070421/56816359550346895dd415e0/html5/thumbnails/3.jpg)
3
Information sharing: blogs, forums, discussions
Search logs: queries, clicks
Client-side behavior: Gaze tracking, mouse movement, scrolling
Online Behavior and Interactions
![Page 4: Modeling Information Seeking Behavior in Social Media](https://reader035.vdocuments.site/reader035/viewer/2022070421/56816359550346895dd415e0/html5/thumbnails/4.jpg)
Research Overview
4
Social media
Health Informati
cs
Cognitive Diagnosti
csIntelligent
search
Discover Models of Behavior
(machine learning/data mining)
![Page 5: Modeling Information Seeking Behavior in Social Media](https://reader035.vdocuments.site/reader035/viewer/2022070421/56816359550346895dd415e0/html5/thumbnails/5.jpg)
Applications that Affect Millions• Search: ranking, evaluation, advertising, search
interfaces, medical search (clinicians, patients)
Collaboratively generated content: searcher intent, success, expertise, content quality
• Health informatics: self reporting of drug side effects, co-morbidity, outreach/education
• Automatic cognitive diagnostics: stress, frustration, Alzheimer’s, Parkinson's, ADHD, ….
5
![Page 6: Modeling Information Seeking Behavior in Social Media](https://reader035.vdocuments.site/reader035/viewer/2022070421/56816359550346895dd415e0/html5/thumbnails/6.jpg)
6
![Page 7: Modeling Information Seeking Behavior in Social Media](https://reader035.vdocuments.site/reader035/viewer/2022070421/56816359550346895dd415e0/html5/thumbnails/7.jpg)
(Text) Social Media TodayPublished:
4Gb/daySocial Media:
10Gb/Day
Technorati+Blogpulse120M blogs2M posts/day
Twitter: since 11/07:2M users3M msgs/day
Facebook/Myspace: 200-300M usersAvg 19 m/day
Yahoo Answers: 90M users, 20M questions, 400M answers[Data from Andrew Tomkins, SSM2008 Keynote]
Yes, we could read your blog. Or, you could tell us about your day
![Page 8: Modeling Information Seeking Behavior in Social Media](https://reader035.vdocuments.site/reader035/viewer/2022070421/56816359550346895dd415e0/html5/thumbnails/8.jpg)
8
![Page 9: Modeling Information Seeking Behavior in Social Media](https://reader035.vdocuments.site/reader035/viewer/2022070421/56816359550346895dd415e0/html5/thumbnails/9.jpg)
9
Total time: 7-10 minutes, active “work”
![Page 10: Modeling Information Seeking Behavior in Social Media](https://reader035.vdocuments.site/reader035/viewer/2022070421/56816359550346895dd415e0/html5/thumbnails/10.jpg)
Someone must know this…
![Page 11: Modeling Information Seeking Behavior in Social Media](https://reader035.vdocuments.site/reader035/viewer/2022070421/56816359550346895dd415e0/html5/thumbnails/11.jpg)
11+1 minute
![Page 12: Modeling Information Seeking Behavior in Social Media](https://reader035.vdocuments.site/reader035/viewer/2022070421/56816359550346895dd415e0/html5/thumbnails/12.jpg)
+7 hours: perfect answer
![Page 13: Modeling Information Seeking Behavior in Social Media](https://reader035.vdocuments.site/reader035/viewer/2022070421/56816359550346895dd415e0/html5/thumbnails/13.jpg)
Update (2/15/2009)
13
![Page 14: Modeling Information Seeking Behavior in Social Media](https://reader035.vdocuments.site/reader035/viewer/2022070421/56816359550346895dd415e0/html5/thumbnails/14.jpg)
14
http://answers.yahoo.com/question/index;_ylt=3?qid=20071008115118AAh1HdO
![Page 15: Modeling Information Seeking Behavior in Social Media](https://reader035.vdocuments.site/reader035/viewer/2022070421/56816359550346895dd415e0/html5/thumbnails/15.jpg)
15
![Page 16: Modeling Information Seeking Behavior in Social Media](https://reader035.vdocuments.site/reader035/viewer/2022070421/56816359550346895dd415e0/html5/thumbnails/16.jpg)
Finding Information Online (Revisited)
16
Next generation of search: Algorithmically-mediated information exchange
CQA (collaborative question answering):• Realistic information exchange
• Searching archives
• Train NLP, IR, QA systems
• Study of social behavior, norms
Content quality,
asker satisfaction
Current andfuture work
![Page 17: Modeling Information Seeking Behavior in Social Media](https://reader035.vdocuments.site/reader035/viewer/2022070421/56816359550346895dd415e0/html5/thumbnails/17.jpg)
(Some) Related Work• Adamic et al., WWW 2007, WWW 2008:
– Expertise sharing, network structure• Elsas et al., SIGIR 2008:
– Blog search• Glance et al.:
– Blog Pulse, popularity, information sharing• Harper et al., CHI 2008, 2009:
– Answer quality across multiple CQA sites• Kraut et al.:
– community participation• Kumar et al., WWW 2004, KDD 2008, …:
– Information diffusion in blogspace, network evolution
SIGIR 2009 Workshop on Searching Social Mediahttp://ir.mathcs.emory.edu/SSM2009/
17
![Page 18: Modeling Information Seeking Behavior in Social Media](https://reader035.vdocuments.site/reader035/viewer/2022070421/56816359550346895dd415e0/html5/thumbnails/18.jpg)
Finding High Quality Content in SM
• Well-written• Interesting• Relevant (answer)• Factually correct• Popular?• Provocative?• Useful?
18
As judged by professional editors
E. Agichtein, C. Castillo, D. Donato, A. Gionis, and G. Mishne, Finding High Quality Content in Social Media, in WSDM 2008
![Page 19: Modeling Information Seeking Behavior in Social Media](https://reader035.vdocuments.site/reader035/viewer/2022070421/56816359550346895dd415e0/html5/thumbnails/19.jpg)
19
Social Media Content Quality E. Agichtein, C. Castillo, D. Donato, A. Gionis, G. Mishne, Finding High Quality Content in Social Media, WSDM 2008
quality
![Page 20: Modeling Information Seeking Behavior in Social Media](https://reader035.vdocuments.site/reader035/viewer/2022070421/56816359550346895dd415e0/html5/thumbnails/20.jpg)
2020
![Page 21: Modeling Information Seeking Behavior in Social Media](https://reader035.vdocuments.site/reader035/viewer/2022070421/56816359550346895dd415e0/html5/thumbnails/21.jpg)
21
How do Question and Answer Quality relate?
![Page 22: Modeling Information Seeking Behavior in Social Media](https://reader035.vdocuments.site/reader035/viewer/2022070421/56816359550346895dd415e0/html5/thumbnails/22.jpg)
2222
![Page 23: Modeling Information Seeking Behavior in Social Media](https://reader035.vdocuments.site/reader035/viewer/2022070421/56816359550346895dd415e0/html5/thumbnails/23.jpg)
2323
![Page 24: Modeling Information Seeking Behavior in Social Media](https://reader035.vdocuments.site/reader035/viewer/2022070421/56816359550346895dd415e0/html5/thumbnails/24.jpg)
2424
![Page 25: Modeling Information Seeking Behavior in Social Media](https://reader035.vdocuments.site/reader035/viewer/2022070421/56816359550346895dd415e0/html5/thumbnails/25.jpg)
2525
![Page 26: Modeling Information Seeking Behavior in Social Media](https://reader035.vdocuments.site/reader035/viewer/2022070421/56816359550346895dd415e0/html5/thumbnails/26.jpg)
26
Community
![Page 27: Modeling Information Seeking Behavior in Social Media](https://reader035.vdocuments.site/reader035/viewer/2022070421/56816359550346895dd415e0/html5/thumbnails/27.jpg)
27
Link Analysis for Authority Estimation
Question 1
Question 2
Answer 5
Answer 1
Answer 2
Answer 4
Answer 3
User 1
User 2
User 3
User 6
User 4
User 5
Answer 6
Question 3
User 1
User 2
User 3
User 6
User 4
User 5
Kj
jAiH..0
)()(
Mi
iHjA..0
)()(
Hub (asker) Authority (answerer)
![Page 28: Modeling Information Seeking Behavior in Social Media](https://reader035.vdocuments.site/reader035/viewer/2022070421/56816359550346895dd415e0/html5/thumbnails/28.jpg)
28
Qualitative Observations
HITS effective
HITS
ineffective
![Page 29: Modeling Information Seeking Behavior in Social Media](https://reader035.vdocuments.site/reader035/viewer/2022070421/56816359550346895dd415e0/html5/thumbnails/29.jpg)
2929
Random forest classifier
![Page 30: Modeling Information Seeking Behavior in Social Media](https://reader035.vdocuments.site/reader035/viewer/2022070421/56816359550346895dd415e0/html5/thumbnails/30.jpg)
Result 1: Identifying High Quality Questions
30
![Page 31: Modeling Information Seeking Behavior in Social Media](https://reader035.vdocuments.site/reader035/viewer/2022070421/56816359550346895dd415e0/html5/thumbnails/31.jpg)
Top Features for Question Classification
• Asker popularity (“stars”)
• Punctuation density
• Question category
• Page views
• KL Divergence from reference LM31
![Page 32: Modeling Information Seeking Behavior in Social Media](https://reader035.vdocuments.site/reader035/viewer/2022070421/56816359550346895dd415e0/html5/thumbnails/32.jpg)
Identifying High Quality Answers
32
![Page 33: Modeling Information Seeking Behavior in Social Media](https://reader035.vdocuments.site/reader035/viewer/2022070421/56816359550346895dd415e0/html5/thumbnails/33.jpg)
Top Features for Answer Classification• Answer length
• Community ratings
• Answerer reputation
• Word overlap
• Kincaid readability score33
![Page 34: Modeling Information Seeking Behavior in Social Media](https://reader035.vdocuments.site/reader035/viewer/2022070421/56816359550346895dd415e0/html5/thumbnails/34.jpg)
Finding Information Online (Revisited)
34
• Next generation of search: • human-machine-human
• CQA: a case study in complex IRContent quality• Asker satisfaction• Understanding the interactions
![Page 35: Modeling Information Seeking Behavior in Social Media](https://reader035.vdocuments.site/reader035/viewer/2022070421/56816359550346895dd415e0/html5/thumbnails/35.jpg)
Dimensions of “Quality”• Well-written• Interesting• Relevant (answer)• Factually correct• Popular?• Timely?• Provocative?• Useful?
35
As judged by the asker (or community)
![Page 36: Modeling Information Seeking Behavior in Social Media](https://reader035.vdocuments.site/reader035/viewer/2022070421/56816359550346895dd415e0/html5/thumbnails/36.jpg)
Are Editor Labels “Meaningful” for CGC?
• Information seeking process: want to find useful information about topic with incomplete knowledge– N. Belkin: “Anomalous states of knowledge”
• Want to model directly if user found satisfactory information
• Specific (amenable) case: CQA
![Page 37: Modeling Information Seeking Behavior in Social Media](https://reader035.vdocuments.site/reader035/viewer/2022070421/56816359550346895dd415e0/html5/thumbnails/37.jpg)
37
Yahoo! Answers: The Good News
• Active community of millions of users in many countries and languages
• Effective for subjective information needs– Great forum for socialization/chat
• Can be invaluable for hard-to-find information not available on the web
![Page 38: Modeling Information Seeking Behavior in Social Media](https://reader035.vdocuments.site/reader035/viewer/2022070421/56816359550346895dd415e0/html5/thumbnails/38.jpg)
38
![Page 39: Modeling Information Seeking Behavior in Social Media](https://reader035.vdocuments.site/reader035/viewer/2022070421/56816359550346895dd415e0/html5/thumbnails/39.jpg)
39
Yahoo! Answers: The Bad NewsMay have to wait a long time to get a satisfactory answer
May never obtain a satisfying answer
0
5
10
15
20
25
30
35
40
1 2 3 4 5 6 7 8 9 10
1. FIFA World Cup2. Optical3. Poetry4. Football (American)5. Soccer6. Medicine7. Winter Sports8. Special Education9. General Health Care10. Outdoor Recreation
Time to close a question (hours)
![Page 40: Modeling Information Seeking Behavior in Social Media](https://reader035.vdocuments.site/reader035/viewer/2022070421/56816359550346895dd415e0/html5/thumbnails/40.jpg)
40
Predicting Asker Satisfaction
Given a question submitted by an asker in CQA, predict whether the user will be satisfied with the answers contributed by the community.
–“Satisfied” :• The asker has closed the question AND• Selected the best answer AND• Rated best answer >= 3 “stars” (# not important)
–Else, “Unsatisfied
Yandong Liu Jiang Bian
Y. Liu, J. Bian, and E. Agichtein, in SIGIR 2008
![Page 41: Modeling Information Seeking Behavior in Social Media](https://reader035.vdocuments.site/reader035/viewer/2022070421/56816359550346895dd415e0/html5/thumbnails/41.jpg)
41
ASP: Asker Satisfaction Prediction
asker is satisfied
asker is not satisfied
TextCategory
Answerer History
Asker History
Answer
Question
Wikipedia
NewsClassifier
![Page 42: Modeling Information Seeking Behavior in Social Media](https://reader035.vdocuments.site/reader035/viewer/2022070421/56816359550346895dd415e0/html5/thumbnails/42.jpg)
42
Experimental Setup: DataCrawled from Yahoo! Answers in early 2008
Questions
Answers
Askers
Categories
% Satisfied
216,170 1,963,615
158,515
100 50.7%
“Anonymized” dataset available at: http://ir.mathcs.emory.edu/shared/
1/2009: Yahoo! Webscope : “Comprehensive” Answers dataset: ~5M questions & answers.
![Page 43: Modeling Information Seeking Behavior in Social Media](https://reader035.vdocuments.site/reader035/viewer/2022070421/56816359550346895dd415e0/html5/thumbnails/43.jpg)
43
Satisfaction by Topic
Topic Questions
Answers
A per Q
Satisfied
Asker rating
Time to close by asker
2006 FIFA World Cup
1194 35,659
329.86
55.4%
2.63 47 minutes
Mental Health
151 1159 7.68 70.9%
4.30 1.5 days
Mathematics
651 2329 3.58 44.5%
4.48 33 minutes
Diet & Fitness
450 2436 5.41 68.4%
4.30 1.5 days
![Page 44: Modeling Information Seeking Behavior in Social Media](https://reader035.vdocuments.site/reader035/viewer/2022070421/56816359550346895dd415e0/html5/thumbnails/44.jpg)
44
Satisfaction Prediction: Human Judges
• Truth: asker’s rating• A random sample of 130 questions• Researchers
– Agreement: 0.82 F1: 0.45 2P*R/(P+R)
• Amazon Mechanical Turk– Five workers per question. – Agreement: 0.9 F1: 0.61 – Best when at least 4 out of 5 raters agree
![Page 45: Modeling Information Seeking Behavior in Social Media](https://reader035.vdocuments.site/reader035/viewer/2022070421/56816359550346895dd415e0/html5/thumbnails/45.jpg)
45
Performance: ASP vs. Humans (F1, Satisfied)
Classifier With Text Without Text Selected Features
ASP_SVM 0.69 0.72 0.62ASP_C4.5 0.75 0.76 0.77ASP_RandomForest
0.70 0.74 0.68
ASP_Boosting 0.67 0.67 0.67ASP_NB 0.61 0.65 0.58Best Human Perf
0.61
Baseline (random)
0.66
ASP is significantly more effective than humans
Human F1 is lower than the random baseline!
![Page 46: Modeling Information Seeking Behavior in Social Media](https://reader035.vdocuments.site/reader035/viewer/2022070421/56816359550346895dd415e0/html5/thumbnails/46.jpg)
46
Top Features by Information Gain• 0.14 Q: Askers’ previous rating• 0.14 Q: Average past rating by
asker• 0.10 UH: Member since (interval)• 0.05 UH: Average # answers for by
past Q• 0.05 UH: Previous Q resolved for the
asker• 0.04 CA: Average asker rating for
category• 0.04 UH: Total number of answers
received…
![Page 47: Modeling Information Seeking Behavior in Social Media](https://reader035.vdocuments.site/reader035/viewer/2022070421/56816359550346895dd415e0/html5/thumbnails/47.jpg)
47
“Offline” vs. “Online” Prediction
• Offline prediction (AFTER answers arrive)– All features( question, answer, asker & category)– F1: 0.77
• Online prediction (BEFORE question posted)– NO answer features– Only asker history and question features (stars,
#comments, sum of votes…)– F1: 0.74
![Page 48: Modeling Information Seeking Behavior in Social Media](https://reader035.vdocuments.site/reader035/viewer/2022070421/56816359550346895dd415e0/html5/thumbnails/48.jpg)
48
Personalized Prediction of Satisfaction
Same information != same usefulness for different searchers!
Personalization vs. “Groupization”?
Y. Liu and E. Agichtein, You've Got Answers: Personalized Models for Predicting Success in Community Question Answering, ACL 2008
![Page 49: Modeling Information Seeking Behavior in Social Media](https://reader035.vdocuments.site/reader035/viewer/2022070421/56816359550346895dd415e0/html5/thumbnails/49.jpg)
49
Example Personalized Models
![Page 50: Modeling Information Seeking Behavior in Social Media](https://reader035.vdocuments.site/reader035/viewer/2022070421/56816359550346895dd415e0/html5/thumbnails/50.jpg)
Outline
50
• Next generation of search: • Algorithmically mediated information exchange
• CQA: a case study in complex IRContent qualityAsker satisfaction
![Page 51: Modeling Information Seeking Behavior in Social Media](https://reader035.vdocuments.site/reader035/viewer/2022070421/56816359550346895dd415e0/html5/thumbnails/51.jpg)
Current Work (in Progress)• Partially supervised models of expertise
(Bian et al., WWW 2009)
• Real-time CQA
• Sentiment, temporal sensitivity analysis
• Understanding Social Media dynamics
![Page 52: Modeling Information Seeking Behavior in Social Media](https://reader035.vdocuments.site/reader035/viewer/2022070421/56816359550346895dd415e0/html5/thumbnails/52.jpg)
Answer Arrival
52
5 10 15 20 25 30 35 40 45 50 55 600
100000
200000
300000
400000
500000
600000
700000
573086
378227
146845
7226046364 34573 27322 23194 19952 17260 15481 13985
First Hour (69%)
Time in minutes
Answer number arrived in < T
![Page 53: Modeling Information Seeking Behavior in Social Media](https://reader035.vdocuments.site/reader035/viewer/2022070421/56816359550346895dd415e0/html5/thumbnails/53.jpg)
Exponential Decay Model [Lerman 2007]
![Page 54: Modeling Information Seeking Behavior in Social Media](https://reader035.vdocuments.site/reader035/viewer/2022070421/56816359550346895dd415e0/html5/thumbnails/54.jpg)
Factors Influencing Dynamics
![Page 55: Modeling Information Seeking Behavior in Social Media](https://reader035.vdocuments.site/reader035/viewer/2022070421/56816359550346895dd415e0/html5/thumbnails/55.jpg)
Example: Answer Arrival | Category
![Page 56: Modeling Information Seeking Behavior in Social Media](https://reader035.vdocuments.site/reader035/viewer/2022070421/56816359550346895dd415e0/html5/thumbnails/56.jpg)
Subjectivity
![Page 57: Modeling Information Seeking Behavior in Social Media](https://reader035.vdocuments.site/reader035/viewer/2022070421/56816359550346895dd415e0/html5/thumbnails/57.jpg)
Answer, Rating Arrival
![Page 58: Modeling Information Seeking Behavior in Social Media](https://reader035.vdocuments.site/reader035/viewer/2022070421/56816359550346895dd415e0/html5/thumbnails/58.jpg)
Preliminary Results: Modeling SM Dynamics for Real-Time Classification
• Adapt SM dynamics models to classificatione.g.: predict ratings
feature value:
![Page 59: Modeling Information Seeking Behavior in Social Media](https://reader035.vdocuments.site/reader035/viewer/2022070421/56816359550346895dd415e0/html5/thumbnails/59.jpg)
Outline
59
• Next generation of search: • Algorithmically mediated information exchange
• CQA: a case study in complex IRContent qualityAsker satisfactionUnderstanding social media dynamics
![Page 60: Modeling Information Seeking Behavior in Social Media](https://reader035.vdocuments.site/reader035/viewer/2022070421/56816359550346895dd415e0/html5/thumbnails/60.jpg)
60
Goal: Query Processing over Web and Social Systems
60
![Page 61: Modeling Information Seeking Behavior in Social Media](https://reader035.vdocuments.site/reader035/viewer/2022070421/56816359550346895dd415e0/html5/thumbnails/61.jpg)
61
Takeaways
Robust machine learning over behavior data system improvements, insights into behavior
Contextualized models for NLP and text mining system improvements, insights into interactions
Mining social media: potential for transformative impact for IR, sociology, psychology, medical informatics, public health, …
![Page 62: Modeling Information Seeking Behavior in Social Media](https://reader035.vdocuments.site/reader035/viewer/2022070421/56816359550346895dd415e0/html5/thumbnails/62.jpg)
References • Modeling web search behavior [SIGIR 2006, 2007]• Estimating content quality [WSDM 2008]• Estimating contributor authority [CIKM 2007]• Searching CQA archives [WWW 2008, WWW 2009]• Inferring asker intent [EMNLP 2008]• Predicting satisfaction [SIGIR 2008, ACL 2008, TKDE]• Coping with spam [AIRWeb 2008]
More information, datasets, papers, slides:http://www.mathcs.emory.edu/~eugene/