recommending the world's knowledge
TRANSCRIPT
Recommending the World’s KnowledgeApplication of Recommender Systems at Quora
Lei Yang ([email protected])
Quora: Share and grow the world’s knowledge
Millions of questions & answers, millions of users, thousands of topics
Data and Scale
Topic Question
User
Answer
Actions
Lots of data relations
Complex network propagation effects
Complex topics network & semantics
RecommendationProblems
Recommending the digest email
Recommend 10 most interesting answers in an
email to get users to re-engage on Quora
● Requirements
○ Relevance
○ Freshness
○ Quality
● Challenges
○ Many candidates
○ Personalized frequency and send times
● Personalized LTR model
○ Predict probability of click on answers
shown
Recommending the digest email
● Candidates
○ User/topic follows
○ Globally popular/viral content
○ Collaborative filtering suggestions
■ Relevant writers
■ Relevant topics
■ Relevant answers
● Features
○ Text features
○ User follow graph
○ Historic data
○ ...
Recommending writers for a question
Recommend experts who are best qualified
to answer a question
● Ask to Answer (A2A)
● Given a questions and a viewer, predict
how “well-suited” a candidate user is for an
A2A suggestion
● Well-suited
○ viewer send a request &
○ candidate write a good answer
Recommending writers for a question
● Candidates
○ Users who have written at least an
answer on one of the topics of the
question
○ A number of filters
● Features
○ Candidate history features
○ Viewer history features
○ Candidate-viewer relevance
○ ...
Recommending the home feed
Present most interesting stories for a user at
a given time
● Stories: questions + answers
● Relevance-ordered vs time-ordered = big
gains in engagement
● Challenges
○ Many candidate stories
○ Real-time ranking
○ Objective function
Recommending the home feed
● Candidates
○ Different temporal windows
○ Multi-stage solution with different
“streams”
● Features
○ Quality of question/answer
○ Topics the user is interested in
or knows about
○ Users the user is following
○ What is trending/popular
○ ...
Recommending topics and users
Recommend new topics and users for
the user to follow, based on
● Topics you already follow
● Users you already follow
● Interactions with
questions/answers/users
● Topic-related features
● Social graph
● …..
Many more problems…
Related Questions Sessions
Trending
Topic feed
What We’ve Learned
Learning to recommend
● ML can often solve recommendation
problems
○ Select candidates
○ ML model to score each candidate
○ Top scored candidates are returned
● Workflow
○ Build data pipeline
○ Define training label
○ Choose your model and metric
○ Create features
ML models used at Quora
● Logistic Regression
● Elastic Nets
● Gradient Boosted Decision Trees
● Random Forests
● (Deep) Neural Networks
● LambdaMART
● Matrix Factorization
● LDA
● ...
Open source project -- QMF
Quora Matrix Factorization
https://github.com/quora/qmf
● Currently BPR and WALS
● Multithreaded implementation in
C++14
Challenges
● Personalization
● Explicit v.s. implicit user feedback
● Explore v.s. exploit and the filter bubble
● Business logic in your system
● Trade-offs between different optimization goals
● Explain the recommendations
● ...
Conclusions
Conclusions
● At Quora we have not only big, but also “rich” data
● Many interesting recommendation problems and unsolved challenges
● We use ML to solve many of our recommendation problems
● Our algorithms need to understand and optimize complex aspects such
as quality, relevance, or user expertise
We are hiring! www.quora.com/careers