multiple objectives in collaborative filtering (recsys 2010)
TRANSCRIPT
Multiple Objectives in Collaborative Filtering
Tamas Jambor and Jun Wang
University College London
Structure of the talk
• Motivation• Multiple objectives• User perspective
– Promoting less popular items
• System perspective– Stock management
Motivation
• In the RecSys community, many research efforts are focused on recommendation accuracy
• And yet accuracy is not a only concern • Practical recommender systems might have
multiple goals
Improved Accuracy != Improved User experience
Algorithm Additional factors
External factors
System related
User related
Speed
Accuracy
Available resources
Cost of delivery
User interface
Diverse choices
Profitability per item
Advertisement
Improved user experience
Available resources
Cost of delivery
User interface
Diverse choices
Profitability per item
Advertisement
Additional factors
Accuracy
Improved user experience
Handling Multiple objectives
• Accuracy is the main objective– Defined in the baseline algorithm
• User perspective– Define and consider user satisfaction as priority
• System perspective– Consider additional system related objectives
• Objectives of the system might contradict
Where to optimize?
• In the objective function or as a post-filter?• Post-filters have the advantage to
– Add to any baseline algorithm– Extend easily – Add multiple goals
The proposed optimization framework(for each user)
• Add additional constraints of w
0
11:tosubject
ˆmax
ww
rwT
Tw
Properties of the framework
• Linear optimization problem• Recommendation as a ranking problem• Constraints provide the means of biasing the
ranking
User case – Promoting the Long Tail
Current systems are biased towards popular items
1 2 3 4 5 6 7 8 9 100
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5
SVD
User-based
Item-based
Random Sample
Ranking Position
Pro
bab
ilit
y o
f an
y o
f 10
0 m
ost
po
pu
lar
item
bei
ng
at
ran
k-in
g p
osi
tio
n
Promoting the Long Tail
• Does that reflect real user needs?• Popular items might not be interesting for the user• Discovering unknown item could be more valuable• The aim is to reduce recommending popular items
– if the user is likely to be an interested in alternative choices
– keep recommending popular items otherwise
Promoting the Long Tail
• Extending the optimization framework
0
11
:tosubject
ˆmax
ww
mwm
rw
T
uT
Tw
Promoting the Long Tail and Diversification
• Diversifying the results
0
11
:tosubject
ˆmax
ww
mwm
wwrw
T
uT
TTw
Diversification
• Increase the covariance between recommended items– Reduce the risk of expanding the system– Provide a wider range of choice
Experimental setup
• MovieLens 1m dataset• 3900 movies, 6040 users• Five-fold cross validation
Evaluation metrics
• Recommendation as a ranking problem• IR measures
– Normalized discounted cumulative gain (NDCG)– Precision– Mean reciprocal rank (MRR)
• Constraint specific measures
Results: Promoting the Long Tail
1 2 3 4 5 6 7 8 9 100
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5
Baseline (SVD)
Long Tail Constraint
Long Tail Constraint and Diversi-fication (λ=6)
Random Sample
Ranking Position
Pro
bab
ilit
y o
f an
y o
f 10
0 m
ost
po
pu
lar
item
bei
ng
at
ran
k-in
g p
osi
tio
n
Results: Promoting the Long Tail
Baseline (SVD) LTC LTC and Div (λ=6)
NDCG@10 0.8808 0.8780 (-0.3%) 0.8715 (-1.0%)
P@10 0.8204 0.8207 (+0.2%) 0.8177 (-0.3%)
MRR 0.9518 0.9453 (-0.6%) 0.9349 (-1.7%)
System case – Resource Constraint
• Introducing external factors to the system• Stock availability of recommended items• The aim is to rank items lower, if less of them are
available• Minimizing performance loss
Simulation
• Online DVD-Rental company– Operates a warehouse– Only a limited number of items are available
• Recommend items that are in stock higher in the ranking list
Simulation
• User choice is based purely on recommendation• Simulating the stock level for 50 days
– Present a list of items to a random number of users– The probability that the item is taken depends on the
rank– Cumulative probability depends on how many times the
item was shown and at which rank position
Cut-off point
• Threshold c controls the cut-off point from which the system starts re-ranking items
cpcpss ti
tititi
,
,,,
if0 if
Resource Constraint
• Extending the optimization framework
0
11
:tosubject
ˆmax
w
w
sws
rw
T
u
Tw
Evaluation: Monitoring the waiting list size
• Waiting list– If item is not in stock, user puts it on their waiting list– When item returns, it goes out to the next user
• Waiting list size represents how long a user has to wait to get their favourite items
Results: Resource Constraint
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 200
20
40
60
80
100
120
140
160
180 baseline c=1.6 c=1.2 c=0.0
Time (days)
Nu
mb
er o
f it
ems
on
th
e w
aiti
ng
lis
t
Results: Resource Constraint
• Trade-off between low waiting list size and good performance
0.750000000000001
0.770000000000001
0.790000000000001
0.810000000000001
0.830000000000001
0.850000000000001
0.870000000000001
0.890000000000001
0.910000000000001baseline c=1.6 c=1.2 c=0.0
Time (days)
ND
CG
@3
Results: Resource Constraint
c=0 c=0.4 c=0.8 c=1.2 c=1.6
NDCG@3(mean) -12.3% -4.32% -1.03% -0.43% -0.13%
NDCG@3(max) -14.7% -5.12% -1.34% -0.56% -0.50%
P@10(mean) -6.42% -3.37% -0.86% -0.06% -0.03%
P@10(max) -8.42% -3.91% -1.11% -0.24% -0.18%
Performance loss over 50 days
Conclusion
• Recommender systems have multiple objectives• Multiple optimization framework
– Expand the system with minor performance loss– It is designed to add objectives flexibly– It can be added to any recommender system
• Two scenarios that offer practical solutions– Long-tail items– Stock simulation
Future plan
• Personalized digital content delivery– Reduce delivery cost
• Diversification and the long tail– Does recommendation kill diversity?
• Evaluate improved user experience– User studies
Thank you.
References
• Deshpande, M., Karypis, G.: Item-based top-N recommendation algorithms. ACM Trans. Inf. Syst. 22(1) (2004)
• Herlocker, J.L., Konstan, J.A., Borchers, A., Riedl, J.: An algorithmic framework for performing collaborative filtering. In: SIGIR '99. (1999)
• Koren, Y., Bell, R., Volinsky, C.: Matrix factorization techniques for recommender systems. Computer 42(8) (2009)
• Wang, J., de Vries, A.P., Reinders, M.J.T.: Unifying user-based and item-based collaborative filtering approaches by similarity fusion. In: SIGIR '06: Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, New York, NY, ACM Press