investigation of various factorization methods for large recommender systems g. takács, i....
TRANSCRIPT
Investigation of Various Factorization Methods for Large Recommender
Systems
G. Takács, I. Pilászy, B. Németh and D. Tikk
www.gravityrd.com
10th International Workshop on High Performance Data Mining (in conjunction with ICDM)
Pisa, December 15th 2008
Content
Problem definition Approaches Matrix factorization
Basics, BRISMF, Semipositive, Retraining Further enhancements
Transductive MF, Neighbor based correction Experimental results
Collaborative filtering
Problem definition I.
1 4 3
44
2 44
Problem definition II.
The phenomenon can be modeled by the random triplet (U, I, R).
A realization of the phenomenon (u, i, r) means that the u-th user rated the i-th item with value r.
user id (range: {1, …, M})item id (range: {1, …, N})rating value (range: {r1, …, rL})
Problem definition III.
The goal: predict R from on (U, I). Error criterion: mean squared error (RMSE). The task is nothing else than the classical
regression estimation. Classical methods fail because of the unusual
characteristics of the predictor variables.
Content
Problem definition Approaches Matrix factorization
Basics, BRISMF, Semipositive, Retraining Further enhancements
Transductive MF, Neighbor based correction Experimental results
Approaches
Matrix factorization: approximates the rating matrix by the product of two lower-rank matrices.
Neighbor based approach: defines similarity between the rows or the columns of the rating matrix.
Support based approach: characterizes the users based on the binarized rating matrix.
Restricted Boltzmann machine: models each user by a stochastic, recurrent neural network.
Global effects: cascades 1-variable predictors.
Content
Problem definition Approaches Matrix factorization
Basics, BRISMF, Semipositive, Retraining Further enhancements
Transductive MF, Neighbor based correction Experimental results
Matrix Factorization (MF)
Idea: approximate the rating matrix as the product of two lower-rank matrices
R ≈ P ∙ Q
Problem: huge number of parameters (e.g. 10
million), R is partially unknown. Solution: incremental gradient descent.
P: user feature matrix (M x K)
Q: item feature matrix: (K x N)R: rating matrix (M x N)
MF sample - learning
Q
P1 4 3
44
2 44
R1.2
1.2
0.4
-0.5
0.9
-0.4
1.4 0.8 -1.3 -0.1 0.5
-0.2 0.5 -0.4 1.6 0.3
MF sample - learning
Q
P1 4 3
44
2 44
R1.2
1.2
0.4
-0.5
0.9
-0.4
1.4 0.8 -1.3 -0.1 0.5
-0.2 0.5 -0.4 1.6 0.3
MF sample - learning
Q
P1 4 3
44
2 44
R1.1
1.2
0.4
-0.4
0.9
-0.4
1.3 0.8 -1.3 -0.1 0.5
-0.1 0.5 -0.4 1.6 0.3
MF sample - learning
Q
P1 4 3
44
2 44
R1.1
1.2
0.4
-0.4
0.9
-0.4
1.3 0.8 -1.3 -0.1 0.5
-0.1 0.5 -0.4 1.6 0.3
MF sample - learning
Q
P1 4 3
44
2 44
R1.2
1.2
0.4
-0.3
0.9
-0.4
1.3 0.9 -1.3 -0.1 0.5
-0.1 0.4 -0.4 1.6 0.3
MF sample - learning
Q
P1 4 3
44
2 44
R1.2
1.2
0.4
-0.3
0.9
-0.4
1.3 0.9 -1.3 -0.1 0.5
-0.1 0.4 -0.4 1.6 0.3
MF sample - learning
Q
P1 4 3
44
2 44
R1.1
1.2
0.4
-0.2
0.9
-0.4
1.3 0.9 -1.3 -0.0 0.5
-0.1 0.4 -0.4 1.5 0.3
MF sample - learning
Q
P1 4 3
44
2 44
R1.1
1.2
0.4
-0.2
0.9
-0.4
1.3 0.9 -1.3 -0.0 0.5
-0.1 0.4 -0.4 1.5 0.3
MF sample - learning
Q
P1 4 3
44
2 44
R1.1
1.1
0.4
-0.2
0.8
-0.4
1.3 0.9 -1.2 -0.0 0.5
-0.1 0.4 -0.3 1.5 0.3
MF sample - learning
Q
P1 4 3
44
2 44
R1.1
1.1
0.4
-0.2
0.8
-0.4
1.3 0.9 -1.2 -0.0 0.5
-0.1 0.4 -0.3 1.5 0.3
MF sample - learning
Q
P1 4 3
44
2 44
R1.1
1.2
0.4
-0.2
0.9
-0.4
1.3 0.9 -1.2 0.1 0.5
-0.1 0.4 -0.3 1.6 0.3
MF sample - learning
Q
P1 4 3
44
2 44
R1.1
1.2
0.4
-0.2
0.9
-0.4
1.3 0.9 -1.2 0.1 0.5
-0.1 0.4 -0.3 1.6 0.3
MF sample - learning
Q
P1 4 3
44
2 44
R1.1
1.2
0.5
-0.2
0.9
-0.3
1.5 0.9 -1.2 0.1 0.5
0.0 0.4 -0.3 1.6 0.3
MF sample - learning
Q
P1 4 3
44
2 44
R1.1
1.2
0.5
-0.2
0.9
-0.3
1.5 0.9 -1.2 0.1 0.5
0.0 0.4 -0.3 1.6 0.3
MF sample - learning
Q
P1 4 3
44
2 44
R1.1
1.2
0.4
-0.2
0.9
-0.2
1.5 0.9 -1.1 0.1 0.5
0.0 0.4 -0.2 1.6 0.3
MF sample - learning
Q
P1 4 3
44
2 44
R1.1
1.2
0.4
-0.2
0.9
-0.2
1.5 0.9 -1.1 0.1 0.5
0.0 0.4 -0.2 1.6 0.3
MF sample - learning
Q
P1 4 3
44
2 44
R1.1
1.2
0.5
-0.2
0.9
-0.1
1.5 0.9 -1.1 0.1 0.6
0.0 0.4 -0.2 1.6 0.2
After a while...
MF sample - learning
Q
P1 4 3
44
2 44
R1.4
0.9
2.5
1.1
1.9
-0.3
1.5 2.1 1.0 0.7 1.6
-1.0 0.8 1.6 1.8 0.0
MF sample - prediction
Q
P1 4 3
44
2 44
R1.4
0.9
2.5
1.1
1.9
-0.3
1.5 2.1 1.0 0.7 1.6
-1.0 0.8 1.6 1.8 0.0
-0.5 3.5
4.9 1.1
3.3 2.4
1.5
BRISMF
Enhancements on the previous model: User and item Biases (offsets). Regularization.
We can call this Biased Regularized Incremental Simultaneous MF (BRISMF).
This is a very effective MF variant indeed. Leaving out any of these characteristics
(B, R, I, S) leads to inferior accuracy.
Semipositive MF
It is useful to put a nonnegativity constraint on the user feature matrix P.
There are many possible ways to implement this (e.g. PLSA, alternating least squares).
Our solution: if a user feature becomes negative after the update, then it is set to zero.
Reset User Features
Disadvantage of BRISMF: user features updated at the beginning of an epoch may be inappropriate at the end of the epoch.
Solution: 1) Reset user features at the end of the training. 2A) Retrain user features. 2B) Retrain both user and item features.
R P
Q
P'
Content
Problem definition Approaches Matrix factorization
Basics, BRISMF, Semipositive, Retraining
Further enhancements Transductive MF, Neighbor based correction
Experimental results
Transductive MF
How is it possible to use the Netflix Qualifying set in the correction phase?
We use the following simple solution:
Fast and Accurate NB Correction I.
Neighbor based (NB) methods can improve the accuracy of factor models, but conventional NB methods are not scalable.
Is it possible to integrate the NB approach into the factor model without losing scalability?
Fast and Accurate NB Correction II.
Where sjk is (normalized scalar product based similarity):
OR (normalized Euclidean distance based similarity)
NB Correction sample
Q
P1 4
R1.4 1.6
1.5 2.1 1.0 2.2 1.6
-1.0 0.8 1.6 0.7 0.0
4.20.5 4.2
4.1
Similarity: 0.2, Error: -0.5
Similarity: 0.8, Error: +0.2
Correction: -0.1
Content
Problem definition Approaches Matrix factorization
Basics, BRISMF, Semipositive, Retraining Further enhancements
Transductive MF, Neighbor based correction
Experimental results
Results I.
Method Name of our methodOur Probe 10 Our Quize Bell et al's QuizeSimple MF BRISMF#10000.8938 0.8939 0.8998Retrained MF BRISMF#1000UM0.8921 0.8918 N/AMF with neighbor correctionBRISMF#1000UM,S10.8905 0.8904 0.8953
MethodName of our
methodOur Probe10 Our Quiz Bell et al's Quiz
Simple MFBRISMF
#10000.8938 0.8939 0.8998
Retrained MFBRISMF
#1000UM0.8921 0.8918 N/A
MF with neighbor correction
BRISMF
#1000UM/S20.8905 0.8904 0.8953
Results II.
Method Name of our methodOur Probe 10 Our Quize Bell et al's QuizeSimple MF BRISMF#10000.8938 0.8939 0.8998Retrained MF BRISMF#1000UM0.8921 0.8918 N/AMF with neighbor correctionBRISMF#1000UM,S10.8905 0.8904 0.8953
Epoch Training Time (sec) RMSE
1 120 0.9188
2 200 0.9071
3 280 0.9057
4 360 0.9028
5 440 0.9008
6 520 0.9002
Thanks!
?