effective missing data prediction for collaborative filtering
TRANSCRIPT
![Page 1: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/1.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Effective Missing Data Prediction forCollaborative Filtering
Hao Ma, Irwin King, and Michael R. Lyu
Department of Computer Science and EngineeringThe Chinese University of Hong Kong
SIGIR 2007, Amsterdam, the NetherlandsJuly 24, 2007
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 2: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/2.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
1 IntroductionSimple Examples of Recommender SystemDefinitions of Some ConceptsA Simple CF ExamplePearson Correlation CoefficientSignificance Weighting
2 Missing Data PredictionCollaborative Filtering ChallengesUser-Item MatrixSimilar Neighbors SelectionMissing Data PredictionParameter Discussion
3 Empirical AnalysisDatasetsMetricsSummary of ExperimentsComparisonsImpact of Parameters
4 Conclusions and Future WorkConclusions and Future Work
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 3: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/3.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Simple Examples of Recommender SystemDefinitions of Some ConceptsA Simple CF ExamplePearson Correlation CoefficientSignificance Weighting
Search Using Google
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 4: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/4.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Simple Examples of Recommender SystemDefinitions of Some ConceptsA Simple CF ExamplePearson Correlation CoefficientSignificance Weighting
Search Using Google
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 5: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/5.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Simple Examples of Recommender SystemDefinitions of Some ConceptsA Simple CF ExamplePearson Correlation CoefficientSignificance Weighting
Search Using Google
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 6: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/6.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Simple Examples of Recommender SystemDefinitions of Some ConceptsA Simple CF ExamplePearson Correlation CoefficientSignificance Weighting
Searching Products on Amazon.com
If a user is viewing the palm Treo 750 Smartphone on Amazon.com, otherrelated information will be recommended to user besides the specificationof Treo 750
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 7: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/7.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Simple Examples of Recommender SystemDefinitions of Some ConceptsA Simple CF ExamplePearson Correlation CoefficientSignificance Weighting
Searching Products on Amazon.com
If a user is viewing the palm Treo 750 Smartphone on Amazon.com, otherrelated information will be recommended to user besides the specificationof Treo 750
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 8: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/8.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Simple Examples of Recommender SystemDefinitions of Some ConceptsA Simple CF ExamplePearson Correlation CoefficientSignificance Weighting
Searching Products on Amazon.com
These methods are very popular in many online recommendation systems
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 9: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/9.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Simple Examples of Recommender SystemDefinitions of Some ConceptsA Simple CF ExamplePearson Correlation CoefficientSignificance Weighting
Searching Products on Amazon.com
These methods are very popular in many online recommendation systems
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 10: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/10.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Simple Examples of Recommender SystemDefinitions of Some ConceptsA Simple CF ExamplePearson Correlation CoefficientSignificance Weighting
More Complicated Recommendations
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 11: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/11.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Simple Examples of Recommender SystemDefinitions of Some ConceptsA Simple CF ExamplePearson Correlation CoefficientSignificance Weighting
More Complicated Recommendations
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 12: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/12.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Simple Examples of Recommender SystemDefinitions of Some ConceptsA Simple CF ExamplePearson Correlation CoefficientSignificance Weighting
More Complicated Recommendations
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 13: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/13.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Simple Examples of Recommender SystemDefinitions of Some ConceptsA Simple CF ExamplePearson Correlation CoefficientSignificance Weighting
More Complicated Recommendations
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 14: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/14.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Simple Examples of Recommender SystemDefinitions of Some ConceptsA Simple CF ExamplePearson Correlation CoefficientSignificance Weighting
More Complicated Recommendations
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 15: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/15.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Simple Examples of Recommender SystemDefinitions of Some ConceptsA Simple CF ExamplePearson Correlation CoefficientSignificance Weighting
More Complicated Recommendations
The technique Amazon.com adopts is called Collaborative Filtering!
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 16: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/16.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Simple Examples of Recommender SystemDefinitions of Some ConceptsA Simple CF ExamplePearson Correlation CoefficientSignificance Weighting
More Complicated Recommendations
The technique Amazon.com adopts is called Collaborative Filtering!
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 17: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/17.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Simple Examples of Recommender SystemDefinitions of Some ConceptsA Simple CF ExamplePearson Correlation CoefficientSignificance Weighting
Similarity calculation
Link analysis
Amazon – Simple Example
User-item matrix is consisted of lots of 0s and 1s
Frequent pattern mining
Amazon – Complicated Example
User-item matrix is consisted of lots of ratings which are rated bydifferent users
Predict other missing data as accurate as possible
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 18: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/18.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Simple Examples of Recommender SystemDefinitions of Some ConceptsA Simple CF ExamplePearson Correlation CoefficientSignificance Weighting
Similarity calculation
Link analysis
Amazon – Simple Example
User-item matrix is consisted of lots of 0s and 1s
Frequent pattern mining
Amazon – Complicated Example
User-item matrix is consisted of lots of ratings which are rated bydifferent users
Predict other missing data as accurate as possible
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 19: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/19.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Simple Examples of Recommender SystemDefinitions of Some ConceptsA Simple CF ExamplePearson Correlation CoefficientSignificance Weighting
Similarity calculation
Link analysis
Amazon – Simple Example
User-item matrix is consisted of lots of 0s and 1s
Frequent pattern mining
Amazon – Complicated Example
User-item matrix is consisted of lots of ratings which are rated bydifferent users
Predict other missing data as accurate as possible
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 20: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/20.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Simple Examples of Recommender SystemDefinitions of Some ConceptsA Simple CF ExamplePearson Correlation CoefficientSignificance Weighting
Similarity calculation
Link analysis
Amazon – Simple Example
User-item matrix is consisted of lots of 0s and 1s
Frequent pattern mining
Amazon – Complicated Example
User-item matrix is consisted of lots of ratings which are rated bydifferent users
Predict other missing data as accurate as possible
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 21: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/21.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Simple Examples of Recommender SystemDefinitions of Some ConceptsA Simple CF ExamplePearson Correlation CoefficientSignificance Weighting
Definition of Recommendation Systems
Computer programs
Predict items that a user may be interested in
Items could be movies, music, books, news,web pages, etc.
Given some information about the user’sprofile
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 22: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/22.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Simple Examples of Recommender SystemDefinitions of Some ConceptsA Simple CF ExamplePearson Correlation CoefficientSignificance Weighting
Definition of Recommendation Systems
Computer programs
Predict items that a user may be interested in
Items could be movies, music, books, news,web pages, etc.
Given some information about the user’sprofile
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 23: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/23.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Simple Examples of Recommender SystemDefinitions of Some ConceptsA Simple CF ExamplePearson Correlation CoefficientSignificance Weighting
Definition of Recommendation Systems
Computer programs
Predict items that a user may be interested in
Items could be movies, music, books, news,web pages, etc.
Given some information about the user’sprofile
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 24: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/24.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Simple Examples of Recommender SystemDefinitions of Some ConceptsA Simple CF ExamplePearson Correlation CoefficientSignificance Weighting
Definition of Recommendation Systems
Computer programs
Predict items that a user may be interested in
Items could be movies, music, books, news,web pages, etc.
Given some information about the user’sprofile
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 25: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/25.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Simple Examples of Recommender SystemDefinitions of Some ConceptsA Simple CF ExamplePearson Correlation CoefficientSignificance Weighting
Definition of Recommendation Systems
Computer programs
Predict items that a user may be interested in
Items could be movies, music, books, news,web pages, etc.
Given some information about the user’sprofile
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 26: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/26.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Simple Examples of Recommender SystemDefinitions of Some ConceptsA Simple CF ExamplePearson Correlation CoefficientSignificance Weighting
Definition of Collaborative Filtering
Making automatic predictions(filtering) about the interests of a user
By collecting taste information frommany other users (collaborating)
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 27: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/27.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Simple Examples of Recommender SystemDefinitions of Some ConceptsA Simple CF ExamplePearson Correlation CoefficientSignificance Weighting
Definition of Collaborative Filtering
Making automatic predictions(filtering) about the interests of a user
By collecting taste information frommany other users (collaborating)
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 28: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/28.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Simple Examples of Recommender SystemDefinitions of Some ConceptsA Simple CF ExamplePearson Correlation CoefficientSignificance Weighting
Definition of Collaborative Filtering
Making automatic predictions(filtering) about the interests of a user
By collecting taste information frommany other users (collaborating)
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 29: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/29.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Simple Examples of Recommender SystemDefinitions of Some ConceptsA Simple CF ExamplePearson Correlation CoefficientSignificance Weighting
User-based Collaborative Filtering
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 30: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/30.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Simple Examples of Recommender SystemDefinitions of Some ConceptsA Simple CF ExamplePearson Correlation CoefficientSignificance Weighting
User-based Collaborative Filtering
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 31: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/31.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Simple Examples of Recommender SystemDefinitions of Some ConceptsA Simple CF ExamplePearson Correlation CoefficientSignificance Weighting
User-based Collaborative Filtering
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 32: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/32.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Simple Examples of Recommender SystemDefinitions of Some ConceptsA Simple CF ExamplePearson Correlation CoefficientSignificance Weighting
User-based Collaborative Filtering
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 33: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/33.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Simple Examples of Recommender SystemDefinitions of Some ConceptsA Simple CF ExamplePearson Correlation CoefficientSignificance Weighting
User-based Collaborative Filtering
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 34: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/34.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Simple Examples of Recommender SystemDefinitions of Some ConceptsA Simple CF ExamplePearson Correlation CoefficientSignificance Weighting
User-based Collaborative Filtering
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 35: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/35.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Simple Examples of Recommender SystemDefinitions of Some ConceptsA Simple CF ExamplePearson Correlation CoefficientSignificance Weighting
User-based Collaborative Filtering
User-based collaborative filtering predicts the ratings of active users basedon the ratings of similar users found in the user-item matrix
The similarity between users could be defined as:
Sim(a, u) =
∑i∈I(a)∩I(u)
(ra,i − ra) · (ru,i − ru)
√ ∑i∈I(a)∩I(u)
(ra,i − ra)2 ·√ ∑
i∈I(a)∩I(u)
(ru,i − ru)2
Sim(a, u) is ranging from [−1, 1], and a larger value means users a and uare more similar
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 36: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/36.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Simple Examples of Recommender SystemDefinitions of Some ConceptsA Simple CF ExamplePearson Correlation CoefficientSignificance Weighting
User-based Collaborative Filtering
User-based collaborative filtering predicts the ratings of active users basedon the ratings of similar users found in the user-item matrix
The similarity between users could be defined as:
Sim(a, u) =
∑i∈I(a)∩I(u)
(ra,i − ra) · (ru,i − ru)
√ ∑i∈I(a)∩I(u)
(ra,i − ra)2 ·√ ∑
i∈I(a)∩I(u)
(ru,i − ru)2
Sim(a, u) is ranging from [−1, 1], and a larger value means users a and uare more similar
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 37: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/37.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Simple Examples of Recommender SystemDefinitions of Some ConceptsA Simple CF ExamplePearson Correlation CoefficientSignificance Weighting
User-based Collaborative Filtering
User-based collaborative filtering predicts the ratings of active users basedon the ratings of similar users found in the user-item matrix
The similarity between users could be defined as:
Sim(a, u) =
∑i∈I(a)∩I(u)
(ra,i − ra) · (ru,i − ru)
√ ∑i∈I(a)∩I(u)
(ra,i − ra)2 ·√ ∑
i∈I(a)∩I(u)
(ru,i − ru)2
Sim(a, u) is ranging from [−1, 1], and a larger value means users a and uare more similar
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 38: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/38.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Simple Examples of Recommender SystemDefinitions of Some ConceptsA Simple CF ExamplePearson Correlation CoefficientSignificance Weighting
User-based Collaborative Filtering
User-based collaborative filtering predicts the ratings of active users basedon the ratings of similar users found in the user-item matrix
The similarity between users could be defined as:
Sim(a, u) =
∑i∈I(a)∩I(u)
(ra,i − ra) · (ru,i − ru)
√ ∑i∈I(a)∩I(u)
(ra,i − ra)2 ·√ ∑
i∈I(a)∩I(u)
(ru,i − ru)2
Sim(a, u) is ranging from [−1, 1], and a larger value means users a and uare more similar
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 39: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/39.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Simple Examples of Recommender SystemDefinitions of Some ConceptsA Simple CF ExamplePearson Correlation CoefficientSignificance Weighting
User-based Collaborative Filtering
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 40: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/40.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Simple Examples of Recommender SystemDefinitions of Some ConceptsA Simple CF ExamplePearson Correlation CoefficientSignificance Weighting
User-based Collaborative Filtering
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 41: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/41.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Simple Examples of Recommender SystemDefinitions of Some ConceptsA Simple CF ExamplePearson Correlation CoefficientSignificance Weighting
Item-based Collaborative Filtering
Item-based collaborative filtering predicts the ratings of active users basedon the information of similar items computed
The similarity between items could be defined as:
Sim(i, j) =
∑u∈U(i)∩U(j)
(ru,i − ri) · (ru,j − rj)√ ∑u∈U(i)∩U(j)
(ru,i − ri)2 ·√ ∑
u∈U(i)∩U(j)
(ru,j − rj)2
Like user similarity, item similarity Sim(i, j) is also ranging from [−1, 1]
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 42: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/42.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Simple Examples of Recommender SystemDefinitions of Some ConceptsA Simple CF ExamplePearson Correlation CoefficientSignificance Weighting
Item-based Collaborative Filtering
Item-based collaborative filtering predicts the ratings of active users basedon the information of similar items computed
The similarity between items could be defined as:
Sim(i, j) =
∑u∈U(i)∩U(j)
(ru,i − ri) · (ru,j − rj)√ ∑u∈U(i)∩U(j)
(ru,i − ri)2 ·√ ∑
u∈U(i)∩U(j)
(ru,j − rj)2
Like user similarity, item similarity Sim(i, j) is also ranging from [−1, 1]
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 43: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/43.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Simple Examples of Recommender SystemDefinitions of Some ConceptsA Simple CF ExamplePearson Correlation CoefficientSignificance Weighting
Item-based Collaborative Filtering
Item-based collaborative filtering predicts the ratings of active users basedon the information of similar items computed
The similarity between items could be defined as:
Sim(i, j) =
∑u∈U(i)∩U(j)
(ru,i − ri) · (ru,j − rj)√ ∑u∈U(i)∩U(j)
(ru,i − ri)2 ·√ ∑
u∈U(i)∩U(j)
(ru,j − rj)2
Like user similarity, item similarity Sim(i, j) is also ranging from [−1, 1]
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 44: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/44.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Simple Examples of Recommender SystemDefinitions of Some ConceptsA Simple CF ExamplePearson Correlation CoefficientSignificance Weighting
Item-based Collaborative Filtering
Item-based collaborative filtering predicts the ratings of active users basedon the information of similar items computed
The similarity between items could be defined as:
Sim(i, j) =
∑u∈U(i)∩U(j)
(ru,i − ri) · (ru,j − rj)√ ∑u∈U(i)∩U(j)
(ru,i − ri)2 ·√ ∑
u∈U(i)∩U(j)
(ru,j − rj)2
Like user similarity, item similarity Sim(i, j) is also ranging from [−1, 1]
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 45: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/45.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Simple Examples of Recommender SystemDefinitions of Some ConceptsA Simple CF ExamplePearson Correlation CoefficientSignificance Weighting
An Example
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 46: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/46.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Simple Examples of Recommender SystemDefinitions of Some ConceptsA Simple CF ExamplePearson Correlation CoefficientSignificance Weighting
An Example
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 47: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/47.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Simple Examples of Recommender SystemDefinitions of Some ConceptsA Simple CF ExamplePearson Correlation CoefficientSignificance Weighting
An Example
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 48: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/48.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Simple Examples of Recommender SystemDefinitions of Some ConceptsA Simple CF ExamplePearson Correlation CoefficientSignificance Weighting
Significance Weighting
We use the following equation to solve this problem:
Sim′(a, u) =Min(|Ia ∩ Iu|, γ)
γ· Sim(a, u),
where |Ia ∩ Iu| is the number of items which user a and user u rated incommon
Then the similarity between items could be defined as:
Sim′(i, j) =Min(|Ui ∩ Uj |, δ)
δ· Sim(i, j),
where |Ui ∩ Uj | is the number of users who rated both item i and item j
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 49: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/49.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Simple Examples of Recommender SystemDefinitions of Some ConceptsA Simple CF ExamplePearson Correlation CoefficientSignificance Weighting
Significance Weighting
We use the following equation to solve this problem:
Sim′(a, u) =Min(|Ia ∩ Iu|, γ)
γ· Sim(a, u),
where |Ia ∩ Iu| is the number of items which user a and user u rated incommon
Then the similarity between items could be defined as:
Sim′(i, j) =Min(|Ui ∩ Uj |, δ)
δ· Sim(i, j),
where |Ui ∩ Uj | is the number of users who rated both item i and item j
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 50: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/50.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Simple Examples of Recommender SystemDefinitions of Some ConceptsA Simple CF ExamplePearson Correlation CoefficientSignificance Weighting
Significance Weighting
We use the following equation to solve this problem:
Sim′(a, u) =Min(|Ia ∩ Iu|, γ)
γ· Sim(a, u),
where |Ia ∩ Iu| is the number of items which user a and user u rated incommon
Then the similarity between items could be defined as:
Sim′(i, j) =Min(|Ui ∩ Uj |, δ)
δ· Sim(i, j),
where |Ui ∩ Uj | is the number of users who rated both item i and item j
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 51: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/51.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Collaborative Filtering ChallengesUser-Item MatrixSimilar Neighbors SelectionMissing Data PredictionParameter Discussion
User-Item Matrix
Challenges of Collaborative Filtering
Data Sparsity
Prediction Accuracy
Scalability
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 52: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/52.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Collaborative Filtering ChallengesUser-Item MatrixSimilar Neighbors SelectionMissing Data PredictionParameter Discussion
User-Item Matrix
Challenges of Collaborative Filtering
Data Sparsity
Prediction Accuracy
Scalability
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 53: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/53.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Collaborative Filtering ChallengesUser-Item MatrixSimilar Neighbors SelectionMissing Data PredictionParameter Discussion
User-Item Matrix
Challenges of Collaborative Filtering
Data Sparsity
Prediction Accuracy
Scalability
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 54: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/54.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Collaborative Filtering ChallengesUser-Item MatrixSimilar Neighbors SelectionMissing Data PredictionParameter Discussion
User-Item Matrix
Challenges of Collaborative Filtering
Data Sparsity
Prediction Accuracy
Scalability
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 55: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/55.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Collaborative Filtering ChallengesUser-Item MatrixSimilar Neighbors SelectionMissing Data PredictionParameter Discussion
User-Item Matrix
Challenges of Collaborative Filtering
Data Sparsity
Prediction Accuracy
Scalability
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 56: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/56.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Collaborative Filtering ChallengesUser-Item MatrixSimilar Neighbors SelectionMissing Data PredictionParameter Discussion
Challenges of Collaborative Filtering
Data Sparsity
Prediction Accuracy
Scalability
Data Sparsity
Propose an algorithm to increase the density of User-Item Matrix
Only predict some of the missing data
Prediction Accuracy
Adopt significance weighting
Linearly combine user information with item information
Predict the missing data with high confidence
Our algorithm increases 6.24% of prediction accuracy over otherstate-of-the-art methods in average
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 57: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/57.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Collaborative Filtering ChallengesUser-Item MatrixSimilar Neighbors SelectionMissing Data PredictionParameter Discussion
Challenges of Collaborative Filtering
Data Sparsity
Prediction Accuracy
Scalability
Data Sparsity
Propose an algorithm to increase the density of User-Item Matrix
Only predict some of the missing data
Prediction Accuracy
Adopt significance weighting
Linearly combine user information with item information
Predict the missing data with high confidence
Our algorithm increases 6.24% of prediction accuracy over otherstate-of-the-art methods in average
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 58: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/58.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Collaborative Filtering ChallengesUser-Item MatrixSimilar Neighbors SelectionMissing Data PredictionParameter Discussion
Challenges of Collaborative Filtering
Data Sparsity
Prediction Accuracy
Scalability
Data Sparsity
Propose an algorithm to increase the density of User-Item Matrix
Only predict some of the missing data
Prediction Accuracy
Adopt significance weighting
Linearly combine user information with item information
Predict the missing data with high confidence
Our algorithm increases 6.24% of prediction accuracy over otherstate-of-the-art methods in average
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 59: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/59.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Collaborative Filtering ChallengesUser-Item MatrixSimilar Neighbors SelectionMissing Data PredictionParameter Discussion
Challenges of Collaborative Filtering
Data Sparsity
Prediction Accuracy
Scalability
Data Sparsity
Propose an algorithm to increase the density of User-Item Matrix
Only predict some of the missing data
Prediction Accuracy
Adopt significance weighting
Linearly combine user information with item information
Predict the missing data with high confidence
Our algorithm increases 6.24% of prediction accuracy over otherstate-of-the-art methods in average
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 60: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/60.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Collaborative Filtering ChallengesUser-Item MatrixSimilar Neighbors SelectionMissing Data PredictionParameter Discussion
Challenges of Collaborative Filtering
Data Sparsity
Prediction Accuracy
Scalability
Data Sparsity
Propose an algorithm to increase the density of User-Item Matrix
Only predict some of the missing data
Prediction Accuracy
Adopt significance weighting
Linearly combine user information with item information
Predict the missing data with high confidence
Our algorithm increases 6.24% of prediction accuracy over otherstate-of-the-art methods in average
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 61: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/61.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Collaborative Filtering ChallengesUser-Item MatrixSimilar Neighbors SelectionMissing Data PredictionParameter Discussion
Challenges of Collaborative Filtering
Data Sparsity
Prediction Accuracy
Scalability
Data Sparsity
Propose an algorithm to increase the density of User-Item Matrix
Only predict some of the missing data
Prediction Accuracy
Adopt significance weighting
Linearly combine user information with item information
Predict the missing data with high confidence
Our algorithm increases 6.24% of prediction accuracy over otherstate-of-the-art methods in average
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 62: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/62.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Collaborative Filtering ChallengesUser-Item MatrixSimilar Neighbors SelectionMissing Data PredictionParameter Discussion
Challenges of Collaborative Filtering
Data Sparsity
Prediction Accuracy
Scalability
Data Sparsity
Propose an algorithm to increase the density of User-Item Matrix
Only predict some of the missing data
Prediction Accuracy
Adopt significance weighting
Linearly combine user information with item information
Predict the missing data with high confidence
Our algorithm increases 6.24% of prediction accuracy over otherstate-of-the-art methods in average
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 63: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/63.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Collaborative Filtering ChallengesUser-Item MatrixSimilar Neighbors SelectionMissing Data PredictionParameter Discussion
Challenges of Collaborative Filtering
Data Sparsity
Prediction Accuracy
Scalability
Data Sparsity
Propose an algorithm to increase the density of User-Item Matrix
Only predict some of the missing data
Prediction Accuracy
Adopt significance weighting
Linearly combine user information with item information
Predict the missing data with high confidence
Our algorithm increases 6.24% of prediction accuracy over otherstate-of-the-art methods in average
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 64: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/64.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Collaborative Filtering ChallengesUser-Item MatrixSimilar Neighbors SelectionMissing Data PredictionParameter Discussion
Challenges of Collaborative Filtering
Data Sparsity
Prediction Accuracy
Scalability
Data Sparsity
Propose an algorithm to increase the density of User-Item Matrix
Only predict some of the missing data
Prediction Accuracy
Adopt significance weighting
Linearly combine user information with item information
Predict the missing data with high confidence
Our algorithm increases 6.24% of prediction accuracy over otherstate-of-the-art methods in average
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 65: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/65.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Collaborative Filtering ChallengesUser-Item MatrixSimilar Neighbors SelectionMissing Data PredictionParameter Discussion
User-Item Matrix Predicted User-Item Matrix
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 66: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/66.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Collaborative Filtering ChallengesUser-Item MatrixSimilar Neighbors SelectionMissing Data PredictionParameter Discussion
Similar Neighbors Selection
For every missing data ru,i, a set of similar users S(u) towards user u canbe generated according to:
S(u) = {ua|Sim′(ua, u) > η, ua 6= u}
where Sim′(ua, u) is computed using Significance Weighting, and η isthe user similarity threshold
At the same time, for every missing data ru,i, a set of similar items S(i)towards item i can be generated according to:
S(i) = {ik|Sim′(ik, i) > θ, ik 6= i}
where θ is the item similarity threshold
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 67: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/67.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Collaborative Filtering ChallengesUser-Item MatrixSimilar Neighbors SelectionMissing Data PredictionParameter Discussion
Similar Neighbors Selection
For every missing data ru,i, a set of similar users S(u) towards user u canbe generated according to:
S(u) = {ua|Sim′(ua, u) > η, ua 6= u}
where Sim′(ua, u) is computed using Significance Weighting, and η isthe user similarity threshold
At the same time, for every missing data ru,i, a set of similar items S(i)towards item i can be generated according to:
S(i) = {ik|Sim′(ik, i) > θ, ik 6= i}
where θ is the item similarity threshold
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 68: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/68.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Collaborative Filtering ChallengesUser-Item MatrixSimilar Neighbors SelectionMissing Data PredictionParameter Discussion
Similar Neighbors Selection
For every missing data ru,i, a set of similar users S(u) towards user u canbe generated according to:
S(u) = {ua|Sim′(ua, u) > η, ua 6= u}
where Sim′(ua, u) is computed using Significance Weighting, and η isthe user similarity threshold
At the same time, for every missing data ru,i, a set of similar items S(i)towards item i can be generated according to:
S(i) = {ik|Sim′(ik, i) > θ, ik 6= i}
where θ is the item similarity threshold
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 69: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/69.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Collaborative Filtering ChallengesUser-Item MatrixSimilar Neighbors SelectionMissing Data PredictionParameter Discussion
Missing Data Prediction Algorithm
Given the missing data ru,i, if S(u) 6= ∅ ∧ S(i) 6= ∅, the prediction ofmissing data P (ru,i) is defined as:
P (ru,i) = λ× (u +
∑ua∈S(u)
Sim′(ua, u) · (rua,i − ua)
∑ua∈S(u)
Sim′(ua, u)) +
(1− λ)× (i +
∑ik∈S(i)
Sim′(ik, i) · (ru,ik − ik)
∑ik∈S(i)
Sim′(ik, i))
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 70: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/70.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Collaborative Filtering ChallengesUser-Item MatrixSimilar Neighbors SelectionMissing Data PredictionParameter Discussion
Missing Data Prediction Algorithm
Given the missing data ru,i, if S(u) 6= ∅ ∧ S(i) 6= ∅, the prediction ofmissing data P (ru,i) is defined as:
P (ru,i) = λ× (u +
∑ua∈S(u)
Sim′(ua, u) · (rua,i − ua)
∑ua∈S(u)
Sim′(ua, u)) +
(1− λ)× (i +
∑ik∈S(i)
Sim′(ik, i) · (ru,ik − ik)
∑ik∈S(i)
Sim′(ik, i))
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 71: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/71.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Collaborative Filtering ChallengesUser-Item MatrixSimilar Neighbors SelectionMissing Data PredictionParameter Discussion
Missing Data Prediction Algorithm
If S(u) 6= ∅ ∧ S(i) = ∅, the prediction of missing data P (ru,i) is definedas:
P (ru,i) = u +
∑ua∈S(u)
Sim′(ua, u) · (rua,i − ua)
∑ua∈S(u)
Sim′(ua, u)
If S(u) = ∅ ∧ S(i) 6= ∅, the prediction of missing data P (ru,i) is definedas:
P (ru,i) = i +
∑ik∈S(i)
Sim′(ik, i) · (ru,ik − ik)
∑ik∈S(i)
Sim′(ik, i)
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 72: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/72.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Collaborative Filtering ChallengesUser-Item MatrixSimilar Neighbors SelectionMissing Data PredictionParameter Discussion
Missing Data Prediction Algorithm
If S(u) 6= ∅ ∧ S(i) = ∅, the prediction of missing data P (ru,i) is definedas:
P (ru,i) = u +
∑ua∈S(u)
Sim′(ua, u) · (rua,i − ua)
∑ua∈S(u)
Sim′(ua, u)
If S(u) = ∅ ∧ S(i) 6= ∅, the prediction of missing data P (ru,i) is definedas:
P (ru,i) = i +
∑ik∈S(i)
Sim′(ik, i) · (ru,ik − ik)
∑ik∈S(i)
Sim′(ik, i)
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 73: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/73.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Collaborative Filtering ChallengesUser-Item MatrixSimilar Neighbors SelectionMissing Data PredictionParameter Discussion
Missing Data Prediction Algorithm
If S(u) 6= ∅ ∧ S(i) = ∅, the prediction of missing data P (ru,i) is definedas:
P (ru,i) = u +
∑ua∈S(u)
Sim′(ua, u) · (rua,i − ua)
∑ua∈S(u)
Sim′(ua, u)
If S(u) = ∅ ∧ S(i) 6= ∅, the prediction of missing data P (ru,i) is definedas:
P (ru,i) = i +
∑ik∈S(i)
Sim′(ik, i) · (ru,ik − ik)
∑ik∈S(i)
Sim′(ik, i)
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 74: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/74.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Collaborative Filtering ChallengesUser-Item MatrixSimilar Neighbors SelectionMissing Data PredictionParameter Discussion
Missing Data Prediction Algorithm
If S(u) = ∅ ∧ S(i) = ∅, the prediction of missing data P (ru,i) is definedas:
P (ru,i) = 0
This consideration is different from all other existing prediction orsmoothing methods – they always try to predict all the missing data in theuser-item matrix, which will predict some missing data with bad quality
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 75: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/75.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Collaborative Filtering ChallengesUser-Item MatrixSimilar Neighbors SelectionMissing Data PredictionParameter Discussion
Missing Data Prediction Algorithm
If S(u) = ∅ ∧ S(i) = ∅, the prediction of missing data P (ru,i) is definedas:
P (ru,i) = 0
This consideration is different from all other existing prediction orsmoothing methods – they always try to predict all the missing data in theuser-item matrix, which will predict some missing data with bad quality
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 76: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/76.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Collaborative Filtering ChallengesUser-Item MatrixSimilar Neighbors SelectionMissing Data PredictionParameter Discussion
Missing Data Prediction Algorithm
If S(u) = ∅ ∧ S(i) = ∅, the prediction of missing data P (ru,i) is definedas:
P (ru,i) = 0
This consideration is different from all other existing prediction orsmoothing methods – they always try to predict all the missing data in theuser-item matrix, which will predict some missing data with bad quality
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 77: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/77.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Collaborative Filtering ChallengesUser-Item MatrixSimilar Neighbors SelectionMissing Data PredictionParameter Discussion
Parameter
γ
δ
η
θ
λ
Discussion on γ and δ
Employed to avoid overestimating the user similarities anditem similarities
Too high =⇒ users or items do not have enough neighbors=⇒ decrease of prediction accuracy
Too low =⇒ overestimate problem still exists =⇒ decreaseof prediction accuracy
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 78: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/78.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Collaborative Filtering ChallengesUser-Item MatrixSimilar Neighbors SelectionMissing Data PredictionParameter Discussion
Parameter
γ
δ
η
θ
λ
Discussion on γ and δ
Employed to avoid overestimating the user similarities anditem similarities
Too high =⇒ users or items do not have enough neighbors=⇒ decrease of prediction accuracy
Too low =⇒ overestimate problem still exists =⇒ decreaseof prediction accuracy
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 79: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/79.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Collaborative Filtering ChallengesUser-Item MatrixSimilar Neighbors SelectionMissing Data PredictionParameter Discussion
Parameter
γ
δ
η
θ
λ
Discussion on γ and δ
Employed to avoid overestimating the user similarities anditem similarities
Too high =⇒ users or items do not have enough neighbors=⇒ decrease of prediction accuracy
Too low =⇒ overestimate problem still exists =⇒ decreaseof prediction accuracy
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 80: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/80.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Collaborative Filtering ChallengesUser-Item MatrixSimilar Neighbors SelectionMissing Data PredictionParameter Discussion
Parameter
γ
δ
η
θ
λ
Discussion on γ and δ
Employed to avoid overestimating the user similarities anditem similarities
Too high =⇒ users or items do not have enough neighbors=⇒ decrease of prediction accuracy
Too low =⇒ overestimate problem still exists =⇒ decreaseof prediction accuracy
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 81: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/81.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Collaborative Filtering ChallengesUser-Item MatrixSimilar Neighbors SelectionMissing Data PredictionParameter Discussion
Parameter
γ
δ
η
θ
λ
Discussion on η and θ
Thresholds to select neighbors
Too high =⇒ few missing data need to be predicted=⇒user-item matrix is very sparse
Too low =⇒ almost all the missing data need to bepredicted =⇒ user-item matrix is very dense
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 82: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/82.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Collaborative Filtering ChallengesUser-Item MatrixSimilar Neighbors SelectionMissing Data PredictionParameter Discussion
Parameter
γ
δ
η
θ
λ
Discussion on η and θ
Thresholds to select neighbors
Too high =⇒ few missing data need to be predicted=⇒user-item matrix is very sparse
Too low =⇒ almost all the missing data need to bepredicted =⇒ user-item matrix is very dense
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 83: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/83.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Collaborative Filtering ChallengesUser-Item MatrixSimilar Neighbors SelectionMissing Data PredictionParameter Discussion
Parameter
γ
δ
η
θ
λ
Discussion on η and θ
Thresholds to select neighbors
Too high =⇒ few missing data need to be predicted=⇒user-item matrix is very sparse
Too low =⇒ almost all the missing data need to bepredicted =⇒ user-item matrix is very dense
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 84: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/84.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Collaborative Filtering ChallengesUser-Item MatrixSimilar Neighbors SelectionMissing Data PredictionParameter Discussion
Parameter
γ
δ
η
θ
λ
Discussion on η and θ
Thresholds to select neighbors
Too high =⇒ few missing data need to be predicted=⇒user-item matrix is very sparse
Too low =⇒ almost all the missing data need to bepredicted =⇒ user-item matrix is very dense
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 85: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/85.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Collaborative Filtering ChallengesUser-Item MatrixSimilar Neighbors SelectionMissing Data PredictionParameter Discussion
Parameter
γ
δ
η
θ
λ
Discussion on λ
Determines how closely the rating prediction relies on userinformation or item information
λ = 1 =⇒ prediction depends completely upon user-basedinformation
λ = 0 =⇒ prediction depends completely upon item-basedinformation
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 86: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/86.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Collaborative Filtering ChallengesUser-Item MatrixSimilar Neighbors SelectionMissing Data PredictionParameter Discussion
Parameter
γ
δ
η
θ
λ
Discussion on λ
Determines how closely the rating prediction relies on userinformation or item information
λ = 1 =⇒ prediction depends completely upon user-basedinformation
λ = 0 =⇒ prediction depends completely upon item-basedinformation
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 87: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/87.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Collaborative Filtering ChallengesUser-Item MatrixSimilar Neighbors SelectionMissing Data PredictionParameter Discussion
Parameter
γ
δ
η
θ
λ
Discussion on λ
Determines how closely the rating prediction relies on userinformation or item information
λ = 1 =⇒ prediction depends completely upon user-basedinformation
λ = 0 =⇒ prediction depends completely upon item-basedinformation
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 88: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/88.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Collaborative Filtering ChallengesUser-Item MatrixSimilar Neighbors SelectionMissing Data PredictionParameter Discussion
Parameter
γ
δ
η
θ
λ
Discussion on λ
Determines how closely the rating prediction relies on userinformation or item information
λ = 1 =⇒ prediction depends completely upon user-basedinformation
λ = 0 =⇒ prediction depends completely upon item-basedinformation
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 89: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/89.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Collaborative Filtering ChallengesUser-Item MatrixSimilar Neighbors SelectionMissing Data PredictionParameter Discussion
Parameter Discussion
Table: The relationship between parameters with other CF approaches(MDP: Mission Data Predicted)
λ η θ Related CF Approaches
1 1 1 User-based CF without MDP
0 1 1 Item-based CF without MDP
1 0 0 User-based CF with full MDP
0 0 0 Item-based CF with full MDP
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 90: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/90.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Collaborative Filtering ChallengesUser-Item MatrixSimilar Neighbors SelectionMissing Data PredictionParameter Discussion
Parameter Discussion
Table: The relationship between parameters with other CF approaches(MDP: Mission Data Predicted)
λ η θ Related CF Approaches
1 1 1 User-based CF without MDP
0 1 1 Item-based CF without MDP
1 0 0 User-based CF with full MDP
0 0 0 Item-based CF with full MDP
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 91: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/91.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
DatasetsMetricsSummary of ExperimentsComparisonsImpact of Parameters
Movielens
It contains 100,000 ratings (1-5 scales) rated by 943 users on 1,682movies, and each user at least rated 20 movies. The density of theuser-item matrix is:
100000
943× 1682= 6.30%
The statistics of dataset MovieLens is summarized in the following table:
Table: Statistics of Dataset MovieLens
Statistics User Item
Min. Num. of Ratings 20 1
Max. Num. of Ratings 737 583
Avg. Num. of Ratings 106.04 59.45
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 92: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/92.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
DatasetsMetricsSummary of ExperimentsComparisonsImpact of Parameters
Movielens
It contains 100,000 ratings (1-5 scales) rated by 943 users on 1,682movies, and each user at least rated 20 movies. The density of theuser-item matrix is:
100000
943× 1682= 6.30%
The statistics of dataset MovieLens is summarized in the following table:
Table: Statistics of Dataset MovieLens
Statistics User Item
Min. Num. of Ratings 20 1
Max. Num. of Ratings 737 583
Avg. Num. of Ratings 106.04 59.45
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 93: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/93.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
DatasetsMetricsSummary of ExperimentsComparisonsImpact of Parameters
Movielens
It contains 100,000 ratings (1-5 scales) rated by 943 users on 1,682movies, and each user at least rated 20 movies. The density of theuser-item matrix is:
100000
943× 1682= 6.30%
The statistics of dataset MovieLens is summarized in the following table:
Table: Statistics of Dataset MovieLens
Statistics User Item
Min. Num. of Ratings 20 1
Max. Num. of Ratings 737 583
Avg. Num. of Ratings 106.04 59.45
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 94: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/94.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
DatasetsMetricsSummary of ExperimentsComparisonsImpact of Parameters
Mean Absolute Errors
We use the Mean Absolute Error (MAE) metrics to measure theprediction quality of our proposed approach with other collaborativefiltering methods
MAE is defined as:
MAE =
∑u,i |ru,i − r̂u,i|
N,
where ru,i denotes the rating that user u gave to item i, and r̂u,i denotesthe rating that user u gave to item i which is predicted by our approach,and N denotes the number of tested ratings
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 95: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/95.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
DatasetsMetricsSummary of ExperimentsComparisonsImpact of Parameters
Mean Absolute Errors
We use the Mean Absolute Error (MAE) metrics to measure theprediction quality of our proposed approach with other collaborativefiltering methods
MAE is defined as:
MAE =
∑u,i |ru,i − r̂u,i|
N,
where ru,i denotes the rating that user u gave to item i, and r̂u,i denotesthe rating that user u gave to item i which is predicted by our approach,and N denotes the number of tested ratings
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 96: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/96.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
DatasetsMetricsSummary of ExperimentsComparisonsImpact of Parameters
Mean Absolute Errors
We use the Mean Absolute Error (MAE) metrics to measure theprediction quality of our proposed approach with other collaborativefiltering methods
MAE is defined as:
MAE =
∑u,i |ru,i − r̂u,i|
N,
where ru,i denotes the rating that user u gave to item i, and r̂u,i denotesthe rating that user u gave to item i which is predicted by our approach,and N denotes the number of tested ratings
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 97: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/97.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
DatasetsMetricsSummary of ExperimentsComparisonsImpact of Parameters
Summary of Experiments
Comparisons with Traditional PCC Methods
Comparisons with State-of-the-Art Algorithms
Impact of Missing Data Prediction
Impact of γ and δ
Impact of λ
Impact of η and θ
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 98: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/98.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
DatasetsMetricsSummary of ExperimentsComparisonsImpact of Parameters
Summary of Experiments
Comparisons with Traditional PCC Methods
Comparisons with State-of-the-Art Algorithms
Impact of Missing Data Prediction
Impact of γ and δ
Impact of λ
Impact of η and θ
Comparisons with Traditional PCC Methods
User-based collaborative filtering using Pearson Correlation Coefficient
Item-based collaborative filtering using Pearson Correlation Coefficient
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 99: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/99.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
DatasetsMetricsSummary of ExperimentsComparisonsImpact of Parameters
Summary of Experiments
Comparisons with Traditional PCC Methods
Comparisons with State-of-the-Art Algorithms
Impact of Missing Data Prediction
Impact of γ and δ
Impact of λ
Impact of η and θ
Comparisons with State-of-the-Art Algorithms
Similarity Fusion (SF) [J. Wang, et al., SIGIR 2006]
Smoothing and Cluster-Based PCC (SCBPCC) [G. Xue, et al., SIGIR2005]
Aspect Model (AM) [T. Hofmann, TOIS 2004]
Personality Diagnosis (PD) [D. M. Pennock, et al., UAI 2000]
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 100: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/100.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
DatasetsMetricsSummary of ExperimentsComparisonsImpact of Parameters
Summary of Experiments
Comparisons with Traditional PCC Methods
Comparisons with State-of-the-Art Algorithms
Impact of Missing Data Prediction
Impact of γ and δ
Impact of λ
Impact of η and θ
Impact of Missing Data Prediction
Effective Missing Data Prediction (EMDP)
Predict Every Missing Data (PEMD)
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 101: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/101.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
DatasetsMetricsSummary of ExperimentsComparisonsImpact of Parameters
Summary of Experiments
Comparisons with Traditional PCC Methods
Comparisons with State-of-the-Art Algorithms
Impact of Missing Data Prediction
Impact of γ and δ
Impact of λ
Impact of η and θ
Impact of Parameters
Impact of each parameter
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 102: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/102.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
DatasetsMetricsSummary of ExperimentsComparisonsImpact of Parameters
MAE Comparisons with PCC Methods
Table: MAE comparison with other approaches (A smaller MAE valuemeans a better performance)
Training Users Methods Given5 Given10 Given20
EMDP 0.784 0.765 0.755MovieLens 300 UPCC 0.838 0.814 0.802
IPCC 0.870 0.838 0.813
EMDP 0.796 0.770 0.761MovieLens 200 UPCC 0.843 0.822 0.807
IPCC 0.855 0.834 0.812
EMDP 0.811 0.778 0.769MovieLens 100 UPCC 0.876 0.847 0.811
IPCC 0.890 0.850 0.824
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 103: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/103.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
DatasetsMetricsSummary of ExperimentsComparisonsImpact of Parameters
MAE Comparisons with PCC Methods
Table: MAE comparison with other approaches (A smaller MAE valuemeans a better performance)
Training Users Methods Given5 Given10 Given20
EMDP 0.784 0.765 0.755MovieLens 300 UPCC 0.838 0.814 0.802
IPCC 0.870 0.838 0.813
EMDP 0.796 0.770 0.761MovieLens 200 UPCC 0.843 0.822 0.807
IPCC 0.855 0.834 0.812
EMDP 0.811 0.778 0.769MovieLens 100 UPCC 0.876 0.847 0.811
IPCC 0.890 0.850 0.824
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 104: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/104.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
DatasetsMetricsSummary of ExperimentsComparisonsImpact of Parameters
MAE Comparisons with State-of-the-Art Algorithms
Table: MAE comparison with state-of-the-art algorithms (A smaller MAEvalue means a better performance)
Num. of Training Users 100 200 300Ratings Given 5 10 20 5 10 20 5 10 20
EMDP 0.807 0.769 0.765 0.793 0.760 0.751 0.788 0.754 0.746SF 0.847 0.774 0.792 0.827 0.773 0.783 0.804 0.761 0.769
SCBPCC 0.848 0.819 0.789 0.831 0.813 0.784 0.822 0.810 0.778AM 0.963 0.922 0.887 0.849 0.837 0.815 0.820 0.822 0.796PD 0.849 0.817 0.808 0.836 0.815 0.792 0.827 0.815 0.789PCC 0.874 0.836 0.818 0.859 0.829 0.813 0.849 0.841 0.820
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 105: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/105.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
DatasetsMetricsSummary of ExperimentsComparisonsImpact of Parameters
MAE Comparisons with State-of-the-Art Algorithms
Table: MAE comparison with state-of-the-art algorithms (A smaller MAEvalue means a better performance)
Num. of Training Users 100 200 300Ratings Given 5 10 20 5 10 20 5 10 20
EMDP 0.807 0.769 0.765 0.793 0.760 0.751 0.788 0.754 0.746SF 0.847 0.774 0.792 0.827 0.773 0.783 0.804 0.761 0.769
SCBPCC 0.848 0.819 0.789 0.831 0.813 0.784 0.822 0.810 0.778AM 0.963 0.922 0.887 0.849 0.837 0.815 0.820 0.822 0.796PD 0.849 0.817 0.808 0.836 0.815 0.792 0.827 0.815 0.789PCC 0.874 0.836 0.818 0.859 0.829 0.813 0.849 0.841 0.820
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 106: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/106.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
DatasetsMetricsSummary of ExperimentsComparisonsImpact of Parameters
Impact of Missing Data Prediction
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10.74
0.75
0.76
0.77
0.78
0.79
0.8
0.81
0.82
0.83
Lambda
MA
E
EMDP−Given20PEMD−Given20EMDP−Given10PEMD−Given10EMDP−Given5PEMD−Given5
Figure: MAE Comparison of EMDP and PEMD (A smaller MAE valuemeans a better performance)
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 107: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/107.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
DatasetsMetricsSummary of ExperimentsComparisonsImpact of Parameters
Impact of γ and δ
Figure: Impact of γ and δ on MAE and Matrix Density
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 108: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/108.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
DatasetsMetricsSummary of ExperimentsComparisonsImpact of Parameters
Impact of λ
Figure: Impact of λ on MAE
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 109: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/109.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
DatasetsMetricsSummary of ExperimentsComparisonsImpact of Parameters
Impact of η and θ
Figure: Impact of η and θ on MAE and Density
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 110: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/110.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Conclusions and Future Work
Conclusions
Proposes an effective missing data prediction algorithm for CollaborativeFiltering
Combines users information and items information together
Outperforms other state-of-the-art collaborative filtering approaches
Future Work
Explore the relationship between user information and item information
Scalability analysis and improvement of our algorithm
Employ more metrics to measure our algorithm
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 111: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/111.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Conclusions and Future Work
Conclusions
Proposes an effective missing data prediction algorithm for CollaborativeFiltering
Combines users information and items information together
Outperforms other state-of-the-art collaborative filtering approaches
Future Work
Explore the relationship between user information and item information
Scalability analysis and improvement of our algorithm
Employ more metrics to measure our algorithm
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 112: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/112.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Conclusions and Future Work
Conclusions
Proposes an effective missing data prediction algorithm for CollaborativeFiltering
Combines users information and items information together
Outperforms other state-of-the-art collaborative filtering approaches
Future Work
Explore the relationship between user information and item information
Scalability analysis and improvement of our algorithm
Employ more metrics to measure our algorithm
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 113: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/113.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Conclusions and Future Work
Conclusions
Proposes an effective missing data prediction algorithm for CollaborativeFiltering
Combines users information and items information together
Outperforms other state-of-the-art collaborative filtering approaches
Future Work
Explore the relationship between user information and item information
Scalability analysis and improvement of our algorithm
Employ more metrics to measure our algorithm
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 114: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/114.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Conclusions and Future Work
Conclusions
Proposes an effective missing data prediction algorithm for CollaborativeFiltering
Combines users information and items information together
Outperforms other state-of-the-art collaborative filtering approaches
Future Work
Explore the relationship between user information and item information
Scalability analysis and improvement of our algorithm
Employ more metrics to measure our algorithm
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 115: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/115.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Conclusions and Future Work
Conclusions
Proposes an effective missing data prediction algorithm for CollaborativeFiltering
Combines users information and items information together
Outperforms other state-of-the-art collaborative filtering approaches
Future Work
Explore the relationship between user information and item information
Scalability analysis and improvement of our algorithm
Employ more metrics to measure our algorithm
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 116: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/116.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Conclusions and Future Work
Conclusions
Proposes an effective missing data prediction algorithm for CollaborativeFiltering
Combines users information and items information together
Outperforms other state-of-the-art collaborative filtering approaches
Future Work
Explore the relationship between user information and item information
Scalability analysis and improvement of our algorithm
Employ more metrics to measure our algorithm
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 117: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/117.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Conclusions and Future Work
Conclusions
Proposes an effective missing data prediction algorithm for CollaborativeFiltering
Combines users information and items information together
Outperforms other state-of-the-art collaborative filtering approaches
Future Work
Explore the relationship between user information and item information
Scalability analysis and improvement of our algorithm
Employ more metrics to measure our algorithm
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering
![Page 118: Effective Missing Data Prediction for Collaborative Filtering](https://reader030.vdocuments.site/reader030/viewer/2022012807/61bd492761276e740b1146ac/html5/thumbnails/118.jpg)
OutlineIntroduction
Missing Data PredictionEmpirical Analysis
Conclusions and Future Work
Conclusions and Future Work
Q & A
Home Page: http://www.cse.cuhk.edu.hk/∼hma
Email: [email protected]
Thanks!
Hao Ma, Irwin King, and Michael R. Lyu Effective Missing Data Prediction for Collaborative Filtering