![Page 1: RecSys 2017 Como, Italy - ACM Recommender Systemsrecsys.acm.org/wp-content/uploads/2017/10/RecSys-17... · 2017. 10. 4. · PowerPoint Presentation Author: Marta Jasicka Created Date:](https://reader035.vdocuments.site/reader035/viewer/2022071112/5fe7cd7d5d23671e840f93c1/html5/thumbnails/1.jpg)
Boosting Recommender Systems with Deep LearningJoão Gomes
RecSys 2017 – Como, Italy
![Page 2: RecSys 2017 Como, Italy - ACM Recommender Systemsrecsys.acm.org/wp-content/uploads/2017/10/RecSys-17... · 2017. 10. 4. · PowerPoint Presentation Author: Marta Jasicka Created Date:](https://reader035.vdocuments.site/reader035/viewer/2022071112/5fe7cd7d5d23671e840f93c1/html5/thumbnails/2.jpg)
230 CountriesPlatform forLuxury Fashion
300K Products4M users
2500 Brands500 Boutiques
1800+ employees20+ in Data Science
200 clickstreamevents / sec
![Page 3: RecSys 2017 Como, Italy - ACM Recommender Systemsrecsys.acm.org/wp-content/uploads/2017/10/RecSys-17... · 2017. 10. 4. · PowerPoint Presentation Author: Marta Jasicka Created Date:](https://reader035.vdocuments.site/reader035/viewer/2022071112/5fe7cd7d5d23671e840f93c1/html5/thumbnails/3.jpg)
Visual Similarity
![Page 4: RecSys 2017 Como, Italy - ACM Recommender Systemsrecsys.acm.org/wp-content/uploads/2017/10/RecSys-17... · 2017. 10. 4. · PowerPoint Presentation Author: Marta Jasicka Created Date:](https://reader035.vdocuments.site/reader035/viewer/2022071112/5fe7cd7d5d23671e840f93c1/html5/thumbnails/4.jpg)
![Page 5: RecSys 2017 Como, Italy - ACM Recommender Systemsrecsys.acm.org/wp-content/uploads/2017/10/RecSys-17... · 2017. 10. 4. · PowerPoint Presentation Author: Marta Jasicka Created Date:](https://reader035.vdocuments.site/reader035/viewer/2022071112/5fe7cd7d5d23671e840f93c1/html5/thumbnails/5.jpg)
Deep Learning for feature extraction
Off-the-shelf Model
• ResNet-50 pre-trained on ImageNet
• Previous to last layer for the embeddings
Find similar items
• Nearest neighbours with cosine similarity
Easy, fast, testable
Useful in some contexts
• Out of stock replacement
• Smart mirror in a fitting room
Visual similarity
ResNet-50
0.5, 0.1, 1.2, 0, 1, ...
0.4, 0.1, 0.6, 0, 1, ...
0.1, 0.3, 0.1, 1.5, 2...
...
0.7, 0.85, 0.1, 0, 1...
![Page 6: RecSys 2017 Como, Italy - ACM Recommender Systemsrecsys.acm.org/wp-content/uploads/2017/10/RecSys-17... · 2017. 10. 4. · PowerPoint Presentation Author: Marta Jasicka Created Date:](https://reader035.vdocuments.site/reader035/viewer/2022071112/5fe7cd7d5d23671e840f93c1/html5/thumbnails/6.jpg)
![Page 7: RecSys 2017 Como, Italy - ACM Recommender Systemsrecsys.acm.org/wp-content/uploads/2017/10/RecSys-17... · 2017. 10. 4. · PowerPoint Presentation Author: Marta Jasicka Created Date:](https://reader035.vdocuments.site/reader035/viewer/2022071112/5fe7cd7d5d23671e840f93c1/html5/thumbnails/7.jpg)
Extend network to predict categories
• Start with ResNet
• Add more dense layers
Retrain
• Start with pre-trained weights
• Fine-tune last layers of ResNet
Use new predictions
• Find and fix catalog erros
• Cross learn item attributes
Use new embeddings
Train for another objective
ResNet-50
Dense Layer(s)
Softmax Layer
Long Dress
![Page 8: RecSys 2017 Como, Italy - ACM Recommender Systemsrecsys.acm.org/wp-content/uploads/2017/10/RecSys-17... · 2017. 10. 4. · PowerPoint Presentation Author: Marta Jasicka Created Date:](https://reader035.vdocuments.site/reader035/viewer/2022071112/5fe7cd7d5d23671e840f93c1/html5/thumbnails/8.jpg)
Complementary Products
![Page 9: RecSys 2017 Como, Italy - ACM Recommender Systemsrecsys.acm.org/wp-content/uploads/2017/10/RecSys-17... · 2017. 10. 4. · PowerPoint Presentation Author: Marta Jasicka Created Date:](https://reader035.vdocuments.site/reader035/viewer/2022071112/5fe7cd7d5d23671e840f93c1/html5/thumbnails/9.jpg)
Can we model complex stylistic relationships?
Pairwise complementarity score
• Learn a function y = f ( i , j ) that takes a pair of items, and outputs a score
A more complex problem
![Page 10: RecSys 2017 Como, Italy - ACM Recommender Systemsrecsys.acm.org/wp-content/uploads/2017/10/RecSys-17... · 2017. 10. 4. · PowerPoint Presentation Author: Marta Jasicka Created Date:](https://reader035.vdocuments.site/reader035/viewer/2022071112/5fe7cd7d5d23671e840f93c1/html5/thumbnails/10.jpg)
Embeddings
• Shared between both legs
• Weights are learned
Fusion Layer
• Concatenation
Merge Layer
• Concatenation
• Element-wise max/min/sum/avg
Deep Siamese Neural Network
Image embedding
Attributeembedding
Descriptionembedding
Fu
sio
n L
ay
er
Me
rge
La
ye
r
De
ns
e L
ay
er(
s)
Image embedding
Attributeembedding
Descriptionembedding
Fu
sio
n L
ay
er
![Page 11: RecSys 2017 Como, Italy - ACM Recommender Systemsrecsys.acm.org/wp-content/uploads/2017/10/RecSys-17... · 2017. 10. 4. · PowerPoint Presentation Author: Marta Jasicka Created Date:](https://reader035.vdocuments.site/reader035/viewer/2022071112/5fe7cd7d5d23671e840f93c1/html5/thumbnails/11.jpg)
Positive pairs
• Next-click / same-basket / same-session pairs are noise day
• We use our collection of >100k manually curated outfits
• External datasets
Negative pairs
• Random may work (if you have enough data)
• Manually labeled data is better
Data augmentation to expand
• Find pairs with items similar to observations
• Image translation, rotation, noise will make the network more robust
Training data
![Page 12: RecSys 2017 Como, Italy - ACM Recommender Systemsrecsys.acm.org/wp-content/uploads/2017/10/RecSys-17... · 2017. 10. 4. · PowerPoint Presentation Author: Marta Jasicka Created Date:](https://reader035.vdocuments.site/reader035/viewer/2022071112/5fe7cd7d5d23671e840f93c1/html5/thumbnails/12.jpg)
Good, reliable, labeled data is a competitive advantage. Involve your company in your problem!
Human in the loop
![Page 13: RecSys 2017 Como, Italy - ACM Recommender Systemsrecsys.acm.org/wp-content/uploads/2017/10/RecSys-17... · 2017. 10. 4. · PowerPoint Presentation Author: Marta Jasicka Created Date:](https://reader035.vdocuments.site/reader035/viewer/2022071112/5fe7cd7d5d23671e840f93c1/html5/thumbnails/13.jpg)
GIVENCHY SALVATORE FERRAGAMO
RICK OWENS PIERRE HARDY
FASHION CLINICTIMELESS
LANVIN LANVIN SIMON MILLER
![Page 14: RecSys 2017 Como, Italy - ACM Recommender Systemsrecsys.acm.org/wp-content/uploads/2017/10/RecSys-17... · 2017. 10. 4. · PowerPoint Presentation Author: Marta Jasicka Created Date:](https://reader035.vdocuments.site/reader035/viewer/2022071112/5fe7cd7d5d23671e840f93c1/html5/thumbnails/14.jpg)
Conclusions
![Page 15: RecSys 2017 Como, Italy - ACM Recommender Systemsrecsys.acm.org/wp-content/uploads/2017/10/RecSys-17... · 2017. 10. 4. · PowerPoint Presentation Author: Marta Jasicka Created Date:](https://reader035.vdocuments.site/reader035/viewer/2022071112/5fe7cd7d5d23671e840f93c1/html5/thumbnails/15.jpg)
Outfit generation
• Pairwise function is not sufficient
• find a function f( i, j , k...) that takes a set of products and outputs goodness of outfit
• Extend our siamese network with more legs
Use DL embeddings in current recommendation models
• In content-based and hybrid models
• As side information in MF
• To solve item cold-start problem
Personalized recommendations with end-to-end DL
• Exciting approaches seen at DLRS!
Next Steps
![Page 16: RecSys 2017 Como, Italy - ACM Recommender Systemsrecsys.acm.org/wp-content/uploads/2017/10/RecSys-17... · 2017. 10. 4. · PowerPoint Presentation Author: Marta Jasicka Created Date:](https://reader035.vdocuments.site/reader035/viewer/2022071112/5fe7cd7d5d23671e840f93c1/html5/thumbnails/16.jpg)
Deep learning is not trivial, but it isn't hard to get started
• You can do incremental improvements to many components of your rec-sys
• Start simple, try off the shelf models
• Fine tune to your problem
Get good data
• Involve your company’s experts
• Crowdsource
Deep network engineering is fun!
• Great potential for innovation
Conclusions
![Page 17: RecSys 2017 Como, Italy - ACM Recommender Systemsrecsys.acm.org/wp-content/uploads/2017/10/RecSys-17... · 2017. 10. 4. · PowerPoint Presentation Author: Marta Jasicka Created Date:](https://reader035.vdocuments.site/reader035/viewer/2022071112/5fe7cd7d5d23671e840f93c1/html5/thumbnails/17.jpg)
João Gomes
We’re hiring!
Get in touch for research collaborations
Thank you!
Porto
Lisbon
London