Applying Transfer Learning in TensorFlow

Scott ThompsonSoftware Engineer

@KontorHQ

Lessons learned building a visual search and classifier for Gilt

A guide for Software Engineers

Classification

Men - Short Sleeve Sport Shirts

Men - Long Sleeve Sport Shirts

Men - Activewear - Activewear Tops

Men - Polos

Visual Search

How to distill the ‘essence’ of an image?

Classification

Nearest-Neighbor Search

Men - Short Sleeve Sport Shirts

Men - Long Sleeve Sport Shirts

Image Features

Source: http://opencv.org

Edge Detection Color Histogram Pixel Segmentation

How to train?

Google Inception-v3

Pre-trained on ImageNethttp://download.tensorflow.org/models/image/imagenet/inception-2015-12-05.tgz

Convolutional Neural Nets

Source: https://research.googleblog.com/2016/03/train-your-own-image-classifier-with.html

1,000,000 images, 1,000 categories

Business office Internet site President

Image Source: http://image-net.org/explore

Why use a pre-trained model?

It’s faster (it’s pre-trained)

It’s cheaper (no need for GPU farm)

It generalizes (avoid overfitting)

Image Source: http://image-net.org/explore

Transfer Learning

Source: https://research.googleblog.com/2016/03/train-your-own-image-classifier-with.html

‘Essence’Edge Detector

Shape Detector

Structure of the Network

t-SNE plot - CIFAR-10 pool_3 features

t-SNE Code: https://lvdmaaten.github.io/tsne/

Output Layer Input LayerSource: https://github.com/sthomp/tensorflow_transfer_cifar10

Performance

About 1 second per image

Gilt has ~40,000 images

~12 hours

Embarrassingly Parallel -> Partition across threads / machines

Cache pool_3 features (numpy array)

Applying pool_3 featuresNearest Neighbor Search

Brute Force

Cosine Distance

Checkout https://github.com/spotify/annoy

Train a classifier

Nearest Neighbor Classifier

Softmax

You don’t need to use TensorFlow for everything

Reading files / Work Queues

Image resizing

Serializing weights