Dress Similarity 03.20.2017 ─ 

Luis Mey Machine Learning Engineer Student 

Overview

This project developed a software capable of helping fashion companies to find similar images in the database, allowing these businesses to cluster its products or to provide product recommendation for its customers. This solution was possible with the current deep learning technology that has shown huge potential for business applications in the

recent years. Using this technology, the artificial intelligence is already trained to receive a set of images (or urls pointing to these images) of different dresses and retrieve what are the most similar dresses for each of the provided dresses.

Domain Background

Developing product recommendation based on image similarity is a hard challenge and recently, with the advance of technology, this problem has been attacked with deep learning, using very recent tools as Tensorflow and Keras (released in 2015) . For example, just recently in 2015 a paper published by researchers at University of North Carolina at Chapel Hill and Illinois at Urbana-Champaign used neural networks to retrieve similar fashion images [1]. In 2016 a start-up, Thread Genius, used a similar technology to retrieve similar fashion images [2].

Problem Statement

Inspired by the recent applications of deep learning in image similarity, the proposal for this project is to develop a simple deep learning software that, given an image of a dress listed by a retailer, similar listed dress images will be returned. Therefore, no supervision to train this model is necessary, since the images itself are used, characterizing the algorithm as an unsupervised learning algorithm. For this project image similarity is be measured by the euclidian distance of the feature vector of the two images being compared. In the context of the Brazilian market that is still young on data science, a simple model would already be competitive with the current product recommendation system presented by big ecommerce retailers.

Datasets and Inputs

A realistic dataset from Brazilian retailers will be used. This dataset consists in about 10,000 scrapped page urls of dresses with product image, name and type, as described by the retailer [3][4]. These images when loaded into memory consume about 17GB and present different shapes, usually with a white or light grey background, highlighting a colored dress that usually a model is wearing. This dataset size was enough to build a simple image similarity software that other retailers could build without requiring expensive processing power.

1

Solution Statement

A convolutional neural network was used to generate a feature vector for each image in the dataset. Each feature vector could be interpreted as a representation of the image in a dimension that is smaller than its pixel representation. In order to save computational effort, this convolutional neural network (CNN) took advantage of pretrained layers in the Inception V3 architecture made available by the deep learning library Keras. The resulting feature vectors were used in the similarity calculations, where the euclidean distance between the feature vectors represented image similarity.

Benchmark Model and Evaluation Metric

Since the similarity between two (or n) dresses has human subjectivity, the proposal for evaluation and benchmarking is to compare a sample of this project’s results with what a large ecommerce retailer suggests as similar products that the customer also would like to see. For example, the first dress below is the searched product, while the following four dresses are the ones recommended by the deep learning model and the last three are similarity recommendations from the ecommerce website used as benchmark. In order to create a numeric metric, it is possible to vote and assign 1 point if the reference image is similar with the recommendation, and zero otherwise. After evaluating 10 randomly sampled images, the performance of the model versus the benchmark could me measured as the number of samples considered similar divided by the total number of sampled images (10).

 

 

Figure 1 - Dress similarity example. The first dress in the left is used as reference, while the remaining are recommended products due to similarity.

2

Project Design

In summary, this project is designed to have 5 steps:

1. Load the images from retailers;

Using the image urls available in the dataset, it is possible to use the skimage library to load all the images and store them in a variable.

2. Preprocess the images to fit into the CNN;

Due to the shape of each image not matching the pre-trained network required input the images need to be reshaped and then stacked as vectors with a shape like (n_images, image_height, image_width, image_channel), where channel is 3 because the images are colored.

3. Build the CNN using a pre-trained network as a base model;

It is possible to use Keras pre-trained Inception V3 model as a base model. This base model receives a (256,256,3) image as input and returns a flat vector. From this flat output vector it is possible to add additional layers, as an additional dense layer with 1024 nodes and a prediction layer with 2 nodes.

4. Generate the feature vectors;

Having some simple information about the dresses, as if it is long or short, it is possible to train the layers below the base model, fine tuning these layers to provide information on the dresses. Then the output from the 1024 nodes base layer generated from each image can be used as a feature vector to represent the image.

5. Retrieve the n-closest images given a reference image.

Finally, a distance metric can be used to calculate how different one feature vector is from the other feature vectors present in the database. This distance metric can be as simple as an euclidean distance that has a numpy implementation. Therefore, being able to calculate these distances allows anyone to just provide an image as input to the model that will calculate or retrieve the feature vector, calculate the minimal distance from other images and finally return n-closest images to the provided input.

3

Data Analysis

During the data exploration it was possible to notice that the dataset required some preprocessing to assure data quality before feeding it to the model. The dataset presented some issues regarding dress labeling. For example, it was possible to notice that some dresses were used by the ecommerce both in the short section and in the long dress section. This type of event could harm the first step of model training that consisted in training the final layers of a CNN ir order to classify the dress as short or long, making the model learn about features of a dress. Due to this ambiguous classification, the problematic examples were removed from the training set.

Additionally, the major source of data, with 11,662 examples, actually had repetitive records that were removed when detected in the data exploration. Moreover, it was possible to notice that image resolution was about 300x400 pixels, higher than the required shape to feed into the pretrained Inception V3 model. This shape mismatch required some image reshaping in the preprocessing phase.

In possession of the preprocessed data, it was possible train the deep learning algorithm that basically consisted in feeding images of the long and short dresses into a pre trained neural network, illustrated by Figure 2, followed by additional dense layers. Each time that a batch of images were feeded into the network, each pixel was transformed through the network up to the last node where a prediction was made. Based on the known labels, short dress or long dress, the network calculated the prediction error and updated the weights of the dense layers. The weights of the dense layers kept multiplying the output of the long convolution neural network, getting closer to a better prediction after several iterations. After processing all the images 30 times (number of epochs), the output of the 1024 nodes dense layer was used as a feature vector that represents each image and, by calculating the euclidian distance between each image it was possible to predict which image was closer to another.

4

Figure 2 - Network used for training: schematic diagram of inception V3[5] that flows into dense layers specifically added for learning about the dresses.

During the exploratory visualization it was also possible to see that most of the dresses only have a generic classification as long or short, as illustrated in the Figure 2 by the first two biggest bars. Even among the most common dress types, the short dresses are 5.7 times more representative in the dataset. The next most common dresses are the printed dresses and the flat dresses that together count about 1,000 examples.

Further illustration is provided by Figure 3 that has one example for each of the five most common dress types.

5

Figure 4 - Unique dress type distribution.

Figure 5 - Top 5 dress types (from left to right: short, long, printed, flat, night).

Methodology Implementation

The initial implementation was executed in the IPython Notebook named Model.ipynb. The notebook’s first section, named “Load and Preprocess Data” is responsible for the steps 1 and 2 previously described in the Project Design section. The process starts

6

from loading the data using the url list available in a csv file, followed by a image reshape to fit into the Inception V3 model and finally a train/test split required to train and test the model. Additionally to the Model.ipynb, the utils.py file was very helpful on organizing the necessary code to load and preprocess the images.

Still on the Model.ipynb, the third step mentioned in the Project Design section was executed (and also the training process mentioned in the step 4). With the help of the model.py it was possible to train the data in an architecture that consists in an input layer that receives 256x256x3 vectors, feeds into a pretrained Inception V3 model, performs an average pooling and feeds to two dense layers, where the first has 1024 nodes and the section only 2 nodes that are activated with a softmax function. This first model only trained the layers below the Inception V3, predicting if a dress was short or long using a rmsprop optimizer and a categorical cross-entropy loss with an accuracy of 90% in the test set.

The fourth step in the Project Design was executed through the Scores Images IPython Notebook. This notebook used the model.py file to create another model that, instead of returning a binary classification, returned the entire dense layer of 1024 nodes, representing the image feature vector. This feature vector model was used to score and save in batch every unique image in the major ecommerce dataset. This process of calculating the feature vectors and saving them in batch is important due to the fact that in the future new products could be added, and with a batch process only this new products would require feature vector calculation, and not the entire dataset again.

The fifth step in the Project Design was executed with the IPython Notebook named Create Similarity Vector. This notebook used the feature vectors to calculate the distance between each image, ranking the 4 closest ones and generating a matrix named similarity matrix. With this similarity matrix, it is possible to know what are the 4 closest images for each of the images in that the retailer has.

Regarding model refinements, it is important to say that the exploratory data analysis was critical to identify the fact that duplicated images were present in the database, what resulted in poor performance of the recommendation system that kept suggesting the same image instead of a similar image. Another refinement was an image augmentation that consisted in feeding not only the images, but also the flipped version of the images. This additional process actually decreased the model accuracy from 93% to 90% for predicting if a dress was long or short, but it is possible to argue that it is important to the model to learn about symmetry because a dress that is just a mirror image should be very similar to its original version.

7

Results

The deep learning model was capable of providing suggestions for similar images given a reference image. In order to evaluate these results, 10 randomly sampled reference images were used. For each of the sampled images, similarity recommendation from the original retailer were extracted using an anonymous browser in order to prevent historical cookies to be used. For the same sampled images, the similarity matrix provided what would be the recommendation from a deep learning model. Organizing the results, it is possible to compare the results as presented below.

Sample 1)

It seems that both recommendations are short dresses, but the deep learning model found similar shades of red, while the ecommerce solution recommended dresses with exposed shoulders. Cumulative points using the proposed metric: 1 (model) vs. 1 (benchmark).

8

Sample 2) Considering this example, it seems that both recommendations does not make much sense. It could be a sign that there are no good similar products in the database. However, it could be argued that recommending other dresses is better than recommending boots and hair equipments. Cumulative points using the proposed metric: 1 (model) vs. 1 (benchmark).

Sample 3)

The deep learning recommendation seems to be similar to the reference image, with very colorful dresses, usually without sleeves and summer like. Having in mind the subjectivity of the comparison, it is possible to argue that the deep learning recommendation seems to be better than the ecommerce proposed solution for this example. Cumulative points using the proposed metric: 2 (model) vs. 1 (benchmark).

9

Sample 4)

The deep learning model recommended what seems to be very similar dresses due to the similar color and size. Considering the ecommerce recommendation, it is possible to notice that probably the recommendation was based on the dress category/style “Sommer” that is common to the reference image. Cumulative points using the proposed metric: 3 (model) vs. 2 (benchmark).

Sample 5)

The deep learning recommendation appears to have a good suggestion here because of the patterns in the dresses. The ecommerce recommendation seems to make sense for the grey dress that has similar color to the reference image. Cumulative points using the proposed metric: 4 (model) vs. 3 (benchmark).

10

Sample 6)

This example presents similar recommendation among the possible solutions due to the fact that both are showing long and formal like dresses. Cumulative points using the proposed metric: 5 (model) vs. 4 (benchmark).

Sample 7)

It seems that the deep learning recommendation provided neutral color options with complex patterns and somehow light look, similarly to the reference image. The ecommerce suggestions seems more aggressive on colors, but it is possible to say that the style is similar. Cumulative points using the proposed metric: 6 (model) vs. 5 (benchmark).

11

Sample 8)

This sample is very interesting because it seems that the deep learning model suggested long dresses, as in the reference image, but also dresses that it is possible to see the feet or even the legs. In contrast, the ecommerce suggestion was short dresses. Cumulative points using the proposed metric: 7 (model) vs. 5 (benchmark).

Sample 9)

This is another example that the ecommerce probably could not provide similar images, but the deep learning model could recommend dresses that usually share a common feature of having stripes. Cumulative points using the proposed metric: 8 (model) vs. 5 (benchmark).

12

Sample 10)

This last sampled example is another probable behavior that the ecommerce suggests similar dresses as similar model (“DF TOP MODA”). This behavior could be incorporated in the deep learning model, making it consider a description similarity as well when recommending dresses. Regarding the deep learning recommendation, it is hard to see similarity in the images, so it is a win for the ecommerce in this example, however the deep learning model finished the test as the winner. Cumulative points using the proposed metric: 8 (model) vs. 6 (benchmark).

In summary, the evaluation resulted in 80% similarity versus 60% similarity according to the proposed similarity metric. This result provides some trust regarding the model performance and, given that little preprocessing was required, there is an indication that the model is robust to compete the with ecommerce benchmark.

Conclusion

This project successfully accomplished the goal of creating similarity model and a process to provide recommendations based on a given image. The final model is competitive and sometimes outperforms the benchmark and also has the advantage of do not requiring a historical user behavior to train the recommendation system. Additional examples from visualizing the final model brings excitement, corroborating with the great quality achieved by the model.

13

14

However, the model still can face difficulties, mainly when there are few similar images in the database. It is extremely interesting that the feature vectors generated in the dense layer after the convolutional neural network were capable of finding similar images using a metric as simple as the euclidean distance. For future work, it would be interesting to use product description as well, capturing similar brands, season or category and extracting even more value from this exciting deep learning technology.

References [1] M. H. Kiapour, X. Han, S. Lazebnik, A. C. Berg and T. L. Berg, “Where to Buy It: Matching Street Clothing Photos in Online Shops” (2015)

[2] Thread Genius, “Robo Bill Cunningham: Shazam for Fashion With Deep Neural Networks” (2016)

[3] https://www.dafiti.com.br/

[4] http://www.lojasrenner.com.br/

[5] Inception V3: https://research.googleblog.com/2016/03/train-your-own-image-classifier-with.html

15