Bag-of-Words based Image Classification

Joost van de Weijer

What is in the image ?

image classification: answers the question what is in the image.

Is there a person ?

Is there a suit-case ?

Is there car ?

Inspiration

The VOC Pascal challenge: a competition on image classification.Participants have to classify 20 classes in over 10.000 images.

Inspiration

http://pascallin.ecs.soton.ac.uk/challenges/VOC/voc2010/results/index.html

The Event Data Set

• 7 event classes: basketball, polo, rowing, castells, marathon, sailing, skiing.• each class has 50 images, devided in 30 training and 20 test images.

Project I

goal: build an image classification system which can successfully classify sport images.

title: Bag-of-Words based Image Classification.

competition: do so better than the other groups.

Why is this difficult ?

Variations in viewpoint and zoom.

Variations in pose.

Why is this difficult ?

Inter-class variation.

lighting changes.

Why is this difficult ?

Back-ground variation.

Maybe the background could help ?

similar backgrounds- different classes.

from images to frequency histogram

•Compute visual words:• detect local regions from a set of images.• describe every local region by a descriptor

• texture• color

• cluster all descriptors into visual words

Given a new image:• detect local regions from a set of image.

assign to visual word N

• assign every region to its nearest visual word.• compute visual word-image histogram

Bag of Visual Words representation

Feature Detection

Bag-of-Words

representation

normalize patches

No spatial relations.

pi(w|Miro)

pi(w|Dali)

Bag of Visual Words representation

Image Representation

2. Extractionshapetexturecolor

Image

1. Feature detection

shape words

3.Le

arn

voca

bula

ry

Shape Voc

image classificationimage retrieval

4. BOW

5. SVM/ distance measures

The Framework

Image Representation

Image

1. Feature detection

1.random

2. Extractionshapetexturecolor

2. RGB

shape words

3.Le

arn

voca

bula

ry

Shape Voc

3. random

4. BOW

4. nearest neighbor

image classificationimage retrieval

5. SVM/ distance measures

5. linear SVM

Existing Implementation:

50 % classification score

Existing Implementation:

properties of BOW implementation:

• you can improve any of the subroutines and analyze the changes based on the classification results.

• several team members can work on feature detection while others work on feature description.

• the final classification results allow us to compare the results between the groups.

Project I: Bag Bag-of-Words based Image Classification

goal : build an image classification system which can successfully classify sport images.

teaching objectives

you will learn:• to represent images robust to changes of cameras, object orientation, and illuminant color.• what photometric invariance theory is and how to apply it to a real-world problem.• understand and use the SIFT descriptor.• how to discretize image features (colors, shapes, and textures).• what the strong and weak points of BOW representations for images are. • how to evaluate retrieval and classification results.

Practical information:

Group Size:The project has to be made in groups of 3 students. Each group should decide on the following roles:

• responsible competition.• responsible presentation.• responsible report

Practical Information:All practical information can be found in the student guide (http://cat.cvc.uab.es/~joost/master.html )

If it is hard to work as a group you can partition the tasks:

• feature detection• feature description• vocabulary construction• learning/evaluation

All group-members should understand all steps in the final program !

Practical information:

Important Dates:22 jan - 19 Feb. : The project will last 1 month. 22 jan. : Start project.29 Jan. : Extra assignment will be handed out. Submission of first results in AP.5 Feb. : Discussion meeting + submission second results in AP.11 Feb. : Publication of final test set.12 Feb. : Discussion meeting with groups separately.15 Feb. : Final submission of classification results in AP for all classes.19 Feb. : Presentation of the project.22 Feb. : Final submission date for report.

Supervision:There will be project meetings on Tuesdays afternoon to discus progress.

For any questions during the three weeks of the project email (joost@cvc.uab.es) or come to office O/119 in the CVC.

Use “PROJECT I” as subject of your emails, which makes it easier to manage.

Practical information:

NotesThe final note will be based on:

• participation (15%)• presentation (25%)• report (50%)• competition (10%)

 

Bugs:For sure there will be several bugs in the code. If you find one, mail me, and I will notify the other groups. Thanks !

Practical information:

Competition:

Dates:29 Jan. : Submission of first results in AP (before 15:00).5 Feb. : Submission second results in AP (before 15:00)

labeled train set labeled test set

19/22 Feb. : Your report/final presentation is based on the labeled test set !

Practical information:

labeled train set no labels for test set !

Competition:

Dates:

11 Feb. : Publication of final test set.15 Feb. : Final submission results in AP for all calsses.

Practical information:

Final Report The final report has to be submitted on 22th of February. The report should contain the following chapters.  • Introduction ( max 1 page )• Feature Detection (max 2 pages).• Feature Description (max 3 pages).• Visual Vocabulary and BOW representation (max 2 pages)• Classification (max 2 pages)• Object Detection (optional: max 2 pages)• Results (max 2 pages).• Conclusions (max 1 page)

What to do next ?

• make groups of and assign :•responsible competition

(send an email to me today or tomorrow )

• install the programs and play with the code. ( http://cat.cvc.uab.es/~joost/master.html )

• This week you should already start working on a feature detector.

What to do next ?

Good Luck !