1

Object Recognition by Discriminative Combinations of Line Segments,

Ellipses and Appearance Features

Professor: S. J. WangStudent : Y. S. Wang

2

OutlineBackgroundSystem OverviewShape-TokenCode-Book of Shape-TokenCode-Word CombinationHybrid DetectorExperimental ResultConclusion

3

BackgroundContour Based Detection Method

Problem of Contour Fragment:◦Storage requirement is large for training.◦Slow matching speed.◦Not scale invariant.

Solution provided is Shape-Token.

4

System Overview

5

Shape TokenWhat is Shape-Tokens?Constructing Shape-TokensDescribing Shape-TokensMatching Shape-Tokens

6

What is Shape-Tokens?Use the combination of line and ellipse

to represent the contour fragments.◦Line for line.◦Ellipse for curve.

Example:

Why shape-tokens?◦Several parameters are enough for us to

describe the contour fragment.

7

Constructing Shape-TokensExtract Shape Primitives of line

segments and ellipses by [16] [17]. Pairing reference primitive to its

neighboring primitive. ◦Different type combination:

Take ellipse as reference.◦Same type combination:

Consider each as reference in turn.Three types of Shape-Tokens:

◦Line-Line, Ellipse-Line, Ellipse-Ellipse.

8

Constructing Shape-TokensLine-Line

◦Combine neighboring line which has any point falling in trapezium searching area.

Ellipse-Line & Ellipse-Ellipse◦Circular Search Area. Consider

primitives has any point within searching area and weakly is connected to reference ellipse.

9

Describing Shape-Tokensd

◦ : Orientation of a Primitive.◦ : Unit vector from center of reference

primitive to center of its neighbor.

◦ : Distance between centers of primitives.◦ : Length and Width for each primitives.

10

Matching Shape-TokensDissimilarity Measure (Shape

Distance)

11

Matching Shape-TokensMore general for multiple scale

matching◦Normalize descriptor against object

scale

12

Codebook of Shape-TokensExtracting Shape-Tokens inside

bounding boxes of training images.

Producing Code-words◦Clustering by Shape◦Clustering by Relative Positions

Selecting representative code-words into codebook for specific target object.

13

K-Medoid MethodSimilar to the k-means method.Procedure:

◦ Randomly select k of the n data points as medoids.

◦ Associate each data point to the closest medoid.◦ For each medoid m

For each non-medoid data point o Swap m and o and compute the total cost of the configuration.

◦ Select the configuration with the lowest cost.◦ Repeat the steps above until there is no change

in the medoid.

14

K-Medoid MethodFirst two steps

15

K-Medoid MethodThird to Fourth step

16

Clustering by ShapeMethod:

◦Use k-medoid method to cluster the shape-tokens for each type separately.

◦Repeat the step above until the dissimilarity value for each cluster is lower then a specific threshold. Metric: Dissimilarity Value: average shape

distance between the medoid and its members.

Threshold: 20% of the maximum of D(.).

17

Clustering by relative positionsTarget:

◦Partition the clusters obtained from previous step by to attain sub-clusters whose members have similar shape and position relative to the centroid of object.

◦ : vector direct from object centroid to the

shape-token centroid.Method: Mean-Shift.

18

Candidate Code-Word Medoid for each sub-cluster.Parameters:

Shape Distance Threshold :Mean shape distance of the cluster plus one standard deviation.

Relative Position Center :Mean of vectors of the sub-clusters members.

Radius :Euclidean distance between to of each sub-cluster member plus one standard deviation.

19

Candidate Code-WordsExample: the Weizmann horse

dataset.

20

Selecting Candidates into CodebookIntuition: Size of cluster.Problem: Lots of selected

candidates belong to background clutter.

What kind of candidates we prefer ?◦Distinctive Shape.◦Flexible enough to accommodate

intra-class variations.◦Precise Location for its members.

21

Selecting Candidates into CodebookInstead of using cluster size directly,

the author scores each candidate by a product “” consists of three values.◦Intra-cluster shape similarity value

“” where is the maximum of the range of shape distance for the type of candidate currently considered.

◦The number of unique training bounding boxes its members are extracted from.

◦Its value of .

22

Selecting Candidates into CodebookOne more problem left:

◦If use to choose the candidate directly, it may cause not ideal spatial distribution.

Solution: Radial Ranking Method

23

Selecting Candidates into CodebookExample: the Weizmann horse

dataset.

24

Code-Word CombinationWhy code-word combination ?

◦One can use a single code-word that is matched in test image to predict object location. => Less discriminative and easy to matched in background.

◦Instead, a combination of several code-words can be more discriminative.

25

Code-Word CombinationMatching a code-word

combination◦Way to match code-word

combination.Finding all matched code-word

combinations in training images◦Exhaustive set of code-word

combinations.Learning discriminative xCC (x-

codeword combination)

26

Matching a Code-Word Combination Criteria:

◦Shape Constraint :Shape distance between each code-word and shape-token in image should be less then shape distance threshold .

◦Geometric Constraint:Centroid prediction by all code-words in the combination concur.

27

Matching a Code-Word Combination Example:

28

Finding all matched code-word combinations in training imagesGoal:

Finding an exhaustive set of possible candidates of code-word combinations.

Method: (Similar to Sliding-Window Search)◦For each candidate window at scale

and location in image I, we try to find there is any match for each code-word or not. And the combination of each matched code-word will be a possible combination candidate.

29

Finding all matched code-word combinations in training imagesSpecify a variable to represent

the matching condition of a specific code-word .

30

Finding all matched code-word combinations in training imagesIf ,then we say that the code-

word is matched at scale and location .

Any combination of these matched code-word will produce a candidate combination.

Why not consider the geometric constraint?

31

Finding all matched code-word combinations in training images

32

Learning Discriminative xCCWe’d like to obtain a xCC which

satisfies the following three constraint.◦Shape Constraint :

Highly related Code-Book Establishment

◦Geometric Constraint: Object Location Agreement.

◦Structural Constraint :Reasonable code-word combination for different poses of object.

33

Learning Discriminative xCCExample:

34

Learning Discriminative xCCBinary Tree to represent a xCC.

◦Each node is a decision statement:

35

Learning Discriminative xCCAdaBoost Training Procedure to

produce one xCC from each iteration.

The Binary Tree depth “k” can be obtained by 3-fold cross validation.

36

Learning Discriminative xCCExample:

37

Learning Discriminative xCCExample:

38

Learning Discriminative xCCExample:

39

Hybrid Detector xMCCIncorporating SIFT as appearance

information to enhance the performance.

Procedure: (same as previous section)

40

Hybrid Detector xMCCExample:

41

Hybrid Detector xMCCExample:

42

Hybrid Detector xMCCExample:

43

Experimental ResultContour only result under

viewpoint change. (train on side-view only)

44

Experimental ResultContour only result for

discriminating similar shape object classes.

45

Experimental ResultCompare with Shotton [6] on

Weizmann Horse test set.

Shotton [6]: Use contour fragment, fixed number of code-words for each combination.

46

Experimental ResultWeizmann Horse Test Set.

47

Experimental ResultGraz-17 classes.

48

Experimental ResultGraz-17 dataset.

49

Experimental ResultHybrid-Method result

50

ConclusionThis article provide a contour based

method that exploits very simple and generic shape primitives of line segments and ellipses for image classification and object detection.

Novelty:◦Shape-Token to reduce the time cost for

matching and the need of memory storage.◦No restriction on the number of shape-tokens

for combinations.◦Allow combination of different feature types.