normalized cuts and image segmentation j. shi and j. malik · shape contexts 9 shape matching and...
TRANSCRIPT
![Page 1: Normalized Cuts and Image Segmentation J. Shi and J. Malik · Shape Contexts 9 Shape Matching and Object Recognition Using Shape Contexts, S. Belongie, J. Malik, and J. Puzicha, 2002](https://reader035.vdocuments.site/reader035/viewer/2022071017/5fd084c17a2e0f18f85b1a4a/html5/thumbnails/1.jpg)
Shape Representation
Soma Biswas
Department of Electrical Engineering,
Indian Institute of Science, Bangalore.
![Page 2: Normalized Cuts and Image Segmentation J. Shi and J. Malik · Shape Contexts 9 Shape Matching and Object Recognition Using Shape Contexts, S. Belongie, J. Malik, and J. Puzicha, 2002](https://reader035.vdocuments.site/reader035/viewer/2022071017/5fd084c17a2e0f18f85b1a4a/html5/thumbnails/2.jpg)
Shape-Based Recognition
2
Analysis of anatomical structures Figure from Grimson & Golland
Pose
Recognition, detection Fig from Opelt et al.
![Page 3: Normalized Cuts and Image Segmentation J. Shi and J. Malik · Shape Contexts 9 Shape Matching and Object Recognition Using Shape Contexts, S. Belongie, J. Malik, and J. Puzicha, 2002](https://reader035.vdocuments.site/reader035/viewer/2022071017/5fd084c17a2e0f18f85b1a4a/html5/thumbnails/3.jpg)
Applications
3
![Page 4: Normalized Cuts and Image Segmentation J. Shi and J. Malik · Shape Contexts 9 Shape Matching and Object Recognition Using Shape Contexts, S. Belongie, J. Malik, and J. Puzicha, 2002](https://reader035.vdocuments.site/reader035/viewer/2022071017/5fd084c17a2e0f18f85b1a4a/html5/thumbnails/4.jpg)
Geometric Transformations
4
![Page 5: Normalized Cuts and Image Segmentation J. Shi and J. Malik · Shape Contexts 9 Shape Matching and Object Recognition Using Shape Contexts, S. Belongie, J. Malik, and J. Puzicha, 2002](https://reader035.vdocuments.site/reader035/viewer/2022071017/5fd084c17a2e0f18f85b1a4a/html5/thumbnails/5.jpg)
Related Problems
Shape representation and decomposition
Finding a set of correspondences between shapes
Transforming one shape into another
Measuring the similarity between shapes
Shape localization and model alignment
Finding a shape similar to a model in a cluttered image
5
![Page 6: Normalized Cuts and Image Segmentation J. Shi and J. Malik · Shape Contexts 9 Shape Matching and Object Recognition Using Shape Contexts, S. Belongie, J. Malik, and J. Puzicha, 2002](https://reader035.vdocuments.site/reader035/viewer/2022071017/5fd084c17a2e0f18f85b1a4a/html5/thumbnails/6.jpg)
Comparing Images Using the Hausdorff Distance
6 D. Huttenlocher, G. Klanderman, and W. Rucklidge, 1993
![Page 7: Normalized Cuts and Image Segmentation J. Shi and J. Malik · Shape Contexts 9 Shape Matching and Object Recognition Using Shape Contexts, S. Belongie, J. Malik, and J. Puzicha, 2002](https://reader035.vdocuments.site/reader035/viewer/2022071017/5fd084c17a2e0f18f85b1a4a/html5/thumbnails/7.jpg)
7
![Page 8: Normalized Cuts and Image Segmentation J. Shi and J. Malik · Shape Contexts 9 Shape Matching and Object Recognition Using Shape Contexts, S. Belongie, J. Malik, and J. Puzicha, 2002](https://reader035.vdocuments.site/reader035/viewer/2022071017/5fd084c17a2e0f18f85b1a4a/html5/thumbnails/8.jpg)
8
![Page 9: Normalized Cuts and Image Segmentation J. Shi and J. Malik · Shape Contexts 9 Shape Matching and Object Recognition Using Shape Contexts, S. Belongie, J. Malik, and J. Puzicha, 2002](https://reader035.vdocuments.site/reader035/viewer/2022071017/5fd084c17a2e0f18f85b1a4a/html5/thumbnails/9.jpg)
Shape Contexts
9
Shape Matching and Object Recognition Using Shape Contexts, S. Belongie, J. Malik, and J.
Puzicha, 2002
Approach for measuring similarity between shapes and apply it for object recognition
Solve for correspondences between points on the two shapes
● Using shape contexts – describe coarse distribution of the rest of the shape
w.r.t. a given point on the shape
Use the correspondences to estimate an aligning transform
● Using regularized thin-plate splines
Compute the distance between the two shapes
![Page 10: Normalized Cuts and Image Segmentation J. Shi and J. Malik · Shape Contexts 9 Shape Matching and Object Recognition Using Shape Contexts, S. Belongie, J. Malik, and J. Puzicha, 2002](https://reader035.vdocuments.site/reader035/viewer/2022071017/5fd084c17a2e0f18f85b1a4a/html5/thumbnails/10.jpg)
10
- Not required to be landmarks/curvature
extrema, etc
- More samples -> better approximation of
underlying shape
![Page 11: Normalized Cuts and Image Segmentation J. Shi and J. Malik · Shape Contexts 9 Shape Matching and Object Recognition Using Shape Contexts, S. Belongie, J. Malik, and J. Puzicha, 2002](https://reader035.vdocuments.site/reader035/viewer/2022071017/5fd084c17a2e0f18f85b1a4a/html5/thumbnails/11.jpg)
11
SC: extremely rich descriptors Finding correspondence between 2 shapes = for each
sample pt on one shape, find sample pt on the other
shape with most similar SC
Maximizing similarities and enforcing uniqueness ->
bipartite graph matching problem / optimal assignment
Can add local appearance similarity at
the 2 points (gray scale images)
Choice is application dependent
For robust handling of outliers, add
dummy nodes to each pt set
When there is no real match, a pt will be
matched to the dummy
![Page 12: Normalized Cuts and Image Segmentation J. Shi and J. Malik · Shape Contexts 9 Shape Matching and Object Recognition Using Shape Contexts, S. Belongie, J. Malik, and J. Puzicha, 2002](https://reader035.vdocuments.site/reader035/viewer/2022071017/5fd084c17a2e0f18f85b1a4a/html5/thumbnails/12.jpg)
Invariance and Robustness
Matching approach should be
1) invariant under scaling and translation
2) robust under small geometrical distortions,
occlusion & presence of outliers
Invariant to translation -> since all measurements are
taken w.r.t. pts on the object
Scale invariance: Normalize all radial distances by
mean distance between the pt pairs in the shape
From expts: insensitive to small perturbations of parts
of the shape, small non-linear transformations,
occlusions and outliers
Can provide complete rotation invariance: use relative
frame – tangent vector at each point as the x-axis
(not suitable for say 6 and 9)
12
-a,b- sampled points
- correspondence found using
bipartite matching
![Page 13: Normalized Cuts and Image Segmentation J. Shi and J. Malik · Shape Contexts 9 Shape Matching and Object Recognition Using Shape Contexts, S. Belongie, J. Malik, and J. Puzicha, 2002](https://reader035.vdocuments.site/reader035/viewer/2022071017/5fd084c17a2e0f18f85b1a4a/html5/thumbnails/13.jpg)
Thin-Plate Spline Model
13
![Page 14: Normalized Cuts and Image Segmentation J. Shi and J. Malik · Shape Contexts 9 Shape Matching and Object Recognition Using Shape Contexts, S. Belongie, J. Malik, and J. Puzicha, 2002](https://reader035.vdocuments.site/reader035/viewer/2022071017/5fd084c17a2e0f18f85b1a4a/html5/thumbnails/14.jpg)
Minimizing Bend Energy
14
![Page 15: Normalized Cuts and Image Segmentation J. Shi and J. Malik · Shape Contexts 9 Shape Matching and Object Recognition Using Shape Contexts, S. Belongie, J. Malik, and J. Puzicha, 2002](https://reader035.vdocuments.site/reader035/viewer/2022071017/5fd084c17a2e0f18f85b1a4a/html5/thumbnails/15.jpg)
Matching Process
15
![Page 16: Normalized Cuts and Image Segmentation J. Shi and J. Malik · Shape Contexts 9 Shape Matching and Object Recognition Using Shape Contexts, S. Belongie, J. Malik, and J. Puzicha, 2002](https://reader035.vdocuments.site/reader035/viewer/2022071017/5fd084c17a2e0f18f85b1a4a/html5/thumbnails/16.jpg)
Object Recognition
Prototype based recognition: categories represented by ideal examples, rather than
logical rules
Eg. Prototype for bird category: sparrow
Soft category membership – as one moves further away from the ideal example, the
association with that prototype falls off
3 distances:
Shape Context Distance between shapes P and Q: sum of SC matching costs over
best matching points
Local image appearance difference: texture and color
Amount of transformation necessary to align the shapes: Bending energy in TPS
16
![Page 17: Normalized Cuts and Image Segmentation J. Shi and J. Malik · Shape Contexts 9 Shape Matching and Object Recognition Using Shape Contexts, S. Belongie, J. Malik, and J. Puzicha, 2002](https://reader035.vdocuments.site/reader035/viewer/2022071017/5fd084c17a2e0f18f85b1a4a/html5/thumbnails/17.jpg)
Results – Digit Recognition
17
![Page 18: Normalized Cuts and Image Segmentation J. Shi and J. Malik · Shape Contexts 9 Shape Matching and Object Recognition Using Shape Contexts, S. Belongie, J. Malik, and J. Puzicha, 2002](https://reader035.vdocuments.site/reader035/viewer/2022071017/5fd084c17a2e0f18f85b1a4a/html5/thumbnails/18.jpg)
Result: Trademark Retrieval
18
-300 trademarks
- 300 sample points
-Computational Needs:
-For 100 sample points - ~200ms
![Page 19: Normalized Cuts and Image Segmentation J. Shi and J. Malik · Shape Contexts 9 Shape Matching and Object Recognition Using Shape Contexts, S. Belongie, J. Malik, and J. Puzicha, 2002](https://reader035.vdocuments.site/reader035/viewer/2022071017/5fd084c17a2e0f18f85b1a4a/html5/thumbnails/19.jpg)
Inner Distance
19
Using the Inner-Distance for Classification of Articulated Shapes, H. Ling and
D. Jacobs, 2005
![Page 20: Normalized Cuts and Image Segmentation J. Shi and J. Malik · Shape Contexts 9 Shape Matching and Object Recognition Using Shape Contexts, S. Belongie, J. Malik, and J. Puzicha, 2002](https://reader035.vdocuments.site/reader035/viewer/2022071017/5fd084c17a2e0f18f85b1a4a/html5/thumbnails/20.jpg)
Model of Articulated Objects
20
![Page 21: Normalized Cuts and Image Segmentation J. Shi and J. Malik · Shape Contexts 9 Shape Matching and Object Recognition Using Shape Contexts, S. Belongie, J. Malik, and J. Puzicha, 2002](https://reader035.vdocuments.site/reader035/viewer/2022071017/5fd084c17a2e0f18f85b1a4a/html5/thumbnails/21.jpg)
Inner Distance
21
![Page 22: Normalized Cuts and Image Segmentation J. Shi and J. Malik · Shape Contexts 9 Shape Matching and Object Recognition Using Shape Contexts, S. Belongie, J. Malik, and J. Puzicha, 2002](https://reader035.vdocuments.site/reader035/viewer/2022071017/5fd084c17a2e0f18f85b1a4a/html5/thumbnails/22.jpg)
Computing the Inner Distance
22
![Page 23: Normalized Cuts and Image Segmentation J. Shi and J. Malik · Shape Contexts 9 Shape Matching and Object Recognition Using Shape Contexts, S. Belongie, J. Malik, and J. Puzicha, 2002](https://reader035.vdocuments.site/reader035/viewer/2022071017/5fd084c17a2e0f18f85b1a4a/html5/thumbnails/23.jpg)
Example
23
![Page 24: Normalized Cuts and Image Segmentation J. Shi and J. Malik · Shape Contexts 9 Shape Matching and Object Recognition Using Shape Contexts, S. Belongie, J. Malik, and J. Puzicha, 2002](https://reader035.vdocuments.site/reader035/viewer/2022071017/5fd084c17a2e0f18f85b1a4a/html5/thumbnails/24.jpg)
Experiments
24
![Page 25: Normalized Cuts and Image Segmentation J. Shi and J. Malik · Shape Contexts 9 Shape Matching and Object Recognition Using Shape Contexts, S. Belongie, J. Malik, and J. Puzicha, 2002](https://reader035.vdocuments.site/reader035/viewer/2022071017/5fd084c17a2e0f18f85b1a4a/html5/thumbnails/25.jpg)
Hierarchical Matching of Deformable Shapes
25 P. Felzenszwalb and J. Schwartz, 2007
Use:
- Compare pair of objects
- Detect objects in cluttered images
![Page 26: Normalized Cuts and Image Segmentation J. Shi and J. Malik · Shape Contexts 9 Shape Matching and Object Recognition Using Shape Contexts, S. Belongie, J. Malik, and J. Puzicha, 2002](https://reader035.vdocuments.site/reader035/viewer/2022071017/5fd084c17a2e0f18f85b1a4a/html5/thumbnails/26.jpg)
The Shape Tree
A be an open curve (a1, . . . , an).
ai be a midpoint on A.
L(ai|a1, an) -> location of ai relative to a1 and an
First & last sample points define a canonical scale
and orientation, so L invariant to similarity transf.
26
o Left child of a node: describes the subcurve from the start to the midpoint
o Right child describes the subcurve from the midpoint to the end.
o Bottom nodes capture local geometric properties such as the angle formed at a point,
o Root nodes capture more global information encoded by the relative locations of points
that are far from each other.
o Representation invariant to similarity transformations : Since contains only the
locations of points relative to two other points.
o Given the tree representation for A, along with the location of its start and end points a1 and an, the
curve can be recursively reconstructed – translated, rotated & scaled version of A
![Page 27: Normalized Cuts and Image Segmentation J. Shi and J. Malik · Shape Contexts 9 Shape Matching and Object Recognition Using Shape Contexts, S. Belongie, J. Malik, and J. Puzicha, 2002](https://reader035.vdocuments.site/reader035/viewer/2022071017/5fd084c17a2e0f18f85b1a4a/html5/thumbnails/27.jpg)
27
Bookstein coordinate of B
w.r.t. A and C
![Page 28: Normalized Cuts and Image Segmentation J. Shi and J. Malik · Shape Contexts 9 Shape Matching and Object Recognition Using Shape Contexts, S. Belongie, J. Malik, and J. Puzicha, 2002](https://reader035.vdocuments.site/reader035/viewer/2022071017/5fd084c17a2e0f18f85b1a4a/html5/thumbnails/28.jpg)
Deformation Model
28
![Page 29: Normalized Cuts and Image Segmentation J. Shi and J. Malik · Shape Contexts 9 Shape Matching and Object Recognition Using Shape Contexts, S. Belongie, J. Malik, and J. Puzicha, 2002](https://reader035.vdocuments.site/reader035/viewer/2022071017/5fd084c17a2e0f18f85b1a4a/html5/thumbnails/29.jpg)
Elastic Matching
A and B be 2 open curves
Build shape tree for A -> look for mapping from points in A to points in B such that
the shape tree of A is deformed as little as possible
Total deformation = sum over deformations applied to each node in the A shape-tree
Hierarchical nature of the shape-tree ensures that both local and global geometric
properties are preserved by a good matching.
Allow larger deformations near the bottom of a shape-tree as these do not change the
global appearance of an object.
29
![Page 30: Normalized Cuts and Image Segmentation J. Shi and J. Malik · Shape Contexts 9 Shape Matching and Object Recognition Using Shape Contexts, S. Belongie, J. Malik, and J. Puzicha, 2002](https://reader035.vdocuments.site/reader035/viewer/2022071017/5fd084c17a2e0f18f85b1a4a/html5/thumbnails/30.jpg)
Experiments
30
![Page 31: Normalized Cuts and Image Segmentation J. Shi and J. Malik · Shape Contexts 9 Shape Matching and Object Recognition Using Shape Contexts, S. Belongie, J. Malik, and J. Puzicha, 2002](https://reader035.vdocuments.site/reader035/viewer/2022071017/5fd084c17a2e0f18f85b1a4a/html5/thumbnails/31.jpg)
31
![Page 32: Normalized Cuts and Image Segmentation J. Shi and J. Malik · Shape Contexts 9 Shape Matching and Object Recognition Using Shape Contexts, S. Belongie, J. Malik, and J. Puzicha, 2002](https://reader035.vdocuments.site/reader035/viewer/2022071017/5fd084c17a2e0f18f85b1a4a/html5/thumbnails/32.jpg)
Results
32
![Page 33: Normalized Cuts and Image Segmentation J. Shi and J. Malik · Shape Contexts 9 Shape Matching and Object Recognition Using Shape Contexts, S. Belongie, J. Malik, and J. Puzicha, 2002](https://reader035.vdocuments.site/reader035/viewer/2022071017/5fd084c17a2e0f18f85b1a4a/html5/thumbnails/33.jpg)
What is a CAPTCHA?
33 Recognizing Objects in Adversarial Clutter: Breaking a Visual CAPTCHA,
G. Mori and J. Malik , 2003