extracting subimages of an unknown category … · extracting subimages of an unknown category from...
TRANSCRIPT
![Page 1: Extracting Subimages of an Unknown Category … · Extracting Subimages of an Unknown Category from a Set of Images Sinisa Todorovic and Narendra Ahuja ... – Remove a node – Add](https://reader031.vdocuments.site/reader031/viewer/2022030920/5b7811e47f8b9a515a8e6ff5/html5/thumbnails/1.jpg)
1
Extracting Subimages of an Unknown Category from a Set of Images
Sinisa Todorovic and Narendra AhujaBeckman Institute, UIUC
Presented by Tingfan Wu
![Page 2: Extracting Subimages of an Unknown Category … · Extracting Subimages of an Unknown Category from a Set of Images Sinisa Todorovic and Narendra Ahuja ... – Remove a node – Add](https://reader031.vdocuments.site/reader031/viewer/2022030920/5b7811e47f8b9a515a8e6ff5/html5/thumbnails/2.jpg)
Objective
![Page 3: Extracting Subimages of an Unknown Category … · Extracting Subimages of an Unknown Category from a Set of Images Sinisa Todorovic and Narendra Ahuja ... – Remove a node – Add](https://reader031.vdocuments.site/reader031/viewer/2022030920/5b7811e47f8b9a515a8e6ff5/html5/thumbnails/3.jpg)
Clusters
General Steps
Training Images
Random segmentsfeature vectors
Unseen image
C3
C2C1
ModelsFt1=(x1,x2….xn)Ft2=(x1,x2….xn)Ft3=(x1,x2….xn)Ft4=(x1,x2….xn)feature vectors
F1=(x1,x2….xn)F2=(x1,x2….xn)F3=(x1,x2….xn)F4=(x1,x2….xn)
• Varieties– Segmetation Methods– Feature Spaces– Clustering Methods
= C1
![Page 4: Extracting Subimages of an Unknown Category … · Extracting Subimages of an Unknown Category from a Set of Images Sinisa Todorovic and Narendra Ahuja ... – Remove a node – Add](https://reader031.vdocuments.site/reader031/viewer/2022030920/5b7811e47f8b9a515a8e6ff5/html5/thumbnails/4.jpg)
Segment out all the cars
….
fused tree model for cars
Unseen image
Training images
Segmented Cars
Segmentation Trees
OverviewOverview
Multiscale Seg.
![Page 5: Extracting Subimages of an Unknown Category … · Extracting Subimages of an Unknown Category from a Set of Images Sinisa Todorovic and Narendra Ahuja ... – Remove a node – Add](https://reader031.vdocuments.site/reader031/viewer/2022030920/5b7811e47f8b9a515a8e6ff5/html5/thumbnails/5.jpg)
Segment out all the cars
….
fused tree model for cars
Unseen image
Training images
Segmented Cars
Segmentation Trees
MultiscaleMultiscaleSegmentation TreeSegmentation Tree
![Page 6: Extracting Subimages of an Unknown Category … · Extracting Subimages of an Unknown Category from a Set of Images Sinisa Todorovic and Narendra Ahuja ... – Remove a node – Add](https://reader031.vdocuments.site/reader031/viewer/2022030920/5b7811e47f8b9a515a8e6ff5/html5/thumbnails/6.jpg)
Feature Extraction = Image Segmentation
![Page 7: Extracting Subimages of an Unknown Category … · Extracting Subimages of an Unknown Category from a Set of Images Sinisa Todorovic and Narendra Ahuja ... – Remove a node – Add](https://reader031.vdocuments.site/reader031/viewer/2022030920/5b7811e47f8b9a515a8e6ff5/html5/thumbnails/7.jpg)
Multiscale Segmentation Tree
![Page 8: Extracting Subimages of an Unknown Category … · Extracting Subimages of an Unknown Category from a Set of Images Sinisa Todorovic and Narendra Ahuja ... – Remove a node – Add](https://reader031.vdocuments.site/reader031/viewer/2022030920/5b7811e47f8b9a515a8e6ff5/html5/thumbnails/8.jpg)
Region Descriptor on Tree Node
Attr(Node) = Description of the region
![Page 9: Extracting Subimages of an Unknown Category … · Extracting Subimages of an Unknown Category from a Set of Images Sinisa Todorovic and Narendra Ahuja ... – Remove a node – Add](https://reader031.vdocuments.site/reader031/viewer/2022030920/5b7811e47f8b9a515a8e6ff5/html5/thumbnails/9.jpg)
What are good region descriptors?
• Photometric– Gray level
• Geometric (rotation invariant)– Area– C.M.– Boundary Shape Histogram
• Hybrid – Salient descriptor
• Topology– Recursive containment of regions
(¹ v ; ¾2v )
hv (1 : : : K )
(av )(xv ; yv )
(©v )hv (1)
hv (2)
hv (8)
Can be rotation invariant
hv (3)
![Page 10: Extracting Subimages of an Unknown Category … · Extracting Subimages of an Unknown Category from a Set of Images Sinisa Todorovic and Narendra Ahuja ... – Remove a node – Add](https://reader031.vdocuments.site/reader031/viewer/2022030920/5b7811e47f8b9a515a8e6ff5/html5/thumbnails/10.jpg)
Salient Descriptor for a Region
• An outstanding region among siblings?– Brighter/darker?– Noisier /more homogenous– Larger/Smaller– Higher/lower entropy
on boundary shape• Empirical result: best λ=0.5
Photometric Geometric
hv (1)hv (2)
hv (8)
. . .
![Page 11: Extracting Subimages of an Unknown Category … · Extracting Subimages of an Unknown Category from a Set of Images Sinisa Todorovic and Narendra Ahuja ... – Remove a node – Add](https://reader031.vdocuments.site/reader031/viewer/2022030920/5b7811e47f8b9a515a8e6ff5/html5/thumbnails/11.jpg)
Salience Contract Flow(microview)
Average Direction and Magnitude
¡!©v
¡!wv=d2+
++
+
++
![Page 12: Extracting Subimages of an Unknown Category … · Extracting Subimages of an Unknown Category from a Set of Images Sinisa Todorovic and Narendra Ahuja ... – Remove a node – Add](https://reader031.vdocuments.site/reader031/viewer/2022030920/5b7811e47f8b9a515a8e6ff5/html5/thumbnails/12.jpg)
Salience Contract Flow(macroview)
![Page 13: Extracting Subimages of an Unknown Category … · Extracting Subimages of an Unknown Category from a Set of Images Sinisa Todorovic and Narendra Ahuja ... – Remove a node – Add](https://reader031.vdocuments.site/reader031/viewer/2022030920/5b7811e47f8b9a515a8e6ff5/html5/thumbnails/13.jpg)
Match salience contract flow
~¡!©1 ¼
¡!©2
![Page 14: Extracting Subimages of an Unknown Category … · Extracting Subimages of an Unknown Category from a Set of Images Sinisa Todorovic and Narendra Ahuja ... – Remove a node – Add](https://reader031.vdocuments.site/reader031/viewer/2022030920/5b7811e47f8b9a515a8e6ff5/html5/thumbnails/14.jpg)
Store Regional Descriptor on Treenode
Photometric Geometric Salient
![Page 15: Extracting Subimages of an Unknown Category … · Extracting Subimages of an Unknown Category from a Set of Images Sinisa Todorovic and Narendra Ahuja ... – Remove a node – Add](https://reader031.vdocuments.site/reader031/viewer/2022030920/5b7811e47f8b9a515a8e6ff5/html5/thumbnails/15.jpg)
Segment out all the cars
….
fused tree model for cars
Unseen image
Training images
Segmented Cars
Segmentation Trees
Maximal CommonMaximal CommonSubtreeSubtree Matching Matching
![Page 16: Extracting Subimages of an Unknown Category … · Extracting Subimages of an Unknown Category from a Set of Images Sinisa Todorovic and Narendra Ahuja ... – Remove a node – Add](https://reader031.vdocuments.site/reader031/viewer/2022030920/5b7811e47f8b9a515a8e6ff5/html5/thumbnails/16.jpg)
Segment out all the cars
….
fused tree model for cars
Unseen image
Training images
Segmented Cars
Segmentation Trees
How does it works?How does it works?
![Page 17: Extracting Subimages of an Unknown Category … · Extracting Subimages of an Unknown Category from a Set of Images Sinisa Todorovic and Narendra Ahuja ... – Remove a node – Add](https://reader031.vdocuments.site/reader031/viewer/2022030920/5b7811e47f8b9a515a8e6ff5/html5/thumbnails/17.jpg)
Inexact Matching: Structural Noise
Use tree edit distance instead
![Page 18: Extracting Subimages of an Unknown Category … · Extracting Subimages of an Unknown Category from a Set of Images Sinisa Todorovic and Narendra Ahuja ... – Remove a node – Add](https://reader031.vdocuments.site/reader031/viewer/2022030920/5b7811e47f8b9a515a8e6ff5/html5/thumbnails/18.jpg)
Tree Edit Distance• Editor Operations : costs ~ Dissimilarity(x,y)
– Remove a node– Add a node – Replace a node
-
+r
![Page 19: Extracting Subimages of an Unknown Category … · Extracting Subimages of an Unknown Category from a Set of Images Sinisa Todorovic and Narendra Ahuja ... – Remove a node – Add](https://reader031.vdocuments.site/reader031/viewer/2022030920/5b7811e47f8b9a515a8e6ff5/html5/thumbnails/19.jpg)
Metaphor: String Edit Distance• Unifying Editor Operations
– Remove a node– Add a node (removal on partner) – Replace a node (paired removal on both string/tree)
AABBBBCCEdit : Add Y
AABBYBBCC
Edit : Remove Y
AABBXBBCCEdit : Replace X with Y
AABBYBBCC
Edit : Remove X Edit : Remove Y
![Page 20: Extracting Subimages of an Unknown Category … · Extracting Subimages of an Unknown Category from a Set of Images Sinisa Todorovic and Narendra Ahuja ... – Remove a node – Add](https://reader031.vdocuments.site/reader031/viewer/2022030920/5b7811e47f8b9a515a8e6ff5/html5/thumbnails/20.jpg)
Tree Edit Distance• Editor Operation (with costs)
– Two way removal only
t t’u=E1(t)∩E2(t’)Dist.(t, t’) = Dist.(t, u) + Dist(u, t’)
E():Sequenceof removal
E1() E1()
![Page 21: Extracting Subimages of an Unknown Category … · Extracting Subimages of an Unknown Category from a Set of Images Sinisa Todorovic and Narendra Ahuja ... – Remove a node – Add](https://reader031.vdocuments.site/reader031/viewer/2022030920/5b7811e47f8b9a515a8e6ff5/html5/thumbnails/21.jpg)
Reduce Edit-Distance matching to Non-edit matching
• Transitive Closure• (see animation)
Closure Original
![Page 22: Extracting Subimages of an Unknown Category … · Extracting Subimages of an Unknown Category from a Set of Images Sinisa Todorovic and Narendra Ahuja ... – Remove a node – Add](https://reader031.vdocuments.site/reader031/viewer/2022030920/5b7811e47f8b9a515a8e6ff5/html5/thumbnails/22.jpg)
Matching Criteria
![Page 23: Extracting Subimages of an Unknown Category … · Extracting Subimages of an Unknown Category from a Set of Images Sinisa Todorovic and Narendra Ahuja ... – Remove a node – Add](https://reader031.vdocuments.site/reader031/viewer/2022030920/5b7811e47f8b9a515a8e6ff5/html5/thumbnails/23.jpg)
Divide and Conquer
Try all pairs of (v, v’) combinations = O(|t| + |t’|)
NP-complete QP approx. O(|Cvv’|)
![Page 24: Extracting Subimages of an Unknown Category … · Extracting Subimages of an Unknown Category from a Set of Images Sinisa Todorovic and Narendra Ahuja ... – Remove a node – Add](https://reader031.vdocuments.site/reader031/viewer/2022030920/5b7811e47f8b9a515a8e6ff5/html5/thumbnails/24.jpg)
Segment out all the cars
….
fused tree model for cars
Unseen image
Training images
Segmented Cars
Segmentation Trees
Model Model GenerationGeneration
![Page 25: Extracting Subimages of an Unknown Category … · Extracting Subimages of an Unknown Category from a Set of Images Sinisa Todorovic and Narendra Ahuja ... – Remove a node – Add](https://reader031.vdocuments.site/reader031/viewer/2022030920/5b7811e47f8b9a515a8e6ff5/html5/thumbnails/25.jpg)
Model: Union of Subtrees
…
NP-Hard
1.Pairwise matching2.One by one union
Sub-optimalOptimal
= ∪
Next TreeT = T u Tnext
![Page 26: Extracting Subimages of an Unknown Category … · Extracting Subimages of an Unknown Category from a Set of Images Sinisa Todorovic and Narendra Ahuja ... – Remove a node – Add](https://reader031.vdocuments.site/reader031/viewer/2022030920/5b7811e47f8b9a515a8e6ff5/html5/thumbnails/26.jpg)
Category Model
![Page 27: Extracting Subimages of an Unknown Category … · Extracting Subimages of an Unknown Category from a Set of Images Sinisa Todorovic and Narendra Ahuja ... – Remove a node – Add](https://reader031.vdocuments.site/reader031/viewer/2022030920/5b7811e47f8b9a515a8e6ff5/html5/thumbnails/27.jpg)
Segment out all the cars
….
fused tree model for cars
Unseen image
Training images
Segmented Cars
Segmentation Trees
Testing:Testing:SegmentationSegmentation
![Page 28: Extracting Subimages of an Unknown Category … · Extracting Subimages of an Unknown Category from a Set of Images Sinisa Todorovic and Narendra Ahuja ... – Remove a node – Add](https://reader031.vdocuments.site/reader031/viewer/2022030920/5b7811e47f8b9a515a8e6ff5/html5/thumbnails/28.jpg)
Segment out all the cars
fused tree model for cars
Unseen image Segmented Cars
Testing: Detect & SegmentationMaximal CommonMaximal CommonSubtreeSubtree Matching Matching
Match : (Similarity > Thresh) (precision/recall)
![Page 29: Extracting Subimages of an Unknown Category … · Extracting Subimages of an Unknown Category from a Set of Images Sinisa Todorovic and Narendra Ahuja ... – Remove a node – Add](https://reader031.vdocuments.site/reader031/viewer/2022030920/5b7811e47f8b9a515a8e6ff5/html5/thumbnails/29.jpg)
Performace Evaluaton
![Page 30: Extracting Subimages of an Unknown Category … · Extracting Subimages of an Unknown Category from a Set of Images Sinisa Todorovic and Narendra Ahuja ... – Remove a node – Add](https://reader031.vdocuments.site/reader031/viewer/2022030920/5b7811e47f8b9a515a8e6ff5/html5/thumbnails/30.jpg)
Results (Caltech 101 Face)
Varying Matching Thresh. (precision/recall)
![Page 31: Extracting Subimages of an Unknown Category … · Extracting Subimages of an Unknown Category from a Set of Images Sinisa Todorovic and Narendra Ahuja ... – Remove a node – Add](https://reader031.vdocuments.site/reader031/viewer/2022030920/5b7811e47f8b9a515a8e6ff5/html5/thumbnails/31.jpg)
Results (UIUC Car Side View)
#positive/#training: 5/10 vs 10/20(2hr on P4-2.4G/2G)
![Page 32: Extracting Subimages of an Unknown Category … · Extracting Subimages of an Unknown Category from a Set of Images Sinisa Todorovic and Narendra Ahuja ... – Remove a node – Add](https://reader031.vdocuments.site/reader031/viewer/2022030920/5b7811e47f8b9a515a8e6ff5/html5/thumbnails/32.jpg)
Results (Caltech 101 Face)
#positive/#training: 3/6 vs 6/12
Rotation Invariant
![Page 33: Extracting Subimages of an Unknown Category … · Extracting Subimages of an Unknown Category from a Set of Images Sinisa Todorovic and Narendra Ahuja ... – Remove a node – Add](https://reader031.vdocuments.site/reader031/viewer/2022030920/5b7811e47f8b9a515a8e6ff5/html5/thumbnails/33.jpg)
Caltech (Cars Rear View)
#positive/#training: 10/20
![Page 34: Extracting Subimages of an Unknown Category … · Extracting Subimages of an Unknown Category from a Set of Images Sinisa Todorovic and Narendra Ahuja ... – Remove a node – Add](https://reader031.vdocuments.site/reader031/viewer/2022030920/5b7811e47f8b9a515a8e6ff5/html5/thumbnails/34.jpg)
Conclusion
• Contribution– Good Image Representation Seg. Tree
• Small amount of training data– Cf. Statistical Learning/Clustering
• Ex. Visual Words + pLSA
• Allow Non-category Images noise• Allow occlusion (disconnected regions)
![Page 35: Extracting Subimages of an Unknown Category … · Extracting Subimages of an Unknown Category from a Set of Images Sinisa Todorovic and Narendra Ahuja ... – Remove a node – Add](https://reader031.vdocuments.site/reader031/viewer/2022030920/5b7811e47f8b9a515a8e6ff5/html5/thumbnails/35.jpg)
Region Descriptor
Annotated Recursive Tree
Salient Flow
Sliced area histogram
Region Area
Graylevel
xx
xxx
x
x
x
TopologicalGeometricPhotmetric
![Page 36: Extracting Subimages of an Unknown Category … · Extracting Subimages of an Unknown Category from a Set of Images Sinisa Todorovic and Narendra Ahuja ... – Remove a node – Add](https://reader031.vdocuments.site/reader031/viewer/2022030920/5b7811e47f8b9a515a8e6ff5/html5/thumbnails/36.jpg)
Thank you
• Quicktopic
![Page 37: Extracting Subimages of an Unknown Category … · Extracting Subimages of an Unknown Category from a Set of Images Sinisa Todorovic and Narendra Ahuja ... – Remove a node – Add](https://reader031.vdocuments.site/reader031/viewer/2022030920/5b7811e47f8b9a515a8e6ff5/html5/thumbnails/37.jpg)
Cf. Visual Words+pLSA
• Visual Words recognize connected object only• Tree Matching is more conservative due to
intersection
![Page 38: Extracting Subimages of an Unknown Category … · Extracting Subimages of an Unknown Category from a Set of Images Sinisa Todorovic and Narendra Ahuja ... – Remove a node – Add](https://reader031.vdocuments.site/reader031/viewer/2022030920/5b7811e47f8b9a515a8e6ff5/html5/thumbnails/38.jpg)
Cf. Visual Words +pLSA
Visual Words/pLSA
Tree matching
![Page 39: Extracting Subimages of an Unknown Category … · Extracting Subimages of an Unknown Category from a Set of Images Sinisa Todorovic and Narendra Ahuja ... – Remove a node – Add](https://reader031.vdocuments.site/reader031/viewer/2022030920/5b7811e47f8b9a515a8e6ff5/html5/thumbnails/39.jpg)
Caltech FacesVisual Words/pLSA
Tree matching
![Page 40: Extracting Subimages of an Unknown Category … · Extracting Subimages of an Unknown Category from a Set of Images Sinisa Todorovic and Narendra Ahuja ... – Remove a node – Add](https://reader031.vdocuments.site/reader031/viewer/2022030920/5b7811e47f8b9a515a8e6ff5/html5/thumbnails/40.jpg)
ReSPEC(Use Color Histogram)