1 deformation invariant shape and image matching polikovsky senya advanced topics in computer vision...
TRANSCRIPT
1
Deformation Invariant Shape and Image
Matching
Polikovsky SenyaAdvanced Topics in Computer Vision Seminar
Faculty of Mathematics and Computer Science
Weizmann Institute
May 2007
2
Based on…
Integral Invariants for Shape Matching
Siddharth Manay, Daniel Cremers, Member, Byung-Woo Hong,Anthony J. Yezzi Jr., and Stefano Soatto
Deformation Invariant Image Matching
Haibin Ling ,David W. Jacobs
Part I : Invariant Shape Matching
Can you guess what it is ?
3
4
Outline
Integral Shape Matching• Introduction
– Basic Definitions, Previous Work – Curvature
• Integral Invariant (II)– Relation of Local Area II to Curvature
– Shape Matching and Distance– Multi-scale Shape Matching
• Implementation and Experimental Results
5
Applications of shape matching
• Airport security :
• Industry quality control :
• Medical images analyses :
6
Ultimate Goal
Compare objects represented as closed
planar contours.
7
Basic Definition
• Object : – Closed planar contour.– No self-intersections.
• Shape : – Equivalence class of objects. Obtained under the action of a finite-dimensional group : Euclidean, similarity, affine, projective group.
8
Basic Definition (cont.)
In Addition:
- articulated
- occluded
- jagged (not obtained with standard additive, zero mean)
Two objects have the same shape if and only if
one can be generated by transformation group actions on other shape.
9
Summary Goal
• Define a distance so that shapes that vary by Euclidean transformations have zero distance.
• Shapes vary by scaling , articulation , occlusion have small distance.
• Resistant to : “small deformations” “high-frequency noise” “localized changes”
10
Summary Goal (cont.)
The babies should have low distance to each other,
but high distance to other classes of shapes.
Low distance High distance
11
Previous Work
• Statistical approach using moments. [35],[27]
High-order moments sensitive to noise.
• Normalized Fourier descriptors. [93],[52],[2]
High order Fourier coefficients are not stable with respect to noise.
• Local neighborhoods using Wavelet transform. [83],[34]
Previous Work (Cont.)
1212
• Global radial histogram of the relative coordinates of the rest of the shape at each point. [6]
• Differential invariants (Curvature)
[44], [36], [17], [58], [76],
[30], [48], [64], [91], [85], [37]
13
Curvature
• Intuitively, curvature is the amount by which a geometric object deviates from being flat.
Less general y = f(x) Plane curve c(t) = (x(t),y(t))
2/322 )()(
yx
xyyxtk
2/32 )1( y
yk
• Curvature can be also be geometrical understood as in terms of osculating (kiss) circle and radius curvature.
14
Curvature (cont.)
• Circle of radius r has curvature 1/r everywhere .• Straight line r = ∞ has curvature 1/ ∞ =0 everywhere
• Features:– Invariant to rotation, reflections of the original curve.
– NOT invariant to scaling.
15
Outline
Integral Shape Matching• Introduction
– Basic Definitions, Previous Work – Curvature
• Integral Invariant (II)– Relation of Local Area II to Curvature
– Shape Matching and Distance– Multi-scale Shape Matching
• Implementation and Experimental Results
16
• Closed planar contour. C:S1 → R2. ds - infinitesimal arclength.
• G - transformation group acting on R2. dx - area %on R2.
• μ curve C corresponding measure dμ(x).
Integral Invariant
17
Integral Invariant
• Function I: R2 → R G-invariant if satisfies:
• I(.) is associates to each point on contour a real number.
GgCgICI )()(
Cp
18
Step 1: Curvature
Curvature κ(C) of curve C is G-invariant.
19
General notion of Integral Invariant
Function IC(p) : R2 → R is and integral G-invariant
Kernel k : R2 X R2 → R.
K( ● , ● ) :
• p don’t necessarily lie on the curve C.
)(),()( xdxpkpIC
},|{
)( ),()( ),(
xGggxg
Ggxdxgpkxdxpkg
p
20
Step 2: Distance integral invariant
CpxdsxppICC )()(
Unlike curvature distance invariant is R+.
(Euclidean distance is always nonnegative)
• For global descriptor , local change of a shape affects the values of the distance integral invariant for the entire shape.
21
Step 2: Distance integral invariant
CpxdsxppICC )()(
22
Step 3: “Shape Context”
)(),()( xdxpkpIC ))(,())(,(),(
),(),(),(
sCpdsCpqxpk
xpdxpqxpk
Preserves locality can be obtained by weighting the integral with a kernel q(p, x).
))(,( sCpq ))(,( sCpd
Local radial histogram.
23
Step 3: “Shape Context” (cont.)• NOT discriminative, same value for different geometric features.
r – radius. p - center of the ball. C - interior of the region bounded by C.
24
Finale Step: Local Area Integral Invar.
• Define a ball Br(p), Br : R2 X R2 {0,1} ; ( R+ )
0
1),(
otherwith
rxpxpBr )(),()( xdxpBpI
C rrC
)(),(
)(),()(
2xdxpB
xdxpBpI
R r
C rrC
25
Local Area Integral Invar.(cont.)
rr
• Naturally forms a multi-scale invariant.
26
Relation of Local AII to Curvature
22)( rpI rC
))(2
1(2)( 12 prcorrpI r
C
R
r
2)cos(
r
C
r
R
θ
C
Assume that C is smooth curve, because of the curvature.
pRp )(
27
Outline
Integral Shape Matching• Introduction
– Basic Definitions, Previous Work – Curvature
• Integral Invariant (II)– Relation of Local Area II to Curvature
– Shape Matching and Distance– Multi-scale Shape Matching
• Implementation and Experimental Results
28
Shape Matching and Distance
• Shape distance is a scalar value that quantifies similarity of the two contours. D(C1,C2).
• Basing on group invariant , integral invariant :– invariant to G group action. – robust to noise and local deformations
• Corresponding points.
C1 = C2=
Local Are Integral Invariant : I1 , I2
,
29
Corresponding points
• Disparity function d(s).
Reparameterizes C1 ,I1 and C2 ,I2 .
• optimal point correspondence
);,,(minarg)(* 21)(
sdIIEsdsd
SssdsCsdsC ,))((~))(( *2
*1
( ~ denotes correspondence)
30
Energy Functional E(...,d)
E1 - measures the similarity of two curves.E2 - elastic energy. α - control parameter α > 0.
dssdsdsIsdsI
dEdIIEdIIE
2'21
0 21
'2,211,21
)())(())((
)(),(),(
• If d(s) = 0 , direct match.• If d’(s) = 0 , circular “shifts”.• Other d(s), “stretch” , “shrink”.
31
Control Parameter α
small α :
large α :
d*(s)
d*(s)
32
Multi-scale Shape Matching
Trace of local extrema across scales
Curvature Scale-space Integral Invariant Scale-space
Curvature scale-space is derived from Gaussian smoothing.
33
Multi-scale Shape Matching (cont.)
• Matching shapes of different sizes.
R R’
'
'
R
r
R
r
• Normalized kernel radius : r / R.
34
Multi-scale Shape Matching (cont.)
Control parameter α. Size of the kernel width r.
Correspondences between two signals influenced by:
fine scale intermediate scale coarse scale
35
Outline
Integral Shape Matching• Introduction
– Basic Definitions, Previous Work – Curvature
• Integral Invariant (II)– Relation of Local Area II to Curvature
– Shape Matching and Distance– Multi-scale Shape Matching
• Implementation and Experimental Results
36
Implementation
C1
C2
C2[j]C2[j+1]
C1[i]
C1[i+1]v[
i , j
]
v[i ,
j +
1]
v[i + 1, j]v[
I +
1, j
+ 1
]
Each point in each curve must have at least onecorresponding point in the other curve.
v[i , j]
v[I + 1, j + 1]
v[i +1, j]
v[i , j+1]
e e
e
e = v(i,j) → v(k,l)
Minimization of the energy functional E.
37
Implementation (cont.)
),()...0,0(0 MNvvvv L
Lvvvp ,..., 10
• Minimization of the energy functional E is equivalent to finding a shortest path that gives a minimum weight.
L
ttt vvwpw
01),()(
w(p) ← E(I1,I2,d)Graph used to compute the correspondence for two
curves with M = N = 5.
nodes = MNedges = 3MN
v[i , j]
v[I + 1, j + 1]
v[i +1, j]
v[i , j+1]
e e
e
M
N
0
0
1
1
C1
C2
38
Implementation (cont.)
• In previous implementations we choose start point in C1 and run all over C2.
• Alternately, observing strong features, in the invariant space.
39
Experimental Results
The gray levels indicate the dissimilarity between points lighter shade indicates higher dissimilarity.
dissimilarity
higher dissimilarity
40
Experimental Results (cont.)
“On Aligning Curves” [76]
• 100 sample on contours• r = 15 • α = 0.1
41
Experimental Results (cont.)
Int.Inv.
Curvature
Curvature
Histograms of shape distance between Shape 24 and1,000 perturbations of Shape 20 with noise at variance = 2.5.
Int.Inv.
Curvature
42
Experimental Results (cont.)
Noisy shapes (across top) and original shapes (along left side)vie differential invariant
dissimilarity
higher dissimilarity
43
Experimental Results (cont.)
Noisy shapes (across top) and original shapes (along left side)via integral invariant
dissimilarity
higher dissimilarity
44
Summary
Curvature.
Four Steps : – Curvature, sensitivity to noise.– Global descriptor, local change.– Local radial histogram, NOT discriminative.– Local Area Integral Invar.
Shape Matching ,Multi-scale Shape Matching.
Implementation.
Results.
M
N
0
0
1
1C1
C2
Part II :Deformation Invariant Image
Matching
45
Outline
Introduction
Deformation Invariant Framework
Experiments
Summary
General Deformation
• One-to-one, continuous mapping.• Intensity values are deformation invariant.
– (their positions may change)• Affine model for lighting change. (Out of the scope)
Solution
• A deformation invariant framework
– Embed images as surfaces in 3D
– Geodesic distance is made deformation invariant by adjusting an embedding parameter
– Build deformation invariant descriptors using geodesic distances
Outline
Introduction
Deformation Invariant Framework- Intuition through 1D images
- 2D images
Experiments
Summary
1D Image Embedding
1D Image I(x):
EMBEDDINGI(x) ( (1-α)x, αI )αI(1-α)x
Aspect weight α : measures the importance of the intensity
2D Surface :
Geodesic Distance
αI
(1-α)x
p qg(p,q)
• Length of the shortest path along surface
Geodesic Distance and α
I1 I2
Geodesic distance becomes deformation invariant
for α close to 1
Image Embedding & Curve Lengths
]1,0[:),( 2 RyxI
dtIyxl ttt 222222 )1()1(
))('),('),('()( tztytxt
Depends only on intensity I Deformation Invariant
IzyyxxI ',)1(',)1('),(
dtIl t
21
Image I:
Embedded Surface σ :
Curve γ on σ:
Length of γ:
Take limit
Geodesic distances for real images
55
Automatically adapts to deformation.
Almost like Euclidean distances
• Interest point p0 = (x0, y0).• Compute Geodesic distances from p0 to all other points on embedded surface σ(I; α).
α = 0
α = 0.98
Geodesic Distance for 2D Images
• Computation– Geodesic level curves – Fast marching [Sethian96]
is the marching speed 2/122222)1(
yx IIF
• Geodesic distance– Shortest path– Deformation invariant
F
T is the geodesic distance
T=1T=2T=3
T=4
p
q1|| FT
Deformation Invariant Sampling
Geodesic Sampling1. Fast marching: get
geodesic level curves with sampling interval Δ
2. Sampling along level curves with Δ
p
sparsedense
Δ
ΔΔ
Δ
Δ
Geodesic-Intensity Histogram(GIH)• Divide 2D intensity-geodesic distance space into K×M bins. K - number of intensity intervals. M - number of geodesic distance intervals.• Insert all points in Pp into Hp.• Normalize each column of Hp Normalize the whole Hp.
Deformation Invariant Descriptor
p qp q
geodesic distance M
Inte
nsit
y
K
geodesic distance M
Inte
nsit
y
K
Real Example
pq
Invariant vs. Descriminative
0
10
1
Deformation Invariant Framework
Image Embedding ( α close to 1)I(x,y) → σ(I, α)
Deformation Invariant SamplingGeodesic Sampling
Build Deformation Invariant Descriptors(GIH)
Practical Issues
• Interest-Point– No special interest-point is required.
Automatically locates the support region by
Geodesic Sampling.
But:– Points on constant region indistinguishable.– Corners may vary due to sampling.
• Extreme point , local intensity extremum,
is more reliable and effective.
Outline
Introduction
Deformation Invariant Framework- Intuition through 1D images
- 2D images
Experiments- Interest-point matching
Summary
Data Sets
Synthetic Deformation & Lighting Change (8 pairs) Real Deformation (3 pairs)
Interest-PointsInterest-point Matching
• Harris-affine points [Mikolajczyk&Schmid04] *
• Affine invariant support regions• Not required by GIH• 200 points per image
• Ground-truth labeling• Automatically for synthetic image pairs• Manually for real image pairs
• Correct match, three pixel distance.
Descriptors & Performance Evaluation
Descriptors• GIH compared with following descriptors:
Steerable filter [Freeman&Adelson91], SIFT [Lowe04], Moments [VanGool&etal96], Complex filter [Schaffalitzky&Zisserman02], Spin Image [Lazebnik&etal05]• α = 0.98 Performance Evaluation• Receiver Operating Characteristics (ROC) curve: detection
rate among top N matches. • Detection rate:
matches possible#
matchescorrect #r
Synthetic Image Pairs
N – top matches.
Real Image Pairs
N – top matches.
Outline
Introduction
Deformation Invariant Framework- Intuition through 1D images
- 2D images
Experiments- Interest-point matching
Summary
Summary
1D Image Intuition.
Geodesic Distance and α.
Geodesic Sampling.
Performance Evaluation.
p ΔΔ ΔΔ
Δ
Thank You!