direct fitting of gaussian mixture modelsmesh registration with p2d method 1. apply a random...
TRANSCRIPT
Direct Fitting of Gaussian Mixture Models
Leonid Keselman, Martial Hebert
Robotics InstituteCarnegie Mellon University
May 29, 2019
https://github.com/leonidk/direct_gmm1
Representations of 3D data
2
Point Cloud Point Cloud+ Normals
Triangular Mesh
Nearest Neighbor Plane Fit Screened Poisson Surface Reconstruction
Gaussian Mixture Models for 3D Shapes
3
GMM fit to object surface
Benefits• Closed-form expression• Can represent contiguous surfaces• Easy to build from noisy data• Sparse
Gaussian Mixture Model (GMM)
4
! " = $%&'
()* +("; .*, Σ*) $
*)* = 1)* ≥ 0
Σ* is symmetric, positive-semidefinite
Gaussian Mixtures as a shape representation
W. Tabib, C. O’Meadhra, N. MichaelIEEE R-AL (2018)
B. Eckart, K. Kim, A. Troccoli, A. Kelly, J. Kautz. CVPR (2016)
B. Eckart, K. Kim, J. Kautz. ECCV (2018)
5
Efficient Representation Mesh Registration Frame Registration
Fitting a Gaussian Mixture Model
1. Obtain 3D Point Cloud2. Select Initial Parameters3. Iterate Expectation & Maximization
i. E-Step: Each point gets a likelihoodii. M-Step: Each mixture gets parameters
6
7
The E-Step (Given GMM parameters)
8
!"# =1&#'" ((*#; ,", Σ")
&# =01'1 ((*#; ,1, Σ1)
Normalization constant for point j
Affiliation between point j & mixture i
The M-Step (Given point-mixture weights)
9
!" = $%&'
($)&'
*+)%log /) 0(2%; 4), Σ))
8!"8/)
= 0
mixturespoints
8!"84)
= 0 8!"8Σ)
= 0
lower-bound loss
To get new parameters: takes derivatives, set equal to zero, and solve
4) =1;)$
%<)%2%/) =
;)=
Σ) =1;)$
%<)%(2%−4))(2%−4))?
<)% = +)%
;) =$%<)%
10
Geometric Objects in a Probability Distribution
11
Known curve in a given 2D probability distribution
Geometric Objects in a Probability Distribution
12
Consider sampling N points from this curve
ℓ curve ≅()*+
,-(/))
Geometric Objects in a Probability Distribution
13
Take a geometric mean to account for sample number
ℓ curve ≅ ()*+
,-(/))
+,
Geometric Objects in a Probability Distribution
14
The curve will be the value in the limit
ℓ curve ≅ ()*+
,-(/))
+,
ℓ curve = lim,→6 ()*+
,-(/))
+,
= lim,→6exp log ()*+
,-(/))
+,
= lim,→6exp1< =
)*+
,log(-(/)))
Geometric Objects in a Probability Distribution
15
ℓ curve ≅ ()*+
,-(/))
+,
ℓ curve = lim,→6 ()*+
,-(/))
+,
= lim,→6exp log ()*+
,-(/))
+,
= lim,→6exp1< =
)*+
,log(-(/))) = exp > log(-(/)) ?/
Geometric Objects in a Probability Distribution
16
!= exp & log(+(,)) .,
1. If +(,) = 0 on curve, then L= 02. Invariant to reparameterization
17
!" Area of each triangle#" Centroid of each triangle$", &", '" Triangle vertices
The E-Step (Given GMM parameters)
18
!"# =1&#'" ((*#; ,", Σ")
&# =01'1 ((*#; ,1, Σ1)
Normalization constant for point j
Affiliation between point j & mixture i
2# Area of each triangle,# Centroid of each triangle3#, 4#, &# Triangle vertices
The New E-Step (Given GMM parameters)
19
!"# =1&#'"(# )(+#; +", Σ")
&# =01'1(1)(+#; +1, Σ1)
Normalization constant for object j
Affiliation between object j & mixture i
(# Area of each triangle+# Centroid of each triangle2#, 3#, &# Triangle vertices
Taylor Approximation(2 terms)
The M-Step (Given point-mixture weights)
20
!" =1%"&
'("')'
*" =%"+
Σ" =1%"&
'("'()'−!")()'−!")0
("' = 1"'
%" =&'("'
The New M-Step (Given point-mixture weights)
21
!" =1%"&
'("')'
*" =%"+
Σ" =1%"&
'("' ()'−!")()'−!")0 + Σ'
("' = 2'3"'
%" =&'("'
2' Area of each triangle!' Centroid of each triangle4', 6', 7' Triangle vertices
Σ' =112 4'4'0 + 6'6'0 + 7'7'0 − 3 !'!'0
What is Σ"?
22
23
ResultsDid all that math actually help us fit better/faster GMMs?
24
25
Using different inputs
classic algorithm• Vertices of the mesh• Triangle centroids
our method• Approximate (E only)• Exact (E + M steps)
Measure the likelihood of a high-density point cloud (higher is better)
Evaluate across a wide range of mixtures(6 to 300)
Full E+M method works in all cases
26
Stable under even random initialization!
27
ApplicationsAre these models actually more useful?
28
Mesh Registration (P2D)
Method1. Apply a random rotation + translation to the point cloud2. Find transformation to maximize the likelihood of the points
• Perform P2D with GMMs fit toi. mesh verticesii. mesh triangles
29
Eckart, Kim, Kautz. “HGMR: Hierarchical Gaussian Mixtures for Adaptive 3D Registration.” ECCV (2018)
Mesh-based GMMs are more accurate
30
Across multiple models
31
0
50
100
150
Armadillo Bunny Dragon Happy Lucy
Rotation Error(% of ICP)
points mesh
0
100
200
Armadillo Bunny Dragon Happy Lucy
Translation Error(% of ICP)
points mesh
Frame Registration (D2D)
Method1. Use a sequence from an RGBD Sensor
• 2,500 frame TUM sequence from a Microsoft Kinect
2. Pairwise registration between t & t-1 frames• Optimize the D2D L2 distance• Build GMMs using square pixels as the geometric object
32
W. Tabib, C. O’Meadhra, N. Michael.“On-Manifold GMM Registration” IEEE R-AL (2018)
Representing points using pixel squares
33
D2D Registration Results
34
Compared to standard GMM• 2.4% improvement in RMSE• 22% faster D2D convergence
Questions?
35
! = exp & log(+(,)) .,
The End!
36
Extra Slides
37
How to fit a Gaussian Mixture Model?
1. Obtain any collection of objects2. Perform Expectation + Maximization
i. E-Step: Each point gets a likelihoodii. M-Step: Each mixture gets new parameters
38
Extension to arbitrary primitives
39Vasconcelos, Lippman. "Learning mixture hierarchies.” Advances in Neural Information Processing Systems (1999)
Approximation
40
area-weighted geometric mean using the primitive’s centroids
Product Integral Formulation
• Product integrals provide a resampling-invariant loss function• Given S samples, of M primitives, with N mixture components
• This can be evaluated in the limit of samples (with a geometric mean)
41
42
43
44
For GMMs we will use the lower bound
45
P2D Registration Results
46
Model Rotation Error(% of ICP)
Translation Error(% of ICP)
points mesh points meshArmadillo 127 37 161 33Bunny 50 28 41 17Dragon 68 25 40 19Happy 101 27 85 27Lucy 95 23 122 35
Mesh Registration with P2D
Method1. Apply a random rotation + translation to the point cloud2. Point-to-Distribution (P2D) registration of point cloud to GMM
• Perform tests with GMMs fit toi. mesh verticesii. mesh triangles
• Optimize the GMM likelihood with rigid body transformation (q & t)• BFGS Optimization using numerical gradients, starting from identity
47
Eckart, Kim, Kautz. “HGMR: Hierarchical Gaussian Mixtures for Adaptive 3D Registration.” ECCV (2018)
Acknowledgements
• Martial Hebert• Robotics Institute• Reviewers• Funding?
48
Representing points using pixel squares
49