1/38
Dr.-Ing. Sung Joon Ahn
CurvSurf, Inc.
Algorithmic Techniques for Parametric Model Recovery
2/38
• Dr. Ahn, Sung Joon
• BS: Seoul Nat’l University (1985)
MS: KAIST (1987)
Dr.-Ing.: University of Stuttgart (2004)
• LG Electronics (1987–1990)
Fraunhofer IPA (1990–2004)
Sungkyunkwan University (2005–2013)
CurvSurf, Inc. (2013– )
• Circular coded target
Image measurement errors
Stripe-projection method
Orthogonal distance fitting (this talk)
Prolog (1)
Production Eng.
Robotics & Autom.
Mechanical Eng.
Mechanical design
Tech. info. proc.
Computer Vision &
Graphics
Meas. automation
Photogrammetry
3-D measurement
Numerical Math.
Edu
Job
Pub
Who I am
3/38
• Parametric model recovery
• Automation of parametric model recovery
= The Holy Grail.
Prolog (2)
4/38
Prolog (3)
• Tools of parametric model recovery
• Hough Transform (Hough 1959)
– Discrete parameter space (bin space)
– Applicable to small 𝑝 and not small 𝑛 (Length of bin−axis−array)𝑝
Resolution of parameters
∝ 𝑛 (should be large enough)
• RANSAC (Fischler 1981)
– Discrete parameter space (pick-points)
– Applicable to small 𝑝 and not small 𝑛 𝑛𝑝
𝑚𝑝
=𝑛 𝑛−1 … 𝑛−𝑝+1
𝑚 𝑚−1 … 𝑚−𝑝+1= Hit−rate of picking p points belong to the target feature
• Least Squares Method (Gauss 1795)
– Continuous parameter space
– Applicable to a wide range of curves and surfaces ( 𝑝 ≤ 𝑛 )
– Needs robust algorithms for segmentation & model fitting.
𝑝 = DOF of parameters
= Dim. of bin space
= No. of pick-points
Plane: 3
Cylinder: 2+3
Sphere: 4
Cone: 6
Torus: 7
𝑚 = No. of data points
𝑛 = No. of target points
5/38
• Parametric model recovery, directly from point cloud or via meshing?
• Via meshing (with Rev. Eng. SW)
– Why meshing?: Point normal and connectivity.
But, meshes from point cloud are arbitrary
– Information loss: Point decimation and noise
reduction are obligatory before meshing,
to hold inter-point distance ≫ variations
– Model meshes visualization: Due
Noisy points meshes model: Nonsense!
• Directly from point cloud (noisy points model)
– Online processing is possible
– Wide range of applications
– Points are individually processed as measured.
Prolog (4)
1
2
1
3
3
4
4
Top
Front
Case 1 Case 2
Var.
6/38
Content
1. Prolog
2. Error Model for Point Cloud Point & points
3. Orthogonal Distance Fitting Best math. tool of model fitting
4. Competent Parameterization Math. model description
5. Local Curvature Analysis Solution to initial value problem
6. Region Growing Segmentation & model updating
7. Automatic Feature Extraction The Holy Grail
8. Closing Now & future
7/38
Error Model for Point Cloud (1)
• Point error model
• Cartesian measuring device: Distance-distance-distance errors
Spherical measuring device: Angle - angle - distance errors
• Assume a Gaussian error ellipsoid
• Maximum likelihood estimation
– Maximize the probability density
The measurement point substitutes an unknown object point.
The nature of
coordinate metrology
8/38
Error Model for Point Cloud (2)
• Curve/surface error model
• Gaussian error band (= Superposition of Gaussian error ellipsoids)
– Measurement uncertainty is observable only in normal direction (= The lateral size of object is unmeasurable by probing in normal direction)
• Maximum likelihood estimation
– Maximize the ‘sum’ of probability density
Model feature substitutes an unknown object feature, minimizing
the sum of the squared error distances to measurement points.
The nature of
coordinate metrology
9/38
Least Squares Curve/Surface Fitting
• Best error measure:
The minimum (shortest/orthogonal/normal) distance
between measurement point and model curve/surface
(Invariant to coordinate transformation)
• Twofold nonlinear minimization problem
• Find the minimum distance points on model curve/surface
• Minimize the sum of the squared minimum distances
through adjusting the model parameters.
Best math. tool of
model fitting
10/38
Moment Method for Line/Plane
• Alias: Eigenvalue decomposition, total least squares,
principal component analysis (cf. linear regression)
• Line/plane fitting:
• Minimize the sum of the squared normal distances between
the measurement points and the model line/plane
• Linear problem: Solved in closed form (Pearson 1901).
Invariant to
coordinate
transformation
Best math. tool of
model fitting
11/38
Orthogonal Distance Fitting (1)
• Alias: Best fitting, Euclidean fitting, geometric fitting
• Curve/surface fitting:
• Minimize the sum of the squared shortest distances between
the measurement points and the model curve/surface
• Nonlinear problem: Solved through iteration (Boggs 1987)
• S.J. Ahn, doctoral diss., LNCS 3151, Springer-Verlag, 2004.
Best math. tool of
model fitting
12/38
Orthogonal Distance Fitting (2)
• Information flow with ODF of implicit curve/surface
Best math. tool of
model fitting
13/38
Orthogonal Distance Fitting (3)
• Infomat’n flow with ODF of parametric curve/surface
Best math. tool of
model fitting
14/38
Doctoral dissertation
University of Stuttgart
2004
Orthogonal Distance Fitting (4) Best math. tool of
model fitting
15/38
Content
1. Prolog
2. Error Model for Point Cloud
3. Orthogonal Distance Fitting
4. Competent Parameterization
5. Local Curvature Analysis
6. Region Growing
7. Automatic Feature Extraction
8. Closing
16/38
• Math. description of ODF line
* The ODF line runs through 𝐗 ( ). While 𝜅 and 𝐗 are relatively insensitive to small changes
in measurement points, 𝑏, 𝑐, and 𝑑 change significantly their values because of the
leverage effect centered at 𝐗 . Thus, the best description of ODF line is using 𝜅 and 𝐗 .
Furthermore, 𝜅 and 𝐗 are invariant to the coordinate transformation of measurement
points, reproducing perfectly the rotation and translation of measurement points.
Competent parameterization (1) Curves and surfaces
in space
rcos𝜅sin𝜅
,
17/38
Competent parameterization (2)
• Curve/surface in space
• Conic section: Which a curve is this, ellipse?
𝑎𝑋2 + 𝑏𝑌2 + 𝑐𝑋𝑌 + 𝑑𝑋 + 𝑒𝑌 + 𝑓 = 0.
If so, axis lengths? Center? Pose angle?
• Quadratic surface: ? ? ? ? !
𝑎𝑋2 + 𝑏𝑌2 + 𝑐𝑍2 + 𝑑𝑋𝑌 + 𝑒𝑌𝑍 + 𝑓𝑍𝑋 + 𝑔𝑋 + ℎ𝑌 + 𝑖𝑍 + 𝑗 = 0
• Cubic, quartic, catenary, 3-D circle, helix, sine curve, … … … .
• Recall the ODF line: 𝑋𝑌
= 𝑢cos𝜅sin𝜅
+ 𝑋
𝑌 , decomposed to
𝑋𝑌
=cos𝜅 −sin𝜅sin𝜅 cos𝜅
𝑥𝑦 + 𝑋
𝑌 Coordinate transformation:
𝐗=𝐑−1𝐱+𝐗
and 𝑦 = 0 or 𝑥𝑦 =
𝑢0
Canonical form:
𝑓 𝑥,𝑦 =0 or 𝑥(𝑢)𝑦(𝑢)
.
Curves and surfaces
in space
18/38
Competent parameterization (3)
• Parameter decomposition (cf. camera parameters in Photogrammetry)
• Canonical form: Specific to model curve/surface 𝐗 = 𝐑−1𝐱 + 𝐗o: Common to all curves and surfaces in space
• Intrinsic: Shape parameters (size, radius, vertex angle, etc.)
Extrinsic: Position/rotation parameters (rigid body motion).
Canonical form of circle
(𝑋 − 𝑋o)2+(𝑌 − 𝑌o)
2−𝑟2 = 0.
+
Curves and surfaces
in space
Don’t
combine!
19/38
Competent parameterization (4)
• ODF with parameter decomposition
Curves and surfaces
in space
Canonical form
20/38
Competent parameterization (5)
• Examples of canonical form (Implementation of ISO 10360-6)
• 2-D line : 𝑦 = 0
• 3-D line : (0, 0, 𝑢)
• Plane : 𝑧 = 0
• 2-D circle : 𝑥2 + 𝑦2 − 𝑟2 = 0
• 3-D circle : (𝑟cos𝑢, 𝑟sin𝑢, 0)
• Sphere : 𝑥2 + 𝑦2 + 𝑧2 − 𝑟2 = 0
• Cylinder : 𝑥2 + 𝑦2 − 𝑟2 = 0
• Cone : 𝑥2 + 𝑦2 − (𝑟 − 𝑧tan(𝜓/2))2= 0
• Torus : 𝑥4 + 𝑦4 + 𝑧4 + 2 𝑥2𝑦2 + 𝑦2𝑧2 + 𝑧2𝑥2 −2 𝑅2 + 𝑟2 𝑥2 + 𝑦2 + 2 𝑅2 − 𝑟2 𝑧2 + 𝑅2 − 𝑟2 2 = 0.
2-D/3-D curve: Ellipse, hyperbola, parabola, sine curve, catenary, helix, ...
Surface:
Ellipsoid, hyperboloid, paraboloid, …
Curves and surfaces
in space
21/38
Competent parameterization (6)
• Position of ODF cylinder/cone (cf. position of ODF line: 𝐗 )
• A point on axis
• The competent one is the closest point to the mass center 𝐗
of data points, resulting in minimum variances because the
ODF feature is adhering to data points.
Curves and surfaces
in space
22/38
Content
1. Prolog
2. Error Model for Point Cloud
3. Orthogonal Distance Fitting
4. Competent Parameterization
5. Local Curvature Analysis
6. Region Growing
7. Automatic Feature Extraction
8. Closing
23/38
Local Curvature Analysis (1)
• Provide ODF with initial values
• Type of feature: Plane, sphere, cylinder, cone, or torus,
determined by comparing the signed curvature radii
• Feature parameters: Radius, position, and axis direction,
derived from the principal curvatures and axes.
Plane Sphere (torus) Cylinder (cone) Torus
* k1, k2 : Principal curvatures
Differential geometry
24/38
• Initial values for ODF
• Radius of sphere/cylinder = |𝑟1| Position of sphere/cylinder = 𝐗f − 𝑟1𝐧 = 𝐗o
Axis direction of cylinder = 𝐯
• Cone and torus:
Beyond this talk’s
scope.
Local Curvature Analysis (2)
𝐗f : Surface point
𝐧 : Surface normal
𝐮, 𝐯 : Principal axes
𝑟1, 𝑟2 : Curvature radii
= 1 𝑘1 , 1 𝑘2 .
|𝑟1| ≤ |𝑟2|.
Differential geometry
25/38
Local Curvature Analysis (3)
• Automatic feature extraction
1. Pick a point in point cloud
2. Cut out a small patch around the pick-point
3. ODF plane to the patch
4. ODF quadratic to the patch, starting from the ODF plane
5. Calculate the principal curvatures and axes
6. Determine the type and initial values of ODF feature
7. Refine the ODF feature by means of region growing.
Differential geometry
26/38
Region Growing (1)
• Loop of segmentation and model updating (fitting)
1. Initial model feature is given
2. Collect inlier candidates around the current model feature
3. Eliminate/clear outliers from inlier candidates
4. Update the model feature through ODF to inliers
5. if (any break condition is invoked) break;
else go to 2.;
• Criteria of continuing, discarding, or accepting
• Rms error of inliers
• Parameter reliability (covariance)
• Density of inliers, etc.
27/38
Region Growing (2)
• Collecting inlier candidates
• Activity level (passive to aggressive) of region growing is
controlled in two mutually orthogonal directions:
– Radial expansion: 2~3 times the current rms error
– Lateral extension: Increment of current feature length/width.
Definition of search domain
hull volume for inlier
candidates around the
current model cylinder
28/38
Content
1. Prolog
2. Error Model for Point Cloud
3. Orthogonal Distance Fitting
4. Competent Parameterization
5. Local Curvature Analysis
6. Region Growing
7. Automatic Feature Extraction
8. Closing
29/38
• Fully Automatic Feature Extraction
– Combine all the techniques presented
– The pick-point is randomly selected
– Not ripe enough yet for industrial applications
– To do: Point cloud + image.
Automatic Feature Extraction (1)
30/38
Automatic Feature Extraction (2)
“Fully automatic feature extraction from point cloud,”
http://www.youtube.com/watch?v=42ZKbkjpBbY
31/38
• Semi-automation (1)
• Keeping the person in the loop
– Agent of feature extraction:
Diligent worker inside point cloud
User:
Intelligent decision maker giving instructions to agent
(Type of target feature and pick-point)
– The agent works in the way and manner the user expected
Boosting user’s confidence & productivity!
Automatic Feature Extraction (3)
32/38
• Semi-automation (2)
• Low cost of producing high-quality deliverables
– Low rate of Type I and Type II errors
Person in the loop
– Best quality of feature extraction (ISO 10360-6)
The nature of ODF
– Short lap time per feature
Single-point clicking
No clipping
Zooming & panning if necessary
• DLL modules (Win32, x64)
– Agent of feature extraction
– Application neutral.
Automatic Feature Extraction (4)
33/38
• DLL modules of CurvSurf, Inc.
• Extracting:
– Plane, sphere, cylinder, cone, or torus
• No. of pick-points:
– 1: Model features of usual form & size
– 2: Long strip plane,
long rod cylinder
– 3: Large plane,
large sphere,
thin disk cylinder,
thin disk cone,
thin ring torus.
Automatic Feature Extraction (5)
34/38
• DLL parameters
• Point cloud
– Point accuracy
– Point density
• Patch around the pick-point
– Touch radius (size of patch)
• Region growing
– Radial Expansion: 0–10
– Lateral Extension: 0–10
• Switching from cone to cylinder
– Threshold value of vertex angle
• Callback to caller program
– Informs the caller of the current states of feature extraction.
Set DLL parameters
Automatic Feature Extraction (6)
Select a DLL module
Pick point(s)
Press ‘F3’
Last again
35/38
Automatic Feature Extraction (7)
“Feature Extraction from 2 Mio. Points”
http://www.youtube.com/watch?v=OdJSIYidEXg
36/38
• Best error measure
• Orthogonal distance fitting
• Competent parameterization
• Local curvature analysis
• Region growing
• Automatic feature extraction
• Long way to go.
You have learned
37/38
• The Status Quo
• Optical 3-D measuring device & data processing machine
– 10 times more versatile & powerful in every 4–5 years
• IT Giants moved in 3-D
– Apple (PrimeSense), Google (Tango), Intel (RealSense),
Microsoft (Kinect)
• The One Problem
– Automatic feature extraction
• Outlook
• ODF: Tool of processing dimensional information
• Competent parameterization:
Rule of how to handle curves and surfaces in space
• Information fusion (multi-sensor, multi-source).
Closing
38/38
• Further discussions are welcomed
Closing
Thank you!