inverse kinematics ii and motion capture
TRANSCRIPT
![Page 1: Inverse Kinematics II and Motion Capture](https://reader036.vdocuments.site/reader036/viewer/2022062318/62a930f75d3cb8245a557173/html5/thumbnails/1.jpg)
Institute of Visual Computing
Inverse Kinematics II and Motion Capture
Luca Ballan
Mathematical Foundations of Computer Graphics and Vision
![Page 2: Inverse Kinematics II and Motion Capture](https://reader036.vdocuments.site/reader036/viewer/2022062318/62a930f75d3cb8245a557173/html5/thumbnails/2.jpg)
Comparison 0 1
2 3
A B C
is equal to the length of the bone computing the derivative on the
angle is easy
the meaning of is not intuitive However this incorporates the real concept of geodesic interpolation/averaging has to be done in this
space
Real exponential map Fake exponential map
![Page 3: Inverse Kinematics II and Motion Capture](https://reader036.vdocuments.site/reader036/viewer/2022062318/62a930f75d3cb8245a557173/html5/thumbnails/3.jpg)
Special Euclidean group SE(3)
Angle/axis representation of the rotation
Twist
exp
Screw motion: rotation along an axis + a translation along the same axis.
Proposition: Any rigid transformation in SE(3) can be expressed as a rotation about an axis combined with a translation parallel to that axis.
![Page 4: Inverse Kinematics II and Motion Capture](https://reader036.vdocuments.site/reader036/viewer/2022062318/62a930f75d3cb8245a557173/html5/thumbnails/4.jpg)
Special Euclidean group SE(3)
Twist exp
Tangent space of SE(3) at the identity I
hat
![Page 5: Inverse Kinematics II and Motion Capture](https://reader036.vdocuments.site/reader036/viewer/2022062318/62a930f75d3cb8245a557173/html5/thumbnails/5.jpg)
Properties
Identity
Inverse (basic property of exp map)
in general not “Linear” (like in so(3))
Derivative
![Page 6: Inverse Kinematics II and Motion Capture](https://reader036.vdocuments.site/reader036/viewer/2022062318/62a930f75d3cb8245a557173/html5/thumbnails/6.jpg)
Metric on SE(3)
Riemannian/Geodesic/Angle metric ( = the length of the geodesic connecting and )
Interpolation and averaging can be performed in the space of rigid transformations also
SLERP (spherical linear interpolation)
I
![Page 7: Inverse Kinematics II and Motion Capture](https://reader036.vdocuments.site/reader036/viewer/2022062318/62a930f75d3cb8245a557173/html5/thumbnails/7.jpg)
Metric on SE(3)
Riemannian/Geodesic/Angle metric ( = the length of the geodesic connecting and )
Interpolation and averaging can be performed in the space of rigid transformations also
SLERP (spherical linear interpolation)
Fréchet mean
I
![Page 8: Inverse Kinematics II and Motion Capture](https://reader036.vdocuments.site/reader036/viewer/2022062318/62a930f75d3cb8245a557173/html5/thumbnails/8.jpg)
A note on Interpolation
![Page 9: Inverse Kinematics II and Motion Capture](https://reader036.vdocuments.site/reader036/viewer/2022062318/62a930f75d3cb8245a557173/html5/thumbnails/9.jpg)
Inverse Kinematics
0 1 2 3
A B C
Target q
Newton’s method let be the current estimate for the solution compute the Taylor expansion of around
![Page 10: Inverse Kinematics II and Motion Capture](https://reader036.vdocuments.site/reader036/viewer/2022062318/62a930f75d3cb8245a557173/html5/thumbnails/10.jpg)
Inverse Kinematics
0 1 2 3
A B C
Target q
Newton’s method let be the current estimate for the solution compute the Taylor expansion of around
Find such that
![Page 11: Inverse Kinematics II and Motion Capture](https://reader036.vdocuments.site/reader036/viewer/2022062318/62a930f75d3cb8245a557173/html5/thumbnails/11.jpg)
Inverses
Find such that
is not injective (in general) (multiple inverse exists for each point of the image)
If it is injective, then the left-Inverse exists
![Page 12: Inverse Kinematics II and Motion Capture](https://reader036.vdocuments.site/reader036/viewer/2022062318/62a930f75d3cb8245a557173/html5/thumbnails/12.jpg)
Inverses
Find such that
might not contain the solution
Better to find the such that is minimized
![Page 13: Inverse Kinematics II and Motion Capture](https://reader036.vdocuments.site/reader036/viewer/2022062318/62a930f75d3cb8245a557173/html5/thumbnails/13.jpg)
Inverses
Find such that is minimized,
To force the unicity of the solution we force to be the one with minimum norm
![Page 14: Inverse Kinematics II and Motion Capture](https://reader036.vdocuments.site/reader036/viewer/2022062318/62a930f75d3cb8245a557173/html5/thumbnails/14.jpg)
Pseudo inverse works well in many cases but not near a singularity (singular values close to 0)
Make the constraint on the norm of soft, not hard as before
Singularities
this is always non singular if the damping factor is correctly chosen
the solution of this is the same as solving for
The Newton’s method with this damping step is known as the Damped Least Square method, or as the Levenberg-Marquardt algorithm.
![Page 15: Inverse Kinematics II and Motion Capture](https://reader036.vdocuments.site/reader036/viewer/2022062318/62a930f75d3cb8245a557173/html5/thumbnails/15.jpg)
Heuristic approaches
Cyclic Coordinate Descent is an alternating optimization approach where only one coordinate at a time is optimized.
Let be the estimate of the solution at iteration t
![Page 16: Inverse Kinematics II and Motion Capture](https://reader036.vdocuments.site/reader036/viewer/2022062318/62a930f75d3cb8245a557173/html5/thumbnails/16.jpg)
Heuristic approaches
Cyclic Coordinate Descent is an alternating optimization approach where only one coordinate at a time is optimized.
Let be the estimate of the solution at iteration t
![Page 17: Inverse Kinematics II and Motion Capture](https://reader036.vdocuments.site/reader036/viewer/2022062318/62a930f75d3cb8245a557173/html5/thumbnails/17.jpg)
Heuristic approaches
Cyclic Coordinate Descent is an alternating optimization approach where only one coordinate at a time is optimized.
Let be the estimate of the solution at iteration t
solving for a single twist is a very easy problem
![Page 18: Inverse Kinematics II and Motion Capture](https://reader036.vdocuments.site/reader036/viewer/2022062318/62a930f75d3cb8245a557173/html5/thumbnails/18.jpg)
Heuristic approaches
*
![Page 19: Inverse Kinematics II and Motion Capture](https://reader036.vdocuments.site/reader036/viewer/2022062318/62a930f75d3cb8245a557173/html5/thumbnails/19.jpg)
Heuristic approaches
![Page 20: Inverse Kinematics II and Motion Capture](https://reader036.vdocuments.site/reader036/viewer/2022062318/62a930f75d3cb8245a557173/html5/thumbnails/20.jpg)
Content
Inverse Kinematics
Kinematic Trees/Graphs
Pose Estimation/Motion Capture
![Page 21: Inverse Kinematics II and Motion Capture](https://reader036.vdocuments.site/reader036/viewer/2022062318/62a930f75d3cb8245a557173/html5/thumbnails/21.jpg)
Kinematic Tree
e2 e1
e3
Root
Root
A
B
C
D
E
F
G
I
H L
M
N
A
B
C
D E
F
G I H
L
M N
e1 e2
e3
a kinematic tree is a tree of rigid transformations
![Page 22: Inverse Kinematics II and Motion Capture](https://reader036.vdocuments.site/reader036/viewer/2022062318/62a930f75d3cb8245a557173/html5/thumbnails/22.jpg)
Kinematic Tree
e2 e1
e3
Root
a kinematic tree is a tree of rigid transformations
Root
A
B
C
D
E
F
G
I
H L
M
N
A
B
C
D E
F
G I H
L
M N
e1 e2
e3
each path in a kinematic tree from the root to any other node is a kinematic chain
Kinematic chain
![Page 23: Inverse Kinematics II and Motion Capture](https://reader036.vdocuments.site/reader036/viewer/2022062318/62a930f75d3cb8245a557173/html5/thumbnails/23.jpg)
Kinematic Tree
e2 e1
e3
Root
a kinematic tree is a tree of rigid transformations
Root
A
B
C
D
E
F
G
I
H L
M
N
A
B
C
D E
F
G I H
L
M N
e1 e2
e3
each path in a kinematic tree from the root to any other node is a kinematic chain
![Page 24: Inverse Kinematics II and Motion Capture](https://reader036.vdocuments.site/reader036/viewer/2022062318/62a930f75d3cb8245a557173/html5/thumbnails/24.jpg)
Kinematic Tree: Human Skeleton a kinematic tree is a tree of rigid transformations
The pelvis is typically denoted as the root of the tree (but this is only a convention)
each path in a kinematic tree from the root to any other node is a kinematic chain
![Page 25: Inverse Kinematics II and Motion Capture](https://reader036.vdocuments.site/reader036/viewer/2022062318/62a930f75d3cb8245a557173/html5/thumbnails/25.jpg)
An Inverse Kinematics Problem on a tree consists in finding the
configuration of the kinematic chain for which the distance between the end effectors and some pre-defined targets points are minimized it is an Inverse Kinematics Problem with Multiple Targets
Inverse Kinematics for Trees Given a kinematic tree
e2 e1
e3
Root
A
B C
D E
F
G I H
L
M N
Target q2
Target q3
Target q1
![Page 26: Inverse Kinematics II and Motion Capture](https://reader036.vdocuments.site/reader036/viewer/2022062318/62a930f75d3cb8245a557173/html5/thumbnails/26.jpg)
Inverse Kinematics for Trees Given a kinematic tree
Multi-objective optimization
e2 e1
e3
Root
A
B C
D E
F
G I H
L
M N
Target q2
Target q3
Target q1
Find x such that it simultaneously minimizes all the distances with the targets
It is difficult to find x satisfying all the minima
![Page 27: Inverse Kinematics II and Motion Capture](https://reader036.vdocuments.site/reader036/viewer/2022062318/62a930f75d3cb8245a557173/html5/thumbnails/27.jpg)
Inverse Kinematics for Trees Given a kinematic tree
e2 e1
e3
Root
A
B C
D E
F
G I H
L
M N
Target q2
Target q3
Target q1
Find the x which minimizes the sum of the squared errors
i.e. the mean solution
equivalent to
norm
![Page 28: Inverse Kinematics II and Motion Capture](https://reader036.vdocuments.site/reader036/viewer/2022062318/62a930f75d3cb8245a557173/html5/thumbnails/28.jpg)
Inverse Kinematics for Trees Given a kinematic tree
e2 e1
e3
Root
A
B C
D E
F
G I H
L
M N
Target q2
Target q3
Target q1
Find the x which minimizes the sum of the errors
i.e. the median solution
equivalent to
norm
![Page 29: Inverse Kinematics II and Motion Capture](https://reader036.vdocuments.site/reader036/viewer/2022062318/62a930f75d3cb8245a557173/html5/thumbnails/29.jpg)
Inverse Kinematics for Trees: Solution
is still a non-linear least square optimization problem
The norm case
Newton’s method or Levenberg-Marquardt
…
![Page 30: Inverse Kinematics II and Motion Capture](https://reader036.vdocuments.site/reader036/viewer/2022062318/62a930f75d3cb8245a557173/html5/thumbnails/30.jpg)
Kinematic Graph
e1
Root
Root
A
B
C
D
E
F
e1
a kinematic graph is a graph of rigid transformations (it contains cycles)
(parallel manipulators)
G
H
I
L
M
N
![Page 31: Inverse Kinematics II and Motion Capture](https://reader036.vdocuments.site/reader036/viewer/2022062318/62a930f75d3cb8245a557173/html5/thumbnails/31.jpg)
Inverse Kinematics for Graphs
e1 e2
e2 e1
Root
Root
A
B
C
D
E
F
G
H
I
L
M
N
e1
subject
Target q
Constrained optimization
Lagrange Multipliers or simply
![Page 32: Inverse Kinematics II and Motion Capture](https://reader036.vdocuments.site/reader036/viewer/2022062318/62a930f75d3cb8245a557173/html5/thumbnails/32.jpg)
Content
Inverse Kinematics
Kinematic Trees/Graphs
Pose Estimation/Motion Capture
![Page 33: Inverse Kinematics II and Motion Capture](https://reader036.vdocuments.site/reader036/viewer/2022062318/62a930f75d3cb8245a557173/html5/thumbnails/33.jpg)
Pose Estimation
Given an image depicting an articulated object, estimate the pose of that object
[Eichner 10]
determine all the DOF of that pose
![Page 34: Inverse Kinematics II and Motion Capture](https://reader036.vdocuments.site/reader036/viewer/2022062318/62a930f75d3cb8245a557173/html5/thumbnails/34.jpg)
Pose Estimation
Marker-based pose estimation Marker-less pose estimation
[Faceshift]
Pose estimation for multiple frames is called motion capture
![Page 35: Inverse Kinematics II and Motion Capture](https://reader036.vdocuments.site/reader036/viewer/2022062318/62a930f75d3cb8245a557173/html5/thumbnails/35.jpg)
Marker-based Pose Estimation
Infrared cameras with IR illuminator
IR reflective markers
Vicon system
![Page 36: Inverse Kinematics II and Motion Capture](https://reader036.vdocuments.site/reader036/viewer/2022062318/62a930f75d3cb8245a557173/html5/thumbnails/36.jpg)
Marker-based Pose Estimation
Images captured from the IR cameras
…
Kinematic tree with a set of end effectors
Set of 3D points (computed using triangulation) Correspondences
between 3D points and end effectors
Triangulate 2D points -> 3D points Compute correspondences between 3D points and end effectors
![Page 37: Inverse Kinematics II and Motion Capture](https://reader036.vdocuments.site/reader036/viewer/2022062318/62a930f75d3cb8245a557173/html5/thumbnails/37.jpg)
Marker-based Pose Estimation
Kinematic tree with a set of end effectors
Targets
…
Multi-objective optimization
…
![Page 38: Inverse Kinematics II and Motion Capture](https://reader036.vdocuments.site/reader036/viewer/2022062318/62a930f75d3cb8245a557173/html5/thumbnails/38.jpg)
Correspondences If correspondences between 3D points and targets are not known (or the heuristic used to
compute them is not accurate), these need to be estimated together with the pose
One of the possible solution to this problem is the ICP registration (ICP=Iterative Closest Points) [Besl & McKay 92]
But how can we formulate mathematically this problem?
![Page 39: Inverse Kinematics II and Motion Capture](https://reader036.vdocuments.site/reader036/viewer/2022062318/62a930f75d3cb8245a557173/html5/thumbnails/39.jpg)
Correspondences
Complete bi-partite graph (with weights ) (representing all the possible matching)
All the edges on this graph are possible matching candidates We need to find the actual matches (unknown to the problem)
Targets End effectors
![Page 40: Inverse Kinematics II and Motion Capture](https://reader036.vdocuments.site/reader036/viewer/2022062318/62a930f75d3cb8245a557173/html5/thumbnails/40.jpg)
Correspondences Target i in X is matched with end effector j in Y
cost of matching target i in X to end effector j in Y
• Binary Integer problem + Continuous problem • Alternating optimization approach
• Integer Programming (Hungarian algorithm) + Gradient Descent
![Page 41: Inverse Kinematics II and Motion Capture](https://reader036.vdocuments.site/reader036/viewer/2022062318/62a930f75d3cb8245a557173/html5/thumbnails/41.jpg)
Correspondences It might happen that some targets are missing or that some targets are wrong
Kinematic tree with a set of end effectors
Targets
Occluded, not visible Wrong estimate
One end effector does not have a corresponding target
One target does not have a corresponding end effector
![Page 42: Inverse Kinematics II and Motion Capture](https://reader036.vdocuments.site/reader036/viewer/2022062318/62a930f75d3cb8245a557173/html5/thumbnails/42.jpg)
Correspondences
virtual nodes
additional unknowns
![Page 43: Inverse Kinematics II and Motion Capture](https://reader036.vdocuments.site/reader036/viewer/2022062318/62a930f75d3cb8245a557173/html5/thumbnails/43.jpg)
Marker-Less Pose Estimation
Output: Input:
Scene Motion (angles and positions)
![Page 44: Inverse Kinematics II and Motion Capture](https://reader036.vdocuments.site/reader036/viewer/2022062318/62a930f75d3cb8245a557173/html5/thumbnails/44.jpg)
Marker-Less Pose Estimation
Comparison
Input video frames at time t
Pose Rendered images
Generative model
Comparison Error
Correct Pose
Minimum
Maximum likelihood
![Page 45: Inverse Kinematics II and Motion Capture](https://reader036.vdocuments.site/reader036/viewer/2022062318/62a930f75d3cb8245a557173/html5/thumbnails/45.jpg)
Generative Model
the motion of a vertex
the linear combination of all the motions that the vertex would undergo if rigidly attached to
every bone, one at a time Kinematic tree + 3D model
At a rigging pose
![Page 46: Inverse Kinematics II and Motion Capture](https://reader036.vdocuments.site/reader036/viewer/2022062318/62a930f75d3cb8245a557173/html5/thumbnails/46.jpg)
Marker-Less Pose Estimation
Comparison
Input video frames at time t
Pose Rendered images
Generative model
Comparison Error
Correct Pose
Minimum
Maximum likelihood
![Page 47: Inverse Kinematics II and Motion Capture](https://reader036.vdocuments.site/reader036/viewer/2022062318/62a930f75d3cb8245a557173/html5/thumbnails/47.jpg)
Comparison
Input video frame at time t
Pose
Generative model
= is a distance map in 3D between a point and S
![Page 48: Inverse Kinematics II and Motion Capture](https://reader036.vdocuments.site/reader036/viewer/2022062318/62a930f75d3cb8245a557173/html5/thumbnails/48.jpg)
Solution
= is a distance map in 3D
![Page 49: Inverse Kinematics II and Motion Capture](https://reader036.vdocuments.site/reader036/viewer/2022062318/62a930f75d3cb8245a557173/html5/thumbnails/49.jpg)
![Page 50: Inverse Kinematics II and Motion Capture](https://reader036.vdocuments.site/reader036/viewer/2022062318/62a930f75d3cb8245a557173/html5/thumbnails/50.jpg)
Final Examination
Homework: 10% each Oral Exam: 60%
(60%-40%) about 40min per person (20 Luca, 20 J.C.) (100%) about 1h per person (30 Luca, 30 J.C.)
each person has to answer different questions/exercises spanning the entire
program ETH will schedule the exam sometime in August
For the ones who didn’t do the homework who did not do all the homework who is not satisfied with the homework grades
and want to improve it
he/she can do the 100% exam to recover the missing points (inform in advance)