tracking and modeling non-rigid objects with rank constraints

Tracking and Modeling Tracking and Modeling Non-Rigid Objects with Non-Rigid Objects with Rank ConstraintsRank Constraints

L. Torresani, D. Yang, E. L. Torresani, D. Yang, E. Alexander, and C. BreglerAlexander, and C. Bregler

Presentation by:

Jeremy Weinsier

MotivationMotivation

Can a non-rigid object’s shape and Can a non-rigid object’s shape and pose be determined from a single pose be determined from a single video sequence without any video sequence without any information about the particular information about the particular scene?scene?

In other words, can we determine a 3D In other words, can we determine a 3D model for the shape of the object model for the shape of the object without any prior knowledge?without any prior knowledge?

BackgroundBackground

In a previous paper, the authors In a previous paper, the authors proved that 2D point tracks from a proved that 2D point tracks from a single view (no stereo) are enough to single view (no stereo) are enough to recover the non-rigid motion as well recover the non-rigid motion as well as the structure of an object by as the structure of an object by exploiting low rank constraints.exploiting low rank constraints.

This procedure takes that idea one This procedure takes that idea one step further: non-rigid motion and step further: non-rigid motion and structure can now be recovered from structure can now be recovered from a single view without any point a single view without any point tracks.tracks.

Rank ConstraintsRank Constraints

Optical flow data is read into two F Optical flow data is read into two F x P matrices, U and V.x P matrices, U and V.

Let Let

If W describes 3D rigid motion, then If W describes 3D rigid motion, then there is an upper bound on the rank of there is an upper bound on the rank of W.W.

UW

V


The particular bound is based on the The particular bound is based on the assumed motion model. For assumed motion model. For example,example,

Orthographic: rOrthographic: r≤4≤4Projective: r≤8Projective: r≤8

These rank constraints are derived These rank constraints are derived from the fact that W can be factored from the fact that W can be factored into Q and S, which represent pose into Q and S, which represent pose and shape.and shape.


Non-rigid motion can also be Non-rigid motion can also be factored into two matrices, but factored into two matrices, but the rank of these matrices will be the rank of these matrices will be higher than that those in the rigid higher than that those in the rigid case.case.

Shape will be represented by K Shape will be represented by K basis shapes, each described by a basis shapes, each described by a 3xP matrix.3xP matrix.


Assuming weak perspective Assuming weak perspective projection:projection:


Eliminate T by subtracting the Eliminate T by subtracting the mean of all 2D points and rewrite mean of all 2D points and rewrite as matrix multiplication:as matrix multiplication:


Combine all individual equations Combine all individual equations into one large matrix:into one large matrix:


Since Q is size 2Fx3K and B is size Since Q is size 2Fx3K and B is size 3KxP, the rank of W, r3KxP, the rank of W, r≤3K.≤3K.

This assumes that the sequence is This assumes that the sequence is free of noise and can be used as free of noise and can be used as the bound on the rank of W in the the bound on the rank of W in the case of non-rigid motion.case of non-rigid motion.

Basis FlowBasis Flow

Assuming rank rAssuming rank r, each column of W can , each column of W can be modeled as a linear combination of r be modeled as a linear combination of r “basis-tracks,” denoted Q. “basis-tracks,” denoted Q.

Q is estimated by removing all but the Q is estimated by removing all but the most reliable tracks in W and then most reliable tracks in W and then computing its SVD. The first r computing its SVD. The first r eigenvectors of the SVD are taken as eigenvectors of the SVD are taken as the first estimate of the basis-tracks.the first estimate of the basis-tracks.

This eigenbase is then applied to the This eigenbase is then applied to the entire W matrix to estimate all P tracks.entire W matrix to estimate all P tracks.


The Lucas Kanade optical flow The Lucas Kanade optical flow equation:equation:


Since the entire sequence is assumed to Since the entire sequence is assumed to have a single image template, this have a single image template, this equation could be rewritten as:equation could be rewritten as:

Where C,D,E are PxP diagonal matrices Where C,D,E are PxP diagonal matrices containing c,d,e values for all patches containing c,d,e values for all patches and F,G are FxP matrices containing f and F,G are FxP matrices containing f and g values for each patch and frame.and g values for each patch and frame.


Now split Q into two matrices, one Now split Q into two matrices, one containing the even rows of Q and containing the even rows of Q and the other containing the odd rows the other containing the odd rows of Q. Since Q is a basis for W:of Q. Since Q is a basis for W:


The determined values are used as The determined values are used as initial estimates of the actual values.initial estimates of the actual values.

The image is warped according to The image is warped according to these values and the process these values and the process iterates to refine these values until iterates to refine these values until they are close to the actual values.they are close to the actual values.

OcclusionOcclusion

By vectorizing the B matrix into a By vectorizing the B matrix into a Pr dimensional vector b, occlusion Pr dimensional vector b, occlusion can be handled:can be handled:

Missing entries due to occlusion or Missing entries due to occlusion or mistracking are removed from the mistracking are removed from the matrices on both sides.matrices on both sides.

OcclusionOcclusion

If enough points are still visible, the If enough points are still visible, the system is still overconstrained system is still overconstrained and can be solved.and can be solved.

The displacement of the missing The displacement of the missing points can later be estimated points can later be estimated using Q*B.using Q*B.

3D Reconstruction3D Reconstruction

The factorization of W into Q and B The factorization of W into Q and B is not unique. Other is not unique. Other factorizations can be found by:factorizations can be found by:

Q does not initially comply with its Q does not initially comply with its necessary structure:necessary structure:

3D Reconstruction3D Reconstruction

Tomasi-Kanade suggest using a Tomasi-Kanade suggest using a linear approximation scheme in linear approximation scheme in the rigid case. The sub-blocks of the rigid case. The sub-blocks of Q are treated as rotation Q are treated as rotation matrices.matrices.

A similar approach can be used in A similar approach can be used in the non-rigid case, but a second the non-rigid case, but a second factorization step is necessary.factorization step is necessary.

Sub-Block Sub-Block FactorizationFactorization

Sub-Block Sub-Block FactorizationFactorization

Since QSince Qtt now has a rank of one, a now has a rank of one, a non-linear optimization method is non-linear optimization method is used to find an invertible A that used to find an invertible A that orthonormalizes all of the sub-orthonormalizes all of the sub-blocks.blocks.

This produces a matrix with scaled This produces a matrix with scaled rotation matrices as its sub-blocks.rotation matrices as its sub-blocks.

LimitationsLimitations

The limitation of sub-block factorization The limitation of sub-block factorization is in the noisy and ambiguous cases. is in the noisy and ambiguous cases. The second and higher eigenvalues The second and higher eigenvalues will not disappear in some of the sub-will not disappear in some of the sub-blocks. This leads to bad rank-1 blocks. This leads to bad rank-1 approximations of Rapproximations of Rtt..

The alternative is an iterative technique The alternative is an iterative technique that solves the system directly.that solves the system directly.

Iterative OptimizationIterative Optimization

The first step in this method is the The first step in this method is the same as in sub-block same as in sub-block factorization. R is factored into factorization. R is factored into QQrigrig and B and Brigrig. Q. Qrigrig is then is then reorganized into a matrix with reorganized into a matrix with sub-blocks that are weak sub-blocks that are weak perspective rotation matrices.perspective rotation matrices.


QQrigrig is then used as an initial guess of the is then used as an initial guess of the pose of the non-rigid object.pose of the non-rigid object.

1.1. Linear least squares is used to find B.Linear least squares is used to find B.

2.2. Linear least squares is used to find L.Linear least squares is used to find L.

3.3. Solve for R, such that RSolve for R, such that Rtt fit the fit the equation: equation:


To solve for R such that RTo solve for R such that Rtt remain remain rotation matrices, Rrotation matrices, Rtt can be can be parameterized with exponential parameterized with exponential coordinates.coordinates.


Linearizing the previous equation Linearizing the previous equation around the previous estimate leads around the previous estimate leads to:to:

These steps are iterated until These steps are iterated until convergence. Missing entries are convergence. Missing entries are handled as before.handled as before.

Multi-View InputMulti-View Input

If M cameras are used, the input If M cameras are used, the input matrix W is enlarged to size 2FW matrix W is enlarged to size 2FW x P. x P.

Multi-View InputMulti-View Input

The process is the same as before, The process is the same as before, but now there is an additional but now there is an additional constraint that the deformation constraint that the deformation must be the same for every must be the same for every camera in every frame.camera in every frame.

Basis ShapesBasis Shapes

The number of basis shapes used The number of basis shapes used has an effect on the error in the has an effect on the error in the result. Here is some sample K vs. result. Here is some sample K vs. error data:error data:

Basis ShapesBasis Shapes

The number of necessary basis The number of necessary basis shapes is not currently an shapes is not currently an automatic calculation.automatic calculation.

A simple solution to this problem is A simple solution to this problem is to continuously increase K until to continuously increase K until the error is below a certain the error is below a certain threshold.threshold.

ResultsResults

tracking and modeling non-rigid objects with rank constraints

Documents