CSE 185 Introduction to Computer Vision Feature Tracking and Optical Flow.
Post on 16-Jan-2016
CSE 185 Introduction to Computer VisionFeature Tracking and Optical Flow
Motion estimationFeature-trackingExtract visual features (corners, textured areas) and track them over multiple frames
Optical flowRecover image motion at each pixel from spatio-temporal image brightness variations (optical flow)
Two problems, one registration method
Feature trackingMany problems, such as structure from motion require matching pointsIf motion is small, tracking is an easy way to get them
ChallengesFigure out which features can be trackedEfficiently track across framesSome points may change appearance over time (e.g., due to rotation, moving into shadows, etc.)Drift: small errors can accumulate as appearance model is updatedPoints may appear or disappear: need to be able to add/delete tracked points
Feature trackingGiven two subsequent frames, estimate the point translationKey assumptions of Lucas-Kanade TrackerBrightness constancy: projection of the same point looks the same in every frameSmall motion: points do not move very farSpatial coherence: points move like their neighborsI(x,y,t)I(x,y,t+1)
Brightness Constancy Equation:Take Taylor expansion of I(x+u, y+v, t+1) at (x,y,t) to linearize the right side:Brightness constancyI(x,y,t)I(x,y,t+1)Image derivative along xDifference over frames
How does this make sense?
What do the static image gradients have to do with motion estimation?
Computing gradients in X-Y-Ttt+1ii+1jj+1timexylikewise for Iy and It
Brightness constancyHow many equations and unknowns per pixel?The component of the motion perpendicular to the gradient (i.e., parallel to the edge) cannot be measurededge(u,v)(u,v)gradient(u+u,v+v)If (u, v) satisfies the equation, so does (u+u, v+v ) if One equation (this is a scalar equation!), two unknowns (u,v)Can we use this equation to recover image motion (u,v) at each pixel?
The aperture problemActual motion
The aperture problemPerceived motion
The aperture problemhttp://en.wikipedia.org/wiki/Motion_perception#The_aperture_problemThegratingappears to be moving down and to the right,perpendicularto the orientation of the bars. But it could be moving in many other directions, such as only down, or only to the right. It is not possible to determine unless the ends of the bars become visible in the aperture
The barber pole illusionhttp://en.wikipedia.org/wiki/Barberpole_illusionThis visual illusion occurs when a diagonally striped pole is rotated around itsvertical axis (horizontally), it appears as though the stripes are moving in the direction of its vertical axis (downwards in the case of the animation to the right) rather than around it.The barber pole turns in place on its vertical axis, but the stripes appear to move upwards rather than turning with the pole
Solving the ambiguityHow to get more equations for a pixel?Spatial coherence constraint Assume the pixels neighbors have the same (u,v)If we use a 5x5 window, that gives us 25 equations per pixelB. Lucas and T. Kanade. An iterative image registration technique with an application to stereo vision. In Proceedings of the International Joint Conference on Artificial Intelligence, pp. 674679, 1981.
Least squares problem:Solving the ambiguity
Matching patches Overconstrained linear systemThe summations are over all pixels in the K x K windowLeast squares solution for d given by
Conditions for solvabilityOptimal (u, v) satisfies Lucas-Kanade equationDoes this remind you of anything?When is this solvable? I.e., what are good points to track?ATA should be invertible ATA should not be too small due to noiseeigenvalues 1 and 2 of ATA should not be too smallATA should be well-conditioned 1/ 2 should not be too large ( 1 = larger eigenvalue)
Criteria for Harris corner detector
Pseudo inversepseudo inverse, left inverse
Eigenvectors and eigenvalues of ATA relate to edge direction and magnitude The eigenvector associated with the larger eigenvalue points in the direction of fastest intensity changeThe other eigenvector is orthogonal to itM = ATA is the second moment matrix !(Harris corner detector)
Low-texture region gradients have small magnitude small l1, small l2
Edge gradients very large or very small large l1, small l2
High-texture region gradients are different, large magnitudes large l1, large l2
The aperture problem resolvedActual motion
The aperture problem resolvedPerceived motion
Larger motion: Iterative refinementInitialize (x,y) = (x,y)Compute (u,v) by
Shift window by (u, v): x=x+u; y=y+v;Recalculate ItRepeat steps 2-4 until small changeUse interpolation for subpixel values
2nd moment matrix for feature patch in first imagedisplacementIt = I(x, y, t+1) - I(x, y, t) Original (x,y) position
Coarse-to-fine registrationrun iterative L-KDealing with large motion
Shi-Tomasi feature trackerFind good features using eigenvalues of second-moment matrix (e.g., Harris detector or threshold on the smallest eigenvalue)Key idea: good features to track are the ones whose motion can be estimated reliablyTrack from frame to frame with Lucas-KanadeThis amounts to assuming a translation model for frame-to-frame feature movementCheck consistency of tracks by affine registration to the first observed instance of the featureAffine model is more accurate for larger displacementsComparing to the first frame helps to minimize driftJ. Shi and C. Tomasi. Good Features to Track. CVPR 1994.
Tracking exampleJ. Shi and C. Tomasi. Good Features to Track. CVPR 1994.
Summary of KLT trackingFind a good point to track (Harris corner)
Use intensity second moment matrix and difference across frames to find displacement
Iterate and use coarse-to-fine search to deal with larger movements
When creating long tracks, check appearance of registered patch against appearance of initial patch to find points that have drifted
Implementation issuesWindow sizeSmall window more sensitive to noise and may miss larger motions (without pyramid)Large window more likely to cross an occlusion boundary (and its slower)15x15 to 31x31 seems typical
Weighting the windowCommon to apply weights so that center matters more (e.g., with Gaussian)
Vector field function of the spatio-temporal image brightness variations Optical flow
Motion and perceptual organizationSometimes, motion is the only cue
Motion and perceptual organizationEven impoverished motion data can evoke a strong perceptG. Johansson, Visual Perception of Biological Motion and a Model For Its Analysis", Perception and Psychophysics 14, 201-211, 1973.
Motion and perceptual organizationEven impoverished motion data can evoke a strong percept G. Johansson, Visual Perception of Biological Motion and a Model For Its Analysis", Perception and Psychophysics 14, 201-211, 1973.
Uses of motionEstimating 3D structureSegmenting objects based on motion cuesLearning and tracking dynamical modelsRecognizing events and activitiesImproving video quality (motion stabilization)
Motion fieldThe motion field is the projection of the 3D scene motion into the imageWhat would the motion field of a non-rotating ball moving towards the camera look like?
Optical flowDefinition: optical flow is the apparent motion of brightness patterns in the imageIdeally, optical flow would be the same as the motion fieldHave to be careful: apparent motion can be caused by lighting changes without any actual motionThink of a uniform rotating sphere under fixed lighting vs. a stationary sphere under moving illumination
Lucas-Kanade optical flowSame as Lucas-Kanade feature tracking, but for each pixelAs we saw, works better for textured pixelsOperations can be done one frame at a time, rather than pixel by pixelEfficient
Multi-scale Lucas Kanade Algorithm
*Iterative refinementIterative Lukas-Kanade AlgorithmEstimate displacement at each pixel by solving Lucas-Kanade equationsWarp I(t) towards I(t+1) using the estimated flow field- Basically, just interpolationRepeat until convergence
Coarse-to-fine optical flow estimationrun iterative L-K
Coarse-to-fine optical flow estimation
Optical flow results
Optical flow results
Errors in Lucas-KanadeThe motion is largePossible Fix: keypoint matchingA point does not move like its neighborsPossible Fix: region-based matchingBrightness constancy does not holdPossible Fix: gradient constancy
State-of-the-art optical flowStart with something similar to Lucas-Kanade+ gradient constancy+ energy minimization with smoothing term+ region matching+ keypoint matching (long-range)Large displacement optical flow, Brox et al., CVPR 2009Region-based+Pixel-based+Keypoint-based
Stereo vs. Optical Flow
Similar dense matching procedures
Why dont we typically use epipolar constraints for optical flow?
B. Lucas and T. Kanade. An iterative image registration technique with an application tostereo vision. In Proceedings of the International Joint Conference on Artificial Intelligence, pp. 674679, 1981.
SummaryMajor contributions from Lucas, Tomasi, KanadeTracking feature pointsOptical flowKey ideasBy assuming brightness constancy, truncated Taylor expansion leads to simple and fast patch matching across framesCoarse-to-fine registration
******Brightness constancy equation is equivalent to minimizing the squared error of I(t) and I(t+1)http://en.wikipedia.org/wiki/Taylor_seriesIn mathematics, a Taylor series is a representation of a function as an infinite sum of terms that are calculated from the values of the function's derivatives at a single point. http://en.wikipedia.org/wiki/Taylor_series#Taylor_series_in_several_variables**A: u and v are unknown, 1 equation********http://en.wikipedia.org/wiki/Invertible_matrix*Suppose M = vv^T . What are the eigenvectors and eigenvalues? Start by considering Mv*****************Iterative refinement:Estimate velocity at each pixel using one iteration of Lucas and Kanade estimationWarp one image toward the other using the estimated flow fieldRefine estimate by repeating the process