Feature Tracking and Optical Flow

Download Feature Tracking and Optical Flow

Post on 06-Feb-2016

43 views

Category:

Documents

0 download

DESCRIPTION

02/09/12. Feature Tracking and Optical Flow. Computer Vision CS 543 / ECE 549 University of Illinois Derek Hoiem. Many slides adapted from Lana Lazebnik, Silvio Saverse, who in turn adapted slides from Steve Seitz, Rick Szeliski, Martial Hebert, Mark Pollefeys, and others. Last class. - PowerPoint PPT Presentation

TRANSCRIPT

Slide 1The brightness constancy constraintHow many equations and unknowns per pixel?The component of the motion perpendicular to the gradient (i.e., parallel to the edge) cannot be measurededge(u,v)(u,v)gradient(u+u,v+v)If (u, v) satisfies the equation, so does (u+u, v+v ) if One equation (this is a scalar equation!), two unknowns (u,v)Can we use this equation to recover image motion (u,v) at each pixel?8A: u and v are unknown, 1 equationLast classInterest point detectors:Harris: detects corners (patches that have strong gradients in two orthogonal directions)DoG: detects peaks/troughs in location-scale space of a fine-scale Laplacian pyramidInterest point descriptorsSIFT (do read the paper)2This class: recovering motionFeature-trackingExtract visual features (corners, textured areas) and track them over multiple framesOptical flowRecover image motion at each pixel from spatio-temporal image brightness variations (optical flow)B. Lucas and T. Kanade. An iterative image registration technique with an application tostereo vision. In Proceedings of the International Joint Conference on Artificial Intelligence, pp. 674679, 1981.Two problems, one registration method3Feature trackingMany problems, such as structure from motion require matching pointsIf motion is small, tracking is an easy way to get them4Feature trackingChallengesFigure out which features can be trackedEfficiently track across framesSome points may change appearance over time (e.g., due to rotation, moving into shadows, etc.)Drift: small errors can accumulate as appearance model is updatedPoints may appear or disappear: need to be able to add/delete tracked points5Brightness Constancy Equation:Take Taylor expansion of I(x+u, y+v, t+1) at (x,y,t) to linearize the right side:The brightness constancy constraintI(x,y,t)I(x,y,t+1)So:Image derivative along xDifference over frames7Brightness constancy equation is equivalent to minimizing the squared error of I(t) and I(t+1)The aperture problemActual motion9The aperture problemPerceived motion10The barber pole illusionhttp://en.wikipedia.org/wiki/Barberpole_illusion11The barber pole illusionhttp://en.wikipedia.org/wiki/Barberpole_illusion12Solving the ambiguityHow to get more equations for a pixel?Spatial coherence constraint Assume the pixels neighbors have the same (u,v)If we use a 5x5 window, that gives us 25 equations per pixelB. Lucas and T. Kanade. An iterative image registration technique with an application to stereo vision. In Proceedings of the International Joint Conference on Artificial Intelligence, pp. 674679, 1981.13Least squares problem:Solving the ambiguity14Matching patches across imagesOverconstrained linear systemThe summations are over all pixels in the K x K windowLeast squares solution for d given by15Conditions for solvabilityOptimal (u, v) satisfies Lucas-Kanade equationDoes this remind you of anything?When is this solvable? I.e., what are good points to track?ATA should be invertible ATA should not be too small due to noiseeigenvalues 1 and 2 of ATA should not be too smallATA should be well-conditioned 1/ 2 should not be too large ( 1 = larger eigenvalue)Criteria for Harris corner detector 16Eigenvectors and eigenvalues of ATA relate to edge direction and magnitude The eigenvector associated with the larger eigenvalue points in the direction of fastest intensity changeThe other eigenvector is orthogonal to itM = ATA is the second moment matrix !(Harris corner detector)17Suppose M = vv^T . What are the eigenvectors and eigenvalues? Start by considering MvLow-texture region gradients have small magnitude small l1, small l218Edge gradients very large or very small large l1, small l219High-texture region gradients are different, large magnitudes large l1, large l220The aperture problem resolvedActual motion21The aperture problem resolvedPerceived motion22Dealing with larger movements: Iterative refinementInitialize (x,y) = (x,y)Compute (u,v) byShift window by (u, v): x=x+u; y=y+v;Recalculate ItRepeat steps 2-4 until small changeUse interpolation for subpixel values2nd moment matrix for feature patch in first imagedisplacementIt = I(x, y, t+1) - I(x, y, t) Original (x,y) positionimage Iimage JGaussian pyramid of image 1 (t)Gaussian pyramid of image 2 (t+1)image 2image 1Dealing with larger movements: coarse-to-fine registrationrun iterative L-Krun iterative L-Kupsample...24Shi-Tomasi feature trackerFind good features using eigenvalues of second-moment matrix (e.g., Harris detector or threshold on the smallest eigenvalue)Key idea: good features to track are the ones whose motion can be estimated reliablyTrack from frame to frame with Lucas-KanadeThis amounts to assuming a translation model for frame-to-frame feature movementCheck consistency of tracks by affine registration to the first observed instance of the featureAffine model is more accurate for larger displacementsComparing to the first frame helps to minimize driftJ. Shi and C. Tomasi. Good Features to Track. CVPR 1994. 25Tracking exampleJ. Shi and C. Tomasi. Good Features to Track. CVPR 1994. 26Summary of KLT trackingFind a good point to track (harris corner)Use intensity second moment matrix and difference across frames to find displacementIterate and use coarse-to-fine search to deal with larger movementsWhen creating long tracks, check appearance of registered patch against appearance of initial patch to find points that have driftedImplementation issuesWindow sizeSmall window more sensitive to noise and may miss larger motions (without pyramid)Large window more likely to cross an occlusion boundary (and its slower)15x15 to 31x31 seems typicalWeighting the windowCommon to apply weights so that center matters more (e.g., with Gaussian)Picture courtesy of Selim Temizer - Learning and Intelligent Systems (LIS) Group, MIT Optical flowVector field function of the spatio-temporal image brightness variations Motion and perceptual organizationSometimes, motion is the only cue30Motion and perceptual organizationEven impoverished motion data can evoke a strong perceptG. Johansson, Visual Perception of Biological Motion and a Model For Its Analysis", Perception and Psychophysics 14, 201-211, 1973.31Motion and perceptual organizationEven impoverished motion data can evoke a strong percept G. Johansson, Visual Perception of Biological Motion and a Model For Its Analysis", Perception and Psychophysics 14, 201-211, 1973.32Uses of motionEstimating 3D structureSegmenting objects based on motion cuesLearning and tracking dynamical modelsRecognizing events and activitiesImproving video quality (motion stabilization)33Motion fieldThe motion field is the projection of the 3D scene motion into the imageWhat would the motion field of a non-rotating ball moving towards the camera look like?34Optical flowDefinition: optical flow is the apparent motion of brightness patterns in the imageIdeally, optical flow would be the same as the motion fieldHave to be careful: apparent motion can be caused by lighting changes without any actual motionThink of a uniform rotating sphere under fixed lighting vs. a stationary sphere under moving illumination35Lucas-Kanade Optical FlowSame as Lucas-Kanade feature tracking, but for each pixelAs we saw, works better for textured pixelsOperations can be done one frame at a time, rather than pixel by pixelEfficientMulti-resolution Lucas Kanade Algorithm38Iterative RefinementIterative Lukas-Kanade AlgorithmEstimate displacement at each pixel by solving Lucas-Kanade equationsWarp I(t) towards I(t+1) using the estimated flow field- Basically, just interpolationRepeat until convergence* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003image Iimage JGaussian pyramid of image 1 (t)Gaussian pyramid of image 2 (t+1)image 2image 1Coarse-to-fine optical flow estimationrun iterative L-Krun iterative L-Kwarp & upsample ...39image Iimage HGaussian pyramid of image 1Gaussian pyramid of image 2image 2image 1u=10 pixelsu=5 pixelsu=2.5 pixelsu=1.25 pixelsCoarse-to-fine optical flow estimation40Example* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003Multi-resolution registration* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003Optical Flow Results* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003Optical Flow Results* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003Errors in Lucas-KanadeThe motion is largePossible Fix: Keypoint matchingA point does not move like its neighborsPossible Fix: Region-based matchingBrightness constancy does not holdPossible Fix: Gradient constancy45Iterative refinement:Estimate velocity at each pixel using one iteration of Lucas and Kanade estimationWarp one image toward the other using the estimated flow fieldRefine estimate by repeating the processState-of-the-art optical flowStart with something similar to Lucas-Kanade+ gradient constancy+ energy minimization with smoothing term+ region matching+ keypoint matching (long-range)Large displacement optical flow, Brox et al., CVPR 2009Region-based+Pixel-based+Keypoint-basedStereo vs. Optical FlowSimilar dense matching proceduresWhy dont we typically use epipolar constraints for optical flow?B. Lucas and T. Kanade. An iterative image registration technique with an application tostereo vision. In Proceedings of the International Joint Conference on Artificial Intelligence, pp. 674679, 1981.SummaryMajor contributions from Lucas, Tomasi, KanadeTracking feature pointsOptical flowStereo (later)Structure from motion (later)Key ideasBy assuming brightness constancy, truncated Taylor expansion leads to simple and fast patch matching across framesCoarse-to-fine registrationNext weekHW 1 due TuesdayIm out of town Friday to SundayAmin Sadeghi has special office hours on Friday at 5pm (see web site)Object/image alignment

Recommended

View more >