![Page 1: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/1.jpg)
Fundamental matrix
Let p be a point in left image, p’ in right image
Epipolar relation
• p maps to epipolar line l’
• p’ maps to epipolar line l
Epipolar mapping described by a 3x3 matrix F
l’l
p p’
![Page 2: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/2.jpg)
Fundamental matrix
This matrix F is called
• the “Essential Matrix”
– when image intrinsic parameters are known
• the “Fundamental Matrix”
– more generally (uncalibrated case)
Can solve for F from point correspondences
• Each (p, p’) pair gives one linear equation in entries of F
• F has 9 entries, but really only 7 or 8 degrees of freedom.
• With 8 points it is simple to solve for F, but it is also possible
with 7. See Marc Pollefey’s notes for a nice tutorial
![Page 3: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/3.jpg)
The scale of algorithm name quality
better
worse
RANSAC
Dynamic Programming
Essential and Fundamental Matrix
SIFT
Deep Learning
Neural Networks
Optical Flow
Hough Transform
![Page 4: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/4.jpg)
Today’s lecture
• Stereo Matching (Spare correspondence to Dense Correspondence)
• Optical Flow (Dense motion estimation)
![Page 5: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/5.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
Stereo Matching
![Page 6: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/6.jpg)
Stereo image rectification
![Page 7: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/7.jpg)
Stereo image rectification
• Reproject image planes onto a common plane parallel to the line between camera centers
• Pixel motion is horizontal after this transformation
• Two homographies (3x3 transform), one for each input image reprojection
C. Loop and Z. Zhang. Computing Rectifying Homographies for Stereo Vision. IEEE Conf. Computer Vision and Pattern Recognition, 1999.
![Page 8: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/8.jpg)
Rectification example
![Page 9: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/9.jpg)
The correspondence problem
• Epipolar geometry constrains our search, but we still have a difficult correspondence problem.
![Page 10: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/10.jpg)
Fundamental Matrix + Sparse correspondence
![Page 11: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/11.jpg)
Fundamental Matrix + Dense correspondence
![Page 12: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/12.jpg)
SIFT + Fundamental Matrix + RANSAC
Building Rome in a Day
By Sameer Agarwal, Yasutaka Furukawa, Noah Snavely, Ian Simon, Brian Curless, Steven M. Seitz, Richard Szeliski
Communications of the ACM, Vol. 54 No. 10, Pages 105-112
![Page 13: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/13.jpg)
Sparse to Dense Correspodence
Building Rome in a Day
By Sameer Agarwal, Yasutaka Furukawa, Noah Snavely, Ian Simon, Brian Curless, Steven M. Seitz, Richard Szeliski
Communications of the ACM, Vol. 54 No. 10, Pages 105-112
![Page 14: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/14.jpg)
Structure from motion (or SLAM)
• Given a set of corresponding points in two or more images, compute the camera parameters and the 3D point coordinates
Camera 1Camera 2 Camera 3
R1,t1 R2,t2R3,t3
? ? ? Slide credit:
Noah Snavely
?
![Page 15: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/15.jpg)
Structure from motion ambiguity
• If we scale the entire scene by some factor k and, at the same time, scale the camera matrices by the factor of 1/k, the projections of the scene points in the image remain exactly the same:
It is impossible to recover the absolute scale of the scene!
)(1
XPPXx kk
![Page 16: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/16.jpg)
How do we know the scale of image content?
![Page 17: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/17.jpg)
![Page 18: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/18.jpg)
![Page 19: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/19.jpg)
Bundle adjustment
• Non-linear method for refining structure and motion
• Minimizing reprojection error
2
1 1
,),(
m
i
n
j
jiijDE XPxXP
x1j
x2j
x3j
Xj
P1
P2
P3
P1Xj
P2Xj
P3Xj
![Page 20: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/20.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
Correspondence problem
Multiple match
hypotheses
satisfy epipolar
constraint, but
which is correct?
Figure from Gee & Cipolla 1999
![Page 21: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/21.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
Correspondence problem
• Beyond the hard constraint of epipolar geometry, there
are “soft” constraints to help identify corresponding points
• Similarity
• Uniqueness
• Ordering
• Disparity gradient
• To find matches in the image pair, we will assume
• Most scene points visible from both views
• Image regions for the matches are similar in appearance
![Page 22: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/22.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
Dense correspondence search
For each epipolar line
For each pixel / window in the left image
• compare with every pixel / window on same epipolar line
in right image• pick position with minimum match cost (e.g., SSD,
normalized correlation)
Adapted from Li Zhang
![Page 23: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/23.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
Matching cost
disparity
Left Right
scanline
Correspondence search with similarity constraint
• Slide a window along the right scanline and compare
contents of that window with the reference window in
the left image
• Matching cost: SSD or normalized correlation
![Page 24: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/24.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
Left Right
scanline
Correspondence search with similarity constraint
SSD
![Page 25: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/25.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
Left Right
scanline
Correspondence search with similarity constraint
Norm. corr
![Page 26: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/26.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
Correspondence problem
Source: Andrew Zisserman
Intensity
profiles
![Page 27: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/27.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
Correspondence problem
Neighborhoods of corresponding points are
similar in intensity patterns.
Source: Andrew Zisserman
![Page 28: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/28.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
Correlation-based window matching
![Page 29: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/29.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
Correlation-based window matching
![Page 30: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/30.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
Correlation-based window matching
![Page 31: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/31.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
Correlation-based window matching
![Page 32: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/32.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
Correlation-based window matching
???
Textureless regions are non-distinct; high ambiguity for matches.
![Page 33: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/33.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
Effect of window size
Source: Andrew Zisserman
![Page 34: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/34.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
W = 3 W = 20
Figures from Li Zhang
Want window large enough to have sufficient intensity
variation, yet small enough to contain only pixels with
about the same disparity.
Effect of window size
![Page 35: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/35.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
Left image Right image
![Page 36: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/36.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
Results with window search
Window-based matching
(best window size)
Ground truth
![Page 37: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/37.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
Better solutions
• Beyond individual correspondences to estimate
disparities:
• Optimize correspondence assignments jointly
• Scanline at a time (DP)
• Full 2D grid (graph cuts)
![Page 38: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/38.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
Stereo as energy minimization
• What defines a good stereo correspondence?
1. Match quality
• Want each pixel to find a good match in the other image
2. Smoothness
• If two pixels are adjacent, they should (usually) move about
the same amount
![Page 39: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/39.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
Stereo matching as energy minimization
I1I2 D
• Energy functions of this form can be minimized using
graph cuts
Y. Boykov, O. Veksler, and R. Zabih, Fast Approximate Energy Minimization via Graph Cuts, PAMI 2001
W1(i) W2(i+D(i)) D(i)
)(),,( smooth21data DEDIIEE
ji
jDiDE,neighbors
smooth )()( 2
21data ))(()( i
iDiWiWE
Source: Steve Seitz
![Page 40: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/40.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
Better results…
Graph cut methodBoykov et al., Fast Approximate Energy Minimization via Graph Cuts,
International Conference on Computer Vision, September 1999.
Ground truth
For the latest and greatest: http://www.middlebury.edu/stereo/
![Page 41: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/41.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
Challenges
• Low-contrast ; textureless image regions
• Occlusions
• Violations of brightness constancy (e.g., specular
reflections)
• Really large baselines (foreshortening and appearance
change)
• Camera calibration errors
![Page 42: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/42.jpg)
Active stereo with structured light
• Project “structured” light patterns onto the object
• Simplifies the correspondence problem
• Allows us to use only one camera
camera
projector
L. Zhang, B. Curless, and S. M. Seitz. Rapid Shape Acquisition Using Color Structured
Light and Multi-pass Dynamic Programming. 3DPVT 2002
![Page 43: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/43.jpg)
Kinect: Structured infrared light
http://bbzippo.wordpress.com/2010/11/28/kinect-in-infrared/
![Page 44: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/44.jpg)
iPhone X
![Page 45: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/45.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
3 minute break
![Page 46: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/46.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
Ninio, J. and Stevens, K. A. (2000) Variations on the Hermann grid: an extinction illusion.
Perception, 29, 1209-1217.
![Page 47: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/47.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
![Page 48: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/48.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
Computer Vision
Motion and Optical Flow
Many slides adapted from S. Seitz, R. Szeliski, M. Pollefeys, K. Grauman and others…
![Page 49: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/49.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
Video
• A video is a sequence of frames captured over time
• Now our image data is a function of space
(x, y) and time (t)
![Page 50: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/50.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
Motion and perceptual organization
Gestalt psychology
(Max Wertheimer,
1880-1943)
![Page 51: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/51.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
Motion and perceptual organization
• Sometimes, motion is the only cue
Gestalt psychology
(Max Wertheimer,
1880-1943)
![Page 52: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/52.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
Motion and perceptual organization
• Sometimes, motion is the only cue
![Page 53: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/53.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
Motion and perceptual organization
• Sometimes, motion is the only cue
![Page 54: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/54.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
Motion and perceptual organization
• Sometimes, motion is the only cue
![Page 55: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/55.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
Motion and perceptual organization
• Even “impoverished” motion data can evoke a strong
percept
![Page 56: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/56.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
Motion and perceptual organization
• Even “impoverished” motion data can evoke a strong
percept
![Page 57: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/57.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
Motion and perceptual organization
Experimental study of apparent behavior.
Fritz Heider & Marianne Simmel. 1944
![Page 58: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/58.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
Motion estimation: Optical flow
Will start by estimating motion of each pixel separately
Then will consider motion of entire image
Optic flow is the apparent motion of objects or surfaces
![Page 59: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/59.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
Problem definition: optical flow
How to estimate pixel motion from image I(x,y,t) to I(x,y,t+1) ?
• Solve pixel correspondence problem
– given a pixel in I(x,y,t), look for nearby pixels of the same color in I(x,y,t+1)
Key assumptions
• color constancy: a point in I(x,y,t) looks the same in I(x,y,t+1)
– For grayscale images, this is brightness constancy
• small motion: points do not move very far
This is called the optical flow problem
( , , )I x y t ( , , 1)I x y t
![Page 60: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/60.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
Optical flow constraints (grayscale images)
• Let’s look at these constraints more closely
• brightness constancy constraint (equation)
• small motion: (u and v are less than 1 pixel, or smooth)
Taylor series expansion of I:
( , , )I x y t ( , , 1)I x y t
( , ) ( , ) [higher order terms]I I
I x u y v I x y u vx y
( , )I I
I x y u vx y
( , , ) ( , , 1)I x y t I x u y v t
![Page 61: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/61.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
0 ( , , 1) ( , , )
( , , 1) ( , , )x y
I x u y v t I x y t
I x y t I u I v I x y t
Optical flow equation• Combining these two equations
(Short hand: 𝐼𝑥 =𝜕𝐼
𝜕𝑥
for t or t+1)
![Page 62: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/62.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
0 ( , , 1) ( , , )
( , , 1) ( , , )
[ ( , , 1) ( , , )]
,
x y
x y
t x y
t
I x u y v t I x y t
I x y t I u I v I x y t
I x y t I x y t I u I v
I I u I v
I I u v
Optical flow equation• Combining these two equations
(Short hand: 𝐼𝑥 =𝜕𝐼
𝜕𝑥
for t or t+1)
![Page 63: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/63.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
0 ( , , 1) ( , , )
( , , 1) ( , , )
[ ( , , 1) ( , , )]
,
x y
x y
t x y
t
I x u y v t I x y t
I x y t I u I v I x y t
I x y t I x y t I u I v
I I u I v
I I u v
Optical flow equation• Combining these two equations
In the limit as u and v go to zero, this becomes exact
Brightness constancy constraint equation
0x y tI u I v I
0 ,tI I u v
(Short hand: 𝐼𝑥 =𝜕𝐼
𝜕𝑥
for t or t+1)
![Page 64: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/64.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
How does this make sense?
• What do the static image gradients have to do with motion
estimation?
Brightness constancy constraint equation
0x y tI u I v I
![Page 65: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/65.jpg)
The brightness constancy constraint
• How many equations and unknowns per pixel?
The component of the motion perpendicular to the
gradient (i.e., parallel to the edge) cannot be measured
edge
(u,v)
(u’,v’)
gradient
(u+u’,v+v’)
If (u, v) satisfies the equation,
so does (u+u’, v+v’ ) if
•One equation (this is a scalar equation!), two unknowns (u,v)
0'v'uIT
Can we use this equation to recover image motion (u,v) at
each pixel?
0x y tI u I v I 0 ,tI I u v or
![Page 66: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/66.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
Aperture problem
![Page 67: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/67.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
Aperture problem
![Page 68: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/68.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
Aperture problem
![Page 69: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/69.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
The barber pole illusion
http://en.wikipedia.org/wiki/Barberpole_illusion
![Page 70: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/70.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
The barber pole illusion
http://en.wikipedia.org/wiki/Barberpole_illusion
![Page 71: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/71.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
Solving the ambiguity…
• How to get more equations for a pixel?
• Spatial coherence constraint
• Assume the pixel’s neighbors have the same (u,v)• If we use a 5x5 window, that gives us 25 equations per pixel
B. Lucas and T. Kanade. An iterative image registration technique with an application to stereo vision. In
Proceedings of the International Joint Conference on Artificial Intelligence, pp. 674–679, 1981.
![Page 72: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/72.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
• Least squares problem:
Solving the ambiguity…
![Page 73: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/73.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
Matching patches across images• Overconstrained linear system
The summations are over all pixels in the K x K window
Least squares solution for d given by
![Page 74: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/74.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
Conditions for solvabilityOptimal (u, v) satisfies Lucas-Kanade equation
Does this remind you of anything?
When is this solvable? I.e., what are good points to track?• ATA should be invertible
• ATA should not be too small due to noise
– eigenvalues 1 and 2 of ATA should not be too small
• ATA should be well-conditioned
– 1/ 2 should not be too large ( 1 = larger eigenvalue)
Criteria for Harris corner detector
![Page 75: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/75.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
Low texture region
– gradients have small magnitude
– small 1, small 2
![Page 76: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/76.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
Edge
– large gradients, all the same
– large 1, small 2
![Page 77: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/77.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
High textured region
– gradients are different, large magnitudes
– large 1, large 2
![Page 78: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/78.jpg)
The aperture problem resolved
Actual motion
![Page 79: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/79.jpg)
The aperture problem resolved
Perceived motion
![Page 80: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/80.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
Errors in Lucas-Kanade
• A point does not move like its neighbors
• Motion segmentation
• Brightness constancy does not hold
• Do exhaustive neighborhood search with normalized correlation -
tracking features – maybe SIFT – more later….
• The motion is large (larger than a pixel)
1. Not-linear: Iterative refinement
2. Local minima: coarse-to-fine estimation
![Page 81: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/81.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
Revisiting the small motion assumption
• Is this motion small enough?
• Probably not—it’s much larger than one pixel
• How might we solve this problem?
![Page 82: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/82.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
image 2image 1
Gaussian pyramid of image 1 Gaussian pyramid of image 2
image 2image 1 u=10 pixels
u=5 pixels
u=2.5 pixels
u=1.25 pixels
Coarse-to-fine optical flow estimation
![Page 83: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/83.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
image Iimage J
Gaussian pyramid of image 1 Gaussian pyramid of image 2
image 2image 1
Coarse-to-fine optical flow estimation
run iterative L-K
run iterative L-K
warp & upsample
.
.
.
![Page 84: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/84.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
Optical Flow Results
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
![Page 85: Fundamental matrixhays/compvision/lectures/11.pdfDeep Learning Neural Networks Optical Flow Hough Transform. Today’s lecture • Stereo Matching (Spare correspondence to Dense Correspondence)](https://reader033.vdocuments.site/reader033/viewer/2022050521/5fa4a58b852b9c685804cf32/html5/thumbnails/85.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
Optical Flow Results
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003