cs6670: computer vision - cornell university · •desiderata: good matches and good baselines 4...
TRANSCRIPT
![Page 1: CS6670: Computer Vision - Cornell University · •Desiderata: good matches AND good baselines 4 best neighboring views reference view. St. Peter 151 images 50 photographers Trevi](https://reader033.vdocuments.site/reader033/viewer/2022050415/5f8b3b54e1cd545a4c1b1b08/html5/thumbnails/1.jpg)
Lecture 23: Structure from motion
CS6670: Computer VisionNoah Snavely
![Page 2: CS6670: Computer Vision - Cornell University · •Desiderata: good matches AND good baselines 4 best neighboring views reference view. St. Peter 151 images 50 photographers Trevi](https://reader033.vdocuments.site/reader033/viewer/2022050415/5f8b3b54e1cd545a4c1b1b08/html5/thumbnails/2.jpg)
Lecture 24: Multi-view stereo
CS6670: Computer VisionNoah Snavely
![Page 3: CS6670: Computer Vision - Cornell University · •Desiderata: good matches AND good baselines 4 best neighboring views reference view. St. Peter 151 images 50 photographers Trevi](https://reader033.vdocuments.site/reader033/viewer/2022050415/5f8b3b54e1cd545a4c1b1b08/html5/thumbnails/3.jpg)
Readings
• Szeliski, Chapter 11.6
![Page 4: CS6670: Computer Vision - Cornell University · •Desiderata: good matches AND good baselines 4 best neighboring views reference view. St. Peter 151 images 50 photographers Trevi](https://reader033.vdocuments.site/reader033/viewer/2022050415/5f8b3b54e1cd545a4c1b1b08/html5/thumbnails/4.jpg)
Final project proposals
• Great job on the proposals!
• You can go ahead and get started – I will contact a few groups with additional feedback today
![Page 5: CS6670: Computer Vision - Cornell University · •Desiderata: good matches AND good baselines 4 best neighboring views reference view. St. Peter 151 images 50 photographers Trevi](https://reader033.vdocuments.site/reader033/viewer/2022050415/5f8b3b54e1cd545a4c1b1b08/html5/thumbnails/5.jpg)
Computer Vision
Geometry
Single view
Two view
Multi-view
Recognition
Computational photography
Light and optics
![Page 6: CS6670: Computer Vision - Cornell University · •Desiderata: good matches AND good baselines 4 best neighboring views reference view. St. Peter 151 images 50 photographers Trevi](https://reader033.vdocuments.site/reader033/viewer/2022050415/5f8b3b54e1cd545a4c1b1b08/html5/thumbnails/6.jpg)
Libration
![Page 7: CS6670: Computer Vision - Cornell University · •Desiderata: good matches AND good baselines 4 best neighboring views reference view. St. Peter 151 images 50 photographers Trevi](https://reader033.vdocuments.site/reader033/viewer/2022050415/5f8b3b54e1cd545a4c1b1b08/html5/thumbnails/7.jpg)
Why SFM might fail…
• Necker reversal
![Page 8: CS6670: Computer Vision - Cornell University · •Desiderata: good matches AND good baselines 4 best neighboring views reference view. St. Peter 151 images 50 photographers Trevi](https://reader033.vdocuments.site/reader033/viewer/2022050415/5f8b3b54e1cd545a4c1b1b08/html5/thumbnails/8.jpg)
SfM – Failure cases
• Necker reversal
![Page 9: CS6670: Computer Vision - Cornell University · •Desiderata: good matches AND good baselines 4 best neighboring views reference view. St. Peter 151 images 50 photographers Trevi](https://reader033.vdocuments.site/reader033/viewer/2022050415/5f8b3b54e1cd545a4c1b1b08/html5/thumbnails/9.jpg)
Structure from Motion – Failure cases
• Repetitive structures
![Page 10: CS6670: Computer Vision - Cornell University · •Desiderata: good matches AND good baselines 4 best neighboring views reference view. St. Peter 151 images 50 photographers Trevi](https://reader033.vdocuments.site/reader033/viewer/2022050415/5f8b3b54e1cd545a4c1b1b08/html5/thumbnails/10.jpg)
Multi-view stereo
Stereo
Multi-view stereo
![Page 11: CS6670: Computer Vision - Cornell University · •Desiderata: good matches AND good baselines 4 best neighboring views reference view. St. Peter 151 images 50 photographers Trevi](https://reader033.vdocuments.site/reader033/viewer/2022050415/5f8b3b54e1cd545a4c1b1b08/html5/thumbnails/11.jpg)
Multi-view Stereo
CMU’s 3D Room
Point Grey’s Bumblebee XB3
Point Grey’s ProFusion 25
![Page 12: CS6670: Computer Vision - Cornell University · •Desiderata: good matches AND good baselines 4 best neighboring views reference view. St. Peter 151 images 50 photographers Trevi](https://reader033.vdocuments.site/reader033/viewer/2022050415/5f8b3b54e1cd545a4c1b1b08/html5/thumbnails/12.jpg)
Multi-view Stereo
![Page 13: CS6670: Computer Vision - Cornell University · •Desiderata: good matches AND good baselines 4 best neighboring views reference view. St. Peter 151 images 50 photographers Trevi](https://reader033.vdocuments.site/reader033/viewer/2022050415/5f8b3b54e1cd545a4c1b1b08/html5/thumbnails/13.jpg)
Multi-view Stereo
Figures by Carlos Hernandez
Input: calibrated images from several viewpoints
Output: 3D object model
![Page 14: CS6670: Computer Vision - Cornell University · •Desiderata: good matches AND good baselines 4 best neighboring views reference view. St. Peter 151 images 50 photographers Trevi](https://reader033.vdocuments.site/reader033/viewer/2022050415/5f8b3b54e1cd545a4c1b1b08/html5/thumbnails/14.jpg)
Fua Narayanan, Rander, KanadeSeitz, Dyer
1995 1997 1998
Faugeras, Keriven
1998
Hernandez, Schmitt Pons, Keriven, Faugeras Furukawa, Ponce
2004 2005 2006
Goesele et al.
2007
Furukawa et al., 2010
![Page 15: CS6670: Computer Vision - Cornell University · •Desiderata: good matches AND good baselines 4 best neighboring views reference view. St. Peter 151 images 50 photographers Trevi](https://reader033.vdocuments.site/reader033/viewer/2022050415/5f8b3b54e1cd545a4c1b1b08/html5/thumbnails/15.jpg)
Stereo: another viewerror
depth
![Page 16: CS6670: Computer Vision - Cornell University · •Desiderata: good matches AND good baselines 4 best neighboring views reference view. St. Peter 151 images 50 photographers Trevi](https://reader033.vdocuments.site/reader033/viewer/2022050415/5f8b3b54e1cd545a4c1b1b08/html5/thumbnails/16.jpg)
width of
a pixel
Choosing the stereo baseline
What’s the optimal baseline?
– Too small: large depth error
– Too large: difficult search problem
Large Baseline Small Baseline
all of these
points project
to the same
pair of pixels
![Page 17: CS6670: Computer Vision - Cornell University · •Desiderata: good matches AND good baselines 4 best neighboring views reference view. St. Peter 151 images 50 photographers Trevi](https://reader033.vdocuments.site/reader033/viewer/2022050415/5f8b3b54e1cd545a4c1b1b08/html5/thumbnails/17.jpg)
The Effect of Baseline on Depth Estimation
![Page 18: CS6670: Computer Vision - Cornell University · •Desiderata: good matches AND good baselines 4 best neighboring views reference view. St. Peter 151 images 50 photographers Trevi](https://reader033.vdocuments.site/reader033/viewer/2022050415/5f8b3b54e1cd545a4c1b1b08/html5/thumbnails/18.jpg)
z
width of
a pixel
width of
a pixel
z
pixel matching score
![Page 19: CS6670: Computer Vision - Cornell University · •Desiderata: good matches AND good baselines 4 best neighboring views reference view. St. Peter 151 images 50 photographers Trevi](https://reader033.vdocuments.site/reader033/viewer/2022050415/5f8b3b54e1cd545a4c1b1b08/html5/thumbnails/19.jpg)
![Page 20: CS6670: Computer Vision - Cornell University · •Desiderata: good matches AND good baselines 4 best neighboring views reference view. St. Peter 151 images 50 photographers Trevi](https://reader033.vdocuments.site/reader033/viewer/2022050415/5f8b3b54e1cd545a4c1b1b08/html5/thumbnails/20.jpg)
Multibaseline Stereo
Basic Approach
– Choose a reference view
– Use your favorite stereo algorithm BUT• replace two-view SSD with SSSD over all baselines
Limitations
– Only gives a depth map (not an “object model”)
– Won’t work for widely distributed views:
![Page 21: CS6670: Computer Vision - Cornell University · •Desiderata: good matches AND good baselines 4 best neighboring views reference view. St. Peter 151 images 50 photographers Trevi](https://reader033.vdocuments.site/reader033/viewer/2022050415/5f8b3b54e1cd545a4c1b1b08/html5/thumbnails/21.jpg)
Some Solutions
• Match only nearby photos [Narayanan 98]
• Use NCC instead of SSD,
Ignore NCC values > threshold
[Hernandez & Schmitt 03]
Problem: visibility
![Page 22: CS6670: Computer Vision - Cornell University · •Desiderata: good matches AND good baselines 4 best neighboring views reference view. St. Peter 151 images 50 photographers Trevi](https://reader033.vdocuments.site/reader033/viewer/2022050415/5f8b3b54e1cd545a4c1b1b08/html5/thumbnails/22.jpg)
Popular matching scores• SSD (Sum Squared Distance)
• NCC (Normalized Cross Correlation)
– where
– what advantages might NCC have?
![Page 23: CS6670: Computer Vision - Cornell University · •Desiderata: good matches AND good baselines 4 best neighboring views reference view. St. Peter 151 images 50 photographers Trevi](https://reader033.vdocuments.site/reader033/viewer/2022050415/5f8b3b54e1cd545a4c1b1b08/html5/thumbnails/23.jpg)
Questions?
![Page 24: CS6670: Computer Vision - Cornell University · •Desiderata: good matches AND good baselines 4 best neighboring views reference view. St. Peter 151 images 50 photographers Trevi](https://reader033.vdocuments.site/reader033/viewer/2022050415/5f8b3b54e1cd545a4c1b1b08/html5/thumbnails/24.jpg)
Multi-view stereo from Internet Collections
[Goesele, Snavely, Curless, Hoppe, Seitz, ICCV 2007]
![Page 25: CS6670: Computer Vision - Cornell University · •Desiderata: good matches AND good baselines 4 best neighboring views reference view. St. Peter 151 images 50 photographers Trevi](https://reader033.vdocuments.site/reader033/viewer/2022050415/5f8b3b54e1cd545a4c1b1b08/html5/thumbnails/25.jpg)
• appearance variation
• resolution
• massive collections
Challenges
![Page 26: CS6670: Computer Vision - Cornell University · •Desiderata: good matches AND good baselines 4 best neighboring views reference view. St. Peter 151 images 50 photographers Trevi](https://reader033.vdocuments.site/reader033/viewer/2022050415/5f8b3b54e1cd545a4c1b1b08/html5/thumbnails/26.jpg)
206 Flickr images taken by 92 photographers
Law of Large Image Collections
![Page 27: CS6670: Computer Vision - Cornell University · •Desiderata: good matches AND good baselines 4 best neighboring views reference view. St. Peter 151 images 50 photographers Trevi](https://reader033.vdocuments.site/reader033/viewer/2022050415/5f8b3b54e1cd545a4c1b1b08/html5/thumbnails/27.jpg)
Local view selection• Automatically select neighboring views for each point in the image
• Desiderata: good matches AND good baselines
4 best neighboring views
reference view
![Page 28: CS6670: Computer Vision - Cornell University · •Desiderata: good matches AND good baselines 4 best neighboring views reference view. St. Peter 151 images 50 photographers Trevi](https://reader033.vdocuments.site/reader033/viewer/2022050415/5f8b3b54e1cd545a4c1b1b08/html5/thumbnails/28.jpg)
Local view selection• Automatically select neighboring views for each point in the image
• Desiderata: good matches AND good baselines
4 best neighboring views
reference view
![Page 29: CS6670: Computer Vision - Cornell University · •Desiderata: good matches AND good baselines 4 best neighboring views reference view. St. Peter 151 images 50 photographers Trevi](https://reader033.vdocuments.site/reader033/viewer/2022050415/5f8b3b54e1cd545a4c1b1b08/html5/thumbnails/29.jpg)
Local view selection• Automatically select neighboring views for each point in the image
• Desiderata: good matches AND good baselines
4 best neighboring views
reference view
![Page 30: CS6670: Computer Vision - Cornell University · •Desiderata: good matches AND good baselines 4 best neighboring views reference view. St. Peter 151 images 50 photographers Trevi](https://reader033.vdocuments.site/reader033/viewer/2022050415/5f8b3b54e1cd545a4c1b1b08/html5/thumbnails/30.jpg)
St. Peter
151 images
50 photographers
Trevi Fountain
106 images
51 photographers
Mt. Rushmore
160 images
60 photographers
Results
![Page 31: CS6670: Computer Vision - Cornell University · •Desiderata: good matches AND good baselines 4 best neighboring views reference view. St. Peter 151 images 50 photographers Trevi](https://reader033.vdocuments.site/reader033/viewer/2022050415/5f8b3b54e1cd545a4c1b1b08/html5/thumbnails/31.jpg)
Notre Dame de Paris
653 images
313 photographers
![Page 32: CS6670: Computer Vision - Cornell University · •Desiderata: good matches AND good baselines 4 best neighboring views reference view. St. Peter 151 images 50 photographers Trevi](https://reader033.vdocuments.site/reader033/viewer/2022050415/5f8b3b54e1cd545a4c1b1b08/html5/thumbnails/32.jpg)
![Page 33: CS6670: Computer Vision - Cornell University · •Desiderata: good matches AND good baselines 4 best neighboring views reference view. St. Peter 151 images 50 photographers Trevi](https://reader033.vdocuments.site/reader033/viewer/2022050415/5f8b3b54e1cd545a4c1b1b08/html5/thumbnails/33.jpg)
![Page 34: CS6670: Computer Vision - Cornell University · •Desiderata: good matches AND good baselines 4 best neighboring views reference view. St. Peter 151 images 50 photographers Trevi](https://reader033.vdocuments.site/reader033/viewer/2022050415/5f8b3b54e1cd545a4c1b1b08/html5/thumbnails/34.jpg)
129 Flickr images taken by 98 photographers
![Page 35: CS6670: Computer Vision - Cornell University · •Desiderata: good matches AND good baselines 4 best neighboring views reference view. St. Peter 151 images 50 photographers Trevi](https://reader033.vdocuments.site/reader033/viewer/2022050415/5f8b3b54e1cd545a4c1b1b08/html5/thumbnails/35.jpg)
merged model of Venus de Milo
![Page 36: CS6670: Computer Vision - Cornell University · •Desiderata: good matches AND good baselines 4 best neighboring views reference view. St. Peter 151 images 50 photographers Trevi](https://reader033.vdocuments.site/reader033/viewer/2022050415/5f8b3b54e1cd545a4c1b1b08/html5/thumbnails/36.jpg)
56 Flickr images taken by 8 photographers
![Page 37: CS6670: Computer Vision - Cornell University · •Desiderata: good matches AND good baselines 4 best neighboring views reference view. St. Peter 151 images 50 photographers Trevi](https://reader033.vdocuments.site/reader033/viewer/2022050415/5f8b3b54e1cd545a4c1b1b08/html5/thumbnails/37.jpg)
merged model of Pisa Cathedral
![Page 38: CS6670: Computer Vision - Cornell University · •Desiderata: good matches AND good baselines 4 best neighboring views reference view. St. Peter 151 images 50 photographers Trevi](https://reader033.vdocuments.site/reader033/viewer/2022050415/5f8b3b54e1cd545a4c1b1b08/html5/thumbnails/38.jpg)
Accuracy compared to laser scanned model:
90% of points within 0.25% of ground truth
![Page 39: CS6670: Computer Vision - Cornell University · •Desiderata: good matches AND good baselines 4 best neighboring views reference view. St. Peter 151 images 50 photographers Trevi](https://reader033.vdocuments.site/reader033/viewer/2022050415/5f8b3b54e1cd545a4c1b1b08/html5/thumbnails/39.jpg)
Demo