3d vision the university of oxford andrew glennerster physiology andrew fitzgibbon engineering
TRANSCRIPT
![Page 1: 3D Vision The University of Oxford Andrew Glennerster Physiology Andrew Fitzgibbon Engineering](https://reader035.vdocuments.site/reader035/viewer/2022081520/56649dca5503460f94ac0e90/html5/thumbnails/1.jpg)
3D Vision
The University of Oxford
Andrew Glennerster Physiology
Andrew Fitzgibbon Engineering
![Page 2: 3D Vision The University of Oxford Andrew Glennerster Physiology Andrew Fitzgibbon Engineering](https://reader035.vdocuments.site/reader035/viewer/2022081520/56649dca5503460f94ac0e90/html5/thumbnails/2.jpg)
Vision
A fundamental biological system
An attractive engineering problem
How are these related?
![Page 3: 3D Vision The University of Oxford Andrew Glennerster Physiology Andrew Fitzgibbon Engineering](https://reader035.vdocuments.site/reader035/viewer/2022081520/56649dca5503460f94ac0e90/html5/thumbnails/3.jpg)
A 3D Visual Task
![Page 4: 3D Vision The University of Oxford Andrew Glennerster Physiology Andrew Fitzgibbon Engineering](https://reader035.vdocuments.site/reader035/viewer/2022081520/56649dca5503460f94ac0e90/html5/thumbnails/4.jpg)
TOC
• Computer vision state of the art• Biology state of the art
– an implausible model?
• Open questions
![Page 5: 3D Vision The University of Oxford Andrew Glennerster Physiology Andrew Fitzgibbon Engineering](https://reader035.vdocuments.site/reader035/viewer/2022081520/56649dca5503460f94ac0e90/html5/thumbnails/5.jpg)
Artificial 3D Vision
![Page 6: 3D Vision The University of Oxford Andrew Glennerster Physiology Andrew Fitzgibbon Engineering](https://reader035.vdocuments.site/reader035/viewer/2022081520/56649dca5503460f94ac0e90/html5/thumbnails/6.jpg)
Measurements
![Page 7: 3D Vision The University of Oxford Andrew Glennerster Physiology Andrew Fitzgibbon Engineering](https://reader035.vdocuments.site/reader035/viewer/2022081520/56649dca5503460f94ac0e90/html5/thumbnails/7.jpg)
3D Structure
![Page 8: 3D Vision The University of Oxford Andrew Glennerster Physiology Andrew Fitzgibbon Engineering](https://reader035.vdocuments.site/reader035/viewer/2022081520/56649dca5503460f94ac0e90/html5/thumbnails/8.jpg)
Application: 3D Video Editing
![Page 9: 3D Vision The University of Oxford Andrew Glennerster Physiology Andrew Fitzgibbon Engineering](https://reader035.vdocuments.site/reader035/viewer/2022081520/56649dca5503460f94ac0e90/html5/thumbnails/9.jpg)
Application: 3D Video Editing
![Page 10: 3D Vision The University of Oxford Andrew Glennerster Physiology Andrew Fitzgibbon Engineering](https://reader035.vdocuments.site/reader035/viewer/2022081520/56649dca5503460f94ac0e90/html5/thumbnails/10.jpg)
“Computer Vision” Summary
• Single representation suits all– pointing– navigation
• Built on– algebra– geometry– statistics
• Really works on real data
![Page 11: 3D Vision The University of Oxford Andrew Glennerster Physiology Andrew Fitzgibbon Engineering](https://reader035.vdocuments.site/reader035/viewer/2022081520/56649dca5503460f94ac0e90/html5/thumbnails/11.jpg)
Biological systems can perform 3D tasksusing visual information
• Pointing to remembered objects• Ants homing• Spiders turning to view prey
![Page 12: 3D Vision The University of Oxford Andrew Glennerster Physiology Andrew Fitzgibbon Engineering](https://reader035.vdocuments.site/reader035/viewer/2022081520/56649dca5503460f94ac0e90/html5/thumbnails/12.jpg)
Computer vision is an unlikely model for animal vision
• A “likely” model needs topredict experimental databe neurally plausible
• We don’t know how to multiply matrices in neurons
• CV is open loop– animate systems characterized by feedback
• 3D not necessary for many tasks
![Page 13: 3D Vision The University of Oxford Andrew Glennerster Physiology Andrew Fitzgibbon Engineering](https://reader035.vdocuments.site/reader035/viewer/2022081520/56649dca5503460f94ac0e90/html5/thumbnails/13.jpg)
What is the incumbent model?
Posterior parietalcortex
Eye-centredcoordinates
Head-centredcoordinates
Body-centredcoordinates
World-centredcoordinates
Eyeposition
Audition
Neck proprioception
Vestibularinformation
Visualinformation
Adapted from figure 3 from Andersen et al (1997)
![Page 14: 3D Vision The University of Oxford Andrew Glennerster Physiology Andrew Fitzgibbon Engineering](https://reader035.vdocuments.site/reader035/viewer/2022081520/56649dca5503460f94ac0e90/html5/thumbnails/14.jpg)
An Alternative
• In simpler animals, the requirement for a map is questioned.
• Many tasks can be performed by local, low-level, reactive rules
![Page 15: 3D Vision The University of Oxford Andrew Glennerster Physiology Andrew Fitzgibbon Engineering](https://reader035.vdocuments.site/reader035/viewer/2022081520/56649dca5503460f94ac0e90/html5/thumbnails/15.jpg)
(Franz and Mallot, 2000)
“Biomimetic” Navigation
![Page 16: 3D Vision The University of Oxford Andrew Glennerster Physiology Andrew Fitzgibbon Engineering](https://reader035.vdocuments.site/reader035/viewer/2022081520/56649dca5503460f94ac0e90/html5/thumbnails/16.jpg)
(Franz and Mallot, 2000)
“Biomimetic” Navigation
![Page 17: 3D Vision The University of Oxford Andrew Glennerster Physiology Andrew Fitzgibbon Engineering](https://reader035.vdocuments.site/reader035/viewer/2022081520/56649dca5503460f94ac0e90/html5/thumbnails/17.jpg)
“Biomimetic” Navigation
• Neurally plausible
• Real robot implementations
• Cannot model 3D pointing yet
• No complete model of ant, for example
![Page 18: 3D Vision The University of Oxford Andrew Glennerster Physiology Andrew Fitzgibbon Engineering](https://reader035.vdocuments.site/reader035/viewer/2022081520/56649dca5503460f94ac0e90/html5/thumbnails/18.jpg)
“computer vision hitting a computational wall”
• Need infinite-time, infinite-space operation
• Computers are getting faster...
• When will O(2t) beat O(n3)?– n is number of features– t is years from now
![Page 19: 3D Vision The University of Oxford Andrew Glennerster Physiology Andrew Fitzgibbon Engineering](https://reader035.vdocuments.site/reader035/viewer/2022081520/56649dca5503460f94ac0e90/html5/thumbnails/19.jpg)
Dynamic Scenes
![Page 20: 3D Vision The University of Oxford Andrew Glennerster Physiology Andrew Fitzgibbon Engineering](https://reader035.vdocuments.site/reader035/viewer/2022081520/56649dca5503460f94ac0e90/html5/thumbnails/20.jpg)
Open Problems
• No complete model of ant navigation– Photons to pointing
• No plausible model of 3D vision in higher animals– Coordinate system: Is there one?– One representation or many?
• Computer vision’s computational wall– One solution will be better engineering– Another may be “parallel slow neurons”
• Dynamic scenes
![Page 21: 3D Vision The University of Oxford Andrew Glennerster Physiology Andrew Fitzgibbon Engineering](https://reader035.vdocuments.site/reader035/viewer/2022081520/56649dca5503460f94ac0e90/html5/thumbnails/21.jpg)
Thanks To
Andrew GlennersterAndrew ParkerAndrew Blake
Andrew ZissermanAndrew DavisonAndrew Stoddart
...and a few non-andrews
![Page 22: 3D Vision The University of Oxford Andrew Glennerster Physiology Andrew Fitzgibbon Engineering](https://reader035.vdocuments.site/reader035/viewer/2022081520/56649dca5503460f94ac0e90/html5/thumbnails/22.jpg)
![Page 23: 3D Vision The University of Oxford Andrew Glennerster Physiology Andrew Fitzgibbon Engineering](https://reader035.vdocuments.site/reader035/viewer/2022081520/56649dca5503460f94ac0e90/html5/thumbnails/23.jpg)
Questions
• How do biological systems perform such tasks?– We want an explanation “from
photons to pointing”
• How might artificial systems perform such tasks?– Artificial systems are “easy” to analyse
![Page 24: 3D Vision The University of Oxford Andrew Glennerster Physiology Andrew Fitzgibbon Engineering](https://reader035.vdocuments.site/reader035/viewer/2022081520/56649dca5503460f94ac0e90/html5/thumbnails/24.jpg)
Status
• We have models for some of the biology
• We have artificial systems which work in useful but limited domains