computational approaches to cameras · 2017-11-16 · computational approaches to 11/16/17 cameras...
TRANSCRIPT
![Page 1: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/1.jpg)
11/16/17Computational Approaches to Cameras
Computational Photography
Derek Hoiem, University of Illinois
Magritte , The False Mirror (1935)
![Page 2: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/2.jpg)
Announcements
• Final project proposal due Monday (see links on webpage)
![Page 3: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/3.jpg)
Conventional cameras
• Conventional cameras are designed to capture light in a medium that is directly viewable
LightLensSensor
Display
Scene
Eye
![Page 4: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/4.jpg)
Computational cameras
• With a computational approach, we can capture light and then figure out what to do with it
Light
Displays
Scene
Eyes
Captured Light
Computation
![Page 5: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/5.jpg)
Questions for today
• How can we represent all of the information contained in light?
• What are the fundamental limitations of cameras?
• What sacrifices have we made in conventional cameras? For what benefits?
• How else can we design cameras for better focus, deblurring, multiple views, depth, etc.?
![Page 6: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/6.jpg)
Representing Light: The Plenoptic Function
Q: What is the set of all things that we can ever see?
A: The Plenoptic Function (Adelson & Bergen)
Let’s start with a stationary person and try to
parameterize everything that he can see…
Figure by Leonard McMillan
Slides from Efros
![Page 7: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/7.jpg)
Grayscale snapshot
is intensity of light • Seen from a single view point
• At a single time
• Averaged over the wavelengths of the visible spectrum
(can also do P(x,y), but spherical coordinate are nicer)
P(q,f)
![Page 8: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/8.jpg)
Color snapshot
is intensity of light • Seen from a single view point
• At a single time
• As a function of wavelength
P(q,f,l)
![Page 9: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/9.jpg)
A movie
is intensity of light • Seen from a single view point
• Over time
• As a function of wavelength
P(q,f,l,t)
![Page 10: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/10.jpg)
Holographic movie
is intensity of light • Seen from ANY viewpoint
• Over time
• As a function of wavelength
P(q,f,l,t,VX,VY,VZ)
![Page 11: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/11.jpg)
The Plenoptic Function
• Can reconstruct every possible view, at every
moment, from every position, at every wavelength
• Contains every photograph, every movie,
everything that anyone has ever seen!
P(q,f,l,t,VX,VY,VZ)
![Page 12: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/12.jpg)
Representing light
The atomic element of light: a pixel a ray
![Page 13: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/13.jpg)
Fundamental limitations and trade-offs
• Only so much light in a given area to capture
• Basic sensor accumulates light at a set of positions from all orientations, over all time
• We want intensity of light at a given time at one position for a set of orientations
• Solutions:
– funnel, constrain, redirect light
– change the sensor
CCD inside camera
![Page 14: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/14.jpg)
Trade-offs of conventional camera
• Add a pinhole– Pixels correspond to small range of orientations at the camera
center, instead of all gathered light at one position– Much less light hits sensor
• Add a lens– More light hits sensor– Limited depth of field– Chromatic aberration
• Add a shutter• Capture average intensity at a particular range of times
• Increase sensor resolution• Each pixel represents a smaller range of orientations• Less light per pixel
• Controls: aperture size, focal length, shutter time
![Page 15: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/15.jpg)
How else can we design cameras?
What do they sacrifice/gain?
![Page 16: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/16.jpg)
1. Light Field Photography with “Plenoptic Camera”
Ng et al. Stanford TR, 2005Adelson and Wang 1992
Conventional Camera
“Plenoptic Camera”
![Page 17: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/17.jpg)
Light field photography
• Like replacing the human retina with an insect compound eye
• Records where light ray hits the lens
![Page 18: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/18.jpg)
Stanford Plenoptic Camera [Ng et al 2005]
4000 × 4000 pixels ÷ 292 × 292 lenses = 14 × 14 pixels per lens
Contax medium format camera Kodak 16-megapixel sensor
Adaptive Optics microlens array 125μ square-sided microlenses
![Page 19: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/19.jpg)
Light field photography: applications
![Page 20: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/20.jpg)
Light field photography: applications
Change in
viewpoint
![Page 21: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/21.jpg)
Light field photography: applications
Change in viewpoint
Lateral
Along Optical Axis
![Page 22: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/22.jpg)
Digital Refocusing
![Page 23: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/23.jpg)
Light field photography w/ microlenses
• We gain
– Ability to refocus or increase depth of field
– Ability for small viewpoint shifts
• What do we lose (vs. conventional camera)?
![Page 24: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/24.jpg)
2. Coded apertures
![Page 25: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/25.jpg)
Image and Depth from a
Conventional Camera with a
Coded Aperture
Anat Levin, Rob Fergus,
Frédo Durand, William Freeman
MIT CSAIL
Slides from SIGGRAPH Presentation
![Page 26: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/26.jpg)
Single input image:
Output #1: Depth map
![Page 27: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/27.jpg)
Single input image:
Output #1: Depth map
Output #2: All-focused image
![Page 28: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/28.jpg)
LensCamera
sensor
Point
spread
function
Image of a point
light source
Lens and defocus
Focal plane
Lens’ aperture
![Page 29: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/29.jpg)
LensObjectCamera
sensor
Point
spread
function
Image of a
defocused point
light source
Lens and defocus
Lens’ aperture
Focal plane
![Page 30: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/30.jpg)
LensCamera
sensor
Point
spread
function
Image of a
defocused point
light source
Lens and defocus
Object
Lens’ aperture
Focal plane
![Page 31: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/31.jpg)
LensCamera
sensor
Point
spread
function
Image of a
defocused point
light source
Lens’ aperture
Lens and defocus
Object
Focal plane
![Page 32: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/32.jpg)
Lens and defocus
LensCamera
sensor
Point
spread
function
Image of a
defocused point
light source
Lens’ aperture
Object
Focal plane
![Page 33: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/33.jpg)
Depth and defocus
Depth from defocus:
Infer depth by analyzing
local scale of defocus blur
Out of focus
In focus
![Page 34: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/34.jpg)
Challenges
• Hard to discriminate a smooth scene from defocus blur
• Hard to undo defocus blur
Input Ringing with conventional
deblurring algorithm
Out of focus?
![Page 35: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/35.jpg)
Key ideas
• Exploit prior on natural images
- Improve deconvolution
- Improve depth discrimination
• Coded aperture (mask inside lens)
- make defocus patterns different from
natural images and easier to discriminate
Natural Unnatural
![Page 36: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/36.jpg)
xfy k
xfy k
xfy k
Defocus as local convolution
xfy k
Depth k=1:
Depth k=2:
Depth k=3:
Input defocused
image
Local
sub-window
Calibrated
blur kernels
at depth
Sharp
sub-window
k
![Page 37: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/37.jpg)
Overview
Correct scale
Smaller scale
Larger scale
Try deconvolving local input windows with different scaled filters:
Somehow: select best scale.
?
?
?
![Page 38: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/38.jpg)
Challenges
• Hard to identify
correct scale:
• Hard to deconvolve even
when kernel is known
Input Ringing with the traditional
Richardson-Lucy deconvolution algorithm
?
? Correct scale
Smaller scale
? Larger scale
• Hard to deconvolve even
when kernel is known
![Page 39: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/39.jpg)
?
yxf
=
Deconvolution is ill posed
![Page 40: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/40.jpg)
Deconvolution is ill posed
? =
=?
Solution 1:
Solution 2:
yxf
![Page 41: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/41.jpg)
Idea 1: Natural images prior
Image
gradient
put a penalty on gradients
Natural images have sparse gradients
Natural Unnatural
What makes images special?
![Page 42: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/42.jpg)
i ix )(l
Deconvolution with prior
_2
+
_ +
2
?
?
Convolution error Derivatives prior
High
Low Equal convolution error
2|| minarg yxfx
![Page 43: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/43.jpg)
?
?Correct scale
Smaller scale
?Larger scale
Try deconvolving local input windows with different scaled filters:
Recall: Overview
Challenge: smaller scale not so different than correct
Somehow: select best scale.
![Page 44: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/44.jpg)
Idea 2: Coded Aperture
• Mask (code) in aperture plane
- make defocus patterns different from
natural images and easier to discriminate
Conventional
aperture
Our coded
aperture
![Page 45: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/45.jpg)
LensCamera
sensor
Point
spread
function
Object
Solution: lens with occluder
Focal plane
![Page 46: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/46.jpg)
Solution: lens with occluder
Lens with coded
aperture
Camera
sensor
Point
spread
function
Image of a
defocused point
light source
Aperture pattern
Object
Focal plane
![Page 47: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/47.jpg)
Lens with coded
aperture
Camera
sensor
Point
spread
function
Image of a
defocused point
light source
Aperture pattern
Solution: lens with occluder
Object
Focal plane
![Page 48: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/48.jpg)
Lens with coded
aperture
Camera
sensor
Point
spread
function
Image of a
defocused point
light source
Aperture pattern
Solution: lens with occluder
Object
Focal plane
![Page 49: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/49.jpg)
Lens with coded
aperture
Camera
sensor
Point
spread
function
Image of a
defocused point
light source
Aperture pattern
Solution: lens with occluder
Object
Focal plane
![Page 50: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/50.jpg)
Lens with coded
aperture
Camera
sensor
Point
spread
function
Image of a
defocused point
light source
Aperture pattern
Solution: lens with occluder
Object
Focal plane
![Page 51: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/51.jpg)
Coded aperture reduces uncertainty in scale identification
Conventional Coded
Correct scale
Smaller scale
Larger scale
![Page 52: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/52.jpg)
Convolution- frequency domain representation
=
=0
Sharp Image
Sharp Image
Spatial convolution frequency multiplication0 Frequency
spectr
um
0 Frequency
spectr
um
0 Frequency
spectr
um
0 Frequency
spectr
um
0 Frequency
spectr
um
0 Frequency
spectr
um
Filter, 1st scale
Filter, 2nd scale
Output spectrum has zeros
where filter spectrum has zeros
2nd observed image
1st observed image
![Page 53: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/53.jpg)
?
?
Coded aperture: Scale estimation and division by zero
=
=
0 Frequency
spectr
um
0 Frequency
spectr
um
0 Frequency
spectr
um
0 Frequency
spectr
um
0 Frequency
spectr
um
Estimated image
spatial ringing
Large magnitude in image
to compensate for tiny
magnitude in filter
Estimated image
Filter, wrong scale
Filter, correct scale
Observed image
![Page 54: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/54.jpg)
?
?
=
0 Frequency
spectr
um
0 Frequency
spectr
um
0 Frequency
spectr
um
0 Frequency
spectr
um
0 Frequency
spectr
um
Estimated image
Estimated image
Division by zero with a conventional aperture?
Filter, wrong scale
Filter, correct scale
Observed image
no spatial ringing
=
![Page 55: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/55.jpg)
Filter Design
Sampled aperture patternsConventional
aperture
More discrimination
between scales
Score
Less discrimination
between scales
Analytically search for a pattern maximizing discrimination
between images at different defocus scales (KL-divergence)
Account for image prior and physical constraints
![Page 56: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/56.jpg)
Depth results
![Page 57: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/57.jpg)
Input Local depth estimation Regularized depth
Regularizing depth estimation
_2
+
Convolution error Derivatives prior
Try deblurring with 10 different aperture scales
Keep minimal error scale in each local window + regularization
i ix )(l2|| minarg yxfx
![Page 58: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/58.jpg)
Input
Local depth estimation
Regularized depth
Regularizing depth estimation
![Page 59: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/59.jpg)
All focused results
![Page 60: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/60.jpg)
Input
![Page 61: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/61.jpg)
All-focused
(deconvolved)
![Page 62: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/62.jpg)
Original image
All-focus image
Close-up
![Page 63: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/63.jpg)
Comparison- conventional aperture result
Ringing due to wrong scale estimation
![Page 64: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/64.jpg)
Comparison- coded aperture result
![Page 65: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/65.jpg)
Input
![Page 66: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/66.jpg)
All-focused
(deconvolved)
![Page 67: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/67.jpg)
Original image All-focus image
Close-up
Naïve sharpening
![Page 68: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/68.jpg)
Application: Digital refocusing from a single image
![Page 69: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/69.jpg)
Application: Digital refocusing from a single image
![Page 70: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/70.jpg)
Application: Digital refocusing from a single image
![Page 71: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/71.jpg)
Application: Digital refocusing from a single image
![Page 72: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/72.jpg)
Application: Digital refocusing from a single image
![Page 73: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/73.jpg)
Application: Digital refocusing from a single image
![Page 74: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/74.jpg)
Application: Digital refocusing from a single image
![Page 75: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/75.jpg)
Image AND depth at a single shot
No loss of image resolution
Simple modification to lens
Depth is coarse
But depth is a pure bonus
Lose some light
But deconvolution increases depth of field
Coded aperture: pros and cons
unable to get depth at untextured areas,
might need manual corrections.
-
++
+-+
+
![Page 76: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/76.jpg)
Depth acquisition: priceless
$1Cardboard:
$79.9550mm f/1.8:
Tape: $1
![Page 77: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/77.jpg)
Some more quick examples
![Page 78: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/78.jpg)
• Quickly move camera in a parabola when taking a picture
• A motion at any speed in the direction of the parabola will give the same blur kernel
![Page 79: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/79.jpg)
Results
Static
Camera
Parabolic
Camera
![Page 80: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/80.jpg)
Results
Motion in
wrong
direction
Static Camera Parabolic Camera
![Page 81: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/81.jpg)
![Page 82: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/82.jpg)
RGBW Sensors
• 2007: Kodak ‘Panchromatic’ Pixels
• Outperforms Bayer Grid– 2X-4X sensitivity (W: no filter loss)
– May improve dynamic range (W >> RGB sensitivity)
http://www.dpreview.com/news/2007/6/14/kodakhighsens
![Page 83: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/83.jpg)
Computational Approaches to Display
• 3D TV without glasses
– 20”, $2900, available in Japan (2010)
– You see different images from different angles
http://news.cnet.com/8301-13506_3-20018421-17.html
Newer version: http://www.pcmag.com/article2/0,2817,2392380,00.asp
http://reviews.cnet.com/3dtv-buying-guide/
Toshiba
![Page 84: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/84.jpg)
Recap of questions
• How can we represent all of the information contained in light?
• What are the fundamental limitations of cameras?
• What sacrifices have we made in conventional cameras? For what benefits?
• How else can we design cameras for better focus, deblurring, multiple views, depth, etc.?
![Page 85: Computational Approaches to Cameras · 2017-11-16 · Computational Approaches to 11/16/17 Cameras Computational Photography Derek Hoiem, ... Light Field Photography with ^Plenoptic](https://reader034.vdocuments.site/reader034/viewer/2022042216/5ebf40b23ed0af43a953a9c5/html5/thumbnails/85.jpg)
Next class
• Exam review
• But first, have a good Thanksgiving break!