15-463: computational photography alexei efros, cmu, fall...
TRANSCRIPT
Capturing Light… in man and machine
15-463: Computational PhotographyAlexei Efros, CMU, Fall 2005Some figures from Steve Seitz, Steve
Palmer, Paul Debevec, and Gonzalez et al.
Image Formation
Digital Camera
The Eye
Film
Digital camera
A digital camera replaces film with a sensor array• Each cell in the array is light-sensitive diode that converts photons to electrons• Two common types
– Charge Coupled Device (CCD)– CMOS
• http://electronics.howstuffworks.com/digital-camera.htm
Sensor Array
CMOS sensor
Sampling and Quantization
Interlace vs. progressive scan
http://www.axis.com/products/video/camera/progressive_scan.htm
Progressive scan
http://www.axis.com/products/video/camera/progressive_scan.htm
Interlace
http://www.axis.com/products/video/camera/progressive_scan.htm
The Eye
The human eye is a camera!• Iris - colored annulus with radial muscles• Pupil - the hole (aperture) whose size is controlled by the iris• What’s the “film”?
– photoreceptor cells (rods and cones) in the retina
The Retina
Cross-section of eye
Ganglion cell layer
Bipolar cell layer
Receptor layer
Pigmentedepithelium
Ganglion axons
Cross section of retina
Retina up-close
Light
© Stephen E. Palmer, 2002
Conescone-shaped less sensitiveoperate in high lightcolor vision
Two types of light-sensitive receptors
cone
rod
Rodsrod-shapedhighly sensitiveoperate at nightgray-scale vision
Rod / Cone sensitivity
The famous sock-matching problem…
© Stephen E. Palmer, 2002
Distribution of Rods and Cones
0
150,000100,00050,000
020 40 60 8020406080
Visual Angle (degrees from fovea)
Rods
Cones Cones
Rods
FoveaBlindSpot
# R
ecep
tors
/mm
2
Night Sky: why are there more stars off-center?
Electromagnetic Spectrum
http://www.yorku.ca/eye/photopik.htm
Human Luminance Sensitivity Function
Why do we see light of these wavelengths?
© Stephen E. Palmer, 2002
.
0 1000 2000 3000
Ene
rgy
Wavelength (nm)
400 700
700 C
2000 C
5000 C
10000 C
VisibleRegion
…because that’s where theSun radiates EM energy
Visible Light
The Physics of Light
Any patch of light can be completely describedphysically by its spectrum: the number of photons (per time unit) at each wavelength 400 - 700 nm.
400 500 600 700
Wavelength (nm.)
# Photons(per ms.)
© Stephen E. Palmer, 2002
The Physics of Light
# P
hoto
ns
D. Normal Daylight
Wavelength (nm.)
B. Gallium Phosphide Crystal
400 500 600 700
# P
hoto
ns
Wavelength (nm.)
A. Ruby Laser
400 500 600 700
400 500 600 700
# P
hoto
ns
C. Tungsten Lightbulb
400 500 600 700
# P
hoto
ns
Some examples of the spectra of light sources
© Stephen E. Palmer, 2002
The Physics of Light
Some examples of the reflectance spectra of surfaces
Wavelength (nm)
% P
hoto
ns R
efle
cted
Red
400 700
Yellow
400 700
Blue
400 700
Purple
400 700
© Stephen E. Palmer, 2002
The Psychophysical Correspondence
There is no simple functional description for the perceivedcolor of all lights under all viewing conditions, but …...
A helpful constraint:Consider only physical spectra with normal distributions
area
Wavelength (nm.)
# Photons
400 700500 600
mean
variance
© Stephen E. Palmer, 2002
The Psychophysical Correspondence
Mean Hue
yellowgreenblue
# P
hoto
ns
Wavelength
© Stephen E. Palmer, 2002
The Psychophysical Correspondence
Variance Saturation
Wavelength
high
medium
low
hi.
med.
low# P
hoto
ns
© Stephen E. Palmer, 2002
The Psychophysical Correspondence
Area Brightness#
Pho
tons
Wavelength
B. Area Lightness
bright
dark
© Stephen E. Palmer, 2002
© Stephen E. Palmer, 2002
400 450 500 550 600 650
RE
LATI
VE
AB
SO
RB
AN
CE
(%)
WAVELENGTH (nm.)
100
50
440
S
530 560 nm.
M L
Three kinds of cones:
Physiology of Color Vision
• Why are M and L cones so close?• Are are there 3?
More Spectra
metamers
Color Sensing in Camera (RGB)
3-chip vs. 1-chip: quality vs. costWhy more green?
http://www.cooldihttp://www.cooldictionary.com/words/Bayertionary.com/words/Bayer--filter.wikipediafilter.wikipedia
Why 3 colors?
Practical Color Sensing: Bayer Grid
Estimate RGBat ‘G’ cels from neighboring values
http://www.cooldictionary.com/words/Bayer-filter.wikipedia
RGB color space
RGB cube• Easy for devices• But not perceptual• Where do the grays live?• Where is hue and saturation?
HSV
Hue, Saturation, Value (Intensity)• RGB cube on its vertex
Decouples the three components (a bit)Use rgb2hsv() and hsv2rgb() in Matlab
White Balance
White World / Gray World assumptions
Programming Assignment #1
• How to compare R,G,B channels?• No right answer
• Sum of Squared Differences (SSD):
• Normalized Correlation (NCC):
Image Pyramids (preview)
Known as a Gaussian Pyramid [Burt and Adelson, 1983]• In computer graphics, a mip map [Williams, 1983]• A precursor to wavelet transform
Image Formation
f(x,y) = reflectance(x,y) * illumination(x,y)Reflectance in [0,1], illumination in [0,inf]
Problem: Dynamic Range
15001500
11
25,00025,000
400,000400,000
2,000,000,0002,000,000,000
The real world isHigh dynamic range
pixel (312, 284) = 42pixel (312, 284) = 42
ImageImage
42 photos?42 photos?
Is Camera a photometer?
Long Exposure
10-6 106
10-6 106
Real world
Picture
0 to 255
High dynamic range
Short Exposure
10-6 106
10-6 106
Real world
Picture
0 to 255
High dynamic range
sceneradiance
(W/sr/m )
scenesceneradianceradiance
(W/(W/sr/msr/m ))
∫∫sensorirradiance
sensorsensorirradianceirradiance
sensorexposuresensorsensor
exposureexposure222
ΔΔtt
LensLensLens ShutterShutterShutter
analogvoltagesanalog
voltagesdigitalvaluesdigitalvalues
pixelvaluespixel
values
CCDCCD ADCADC RemappingRemapping
Image Acquisition Pipeline
Camera is NOT a photometer!
Varying Exposure
What does the eye sees?
The eye has a huge dynamic rangeDo we see a true radiance map?
Eye is not a photometer!
"Every light is a shade, compared to the higher lights, till you come to the sun; and every shade is a light, compared to the deeper shades, till you come to the night."
— John Ruskin, 1879
Cornsweet Illusion
CampbellCampbell--Robson contrast sensitivity curveRobson contrast sensitivity curve
Sine wave
Metamers
Eye is sensitive to changes(more on this later…)