cameras and image processing
TRANSCRIPT
Cameras and Image Processing
Computational Photography
CSE 291
Lecture 2
Announcements
โข Assignment 1 will be released today
โ Due Apr 8, 11:59 PM
CSE 291, Spring 2020 2
Traditional photography
โข The acquisition of images by recording light (or other electromagnetic radiation) either electronically (using an image sensor) or chemically (using film)
โข Typically, a lens is used to focus the light reflected or emitted from objects in the scene being imaged into an image on the light sensitive surface inside a camera during a timed exposure
CSE 291, Spring 2020 3
Geometric image formation
โข How do 3D world points project to 2D image points?
CSE 291, Spring 2020 4
The projective camera
โข Extrinsic Parameters: Since camera may not be at the origin, there is a rigid transformation between the world coordinates and the camera coordinates
โข Intrinsic parameters: Since scene units (e.g., cm) differ image units (e.g., pixels) and coordinate system may not be centered in image, we capture that with a 3x3 transformation comprised of focal length, principal point, pixel aspect ratio, and skew
=
T
Z
Y
X
w
y
x
parameters extrinsic
by drepresente
tionTransforma Rigid
0100
0010
0001
parameters intrinsic
by drepresente
tionTransforma
3 x 3 4 x 4
CSE 291, Spring 2020 5
The reason for lenses
CSE 291, Spring 2020 6
We need light, but big pinholes cause blur
Thin lens, image of point
CSE 291, Spring 2020 7
O
All rays passing through lens and starting at Pconverge upon Pโ
So light gather capability of lens is given the area of the lens and all the rays focus on Pโ instead of become blurred like a pinhole
F
P
Pโ
Thin lens, image plane
CSE 291, Spring 2020 8
OF
P
Pโ
Image Plane
Qโ
Q
A price: Whereas the image of P is in focus,the image of Q isnโt.
Thin lens, aperture
CSE 291, Spring 2020 9
O
P
Pโ
Image Plane โข Smaller Aperture-> Less Blur
โข Pinhole -> No Blur
Lens distortion
โข Radial distortion
โข Tangential distortion
CSE 291, Spring 2020 10
Images of straight lines should be
straight
Photometric image formation
โข What color is the projected point?
CSE 291, Spring 2020 11
Need bidirectional reflectance distribution function (BRDF) at
point on surface
Lighting, reflectance, and shading
CSE 291, Spring 2020 12
BRDF
Image acquisition
CSE 291, Spring 2020 13
Digitization, one row of image
CSE 291, Spring 2020 14
Digitization, whole image
CSE 291, Spring 2020 15
Number of quantization levels
CSE 291, Spring 2020 16
256 128
64 32
16 8
4 2
Image sensing pipeline
CSE 291, Spring 2020 17
Image processing
Image processing
โข A discipline in which both the input and output of a process are images
โ There are usually other input parameters to the process
CSE 291, Spring 2020 18
Demosaicing
CSE 291, Spring 2020 19
Image sensor
Color filter array (CFA)
Bayer pattern
CFA Interpolated (lower case)pixel values
Image processing
โข Color spaces
โข Gamut mapping
โข White balancing and color balancing
CSE 291, Spring 2020 20
Image coding
โข Common standardsโ Video
โข Recommendation ITU-R BT.601 (standard-definition television (SDTV))
โข SMPTE standard 240M (precursor to Rec. 709)โข Recommendation ITU-R BT.709 (high-definition television
(HDTV))
โ Imageโข sRGBโข Adobe RGBโข Wide gamut RGB (or Adobe Wide Gamut RGB)โข ProPhoto RGB (or reference output medium metric (ROMM)
RGB)
CSE 291, Spring 2020 21
Different codings, different pixel values
CSE 291, Spring 2020 22
Color specification: chromaticity
โข Chromaticity coordinates
โ (x, y, z)
where x + y + z = 1
โ Usually specified by (x, y)
where z = 1 โ x โ y
The CIE 1931 color space chromaticity diagram
CSE 291, Spring 2020 23
Chromaticities
โข Set of chromaticities
โ Red
โ Green
โ Blue
โ White (point)
CSE 291, Spring 2020 24
Standard illuminantsHue of each white point, calculated with luminance Y = 0.54
CSE 291, Spring 2020 25
Chromaticities of common video standards
CSE 291, Spring 2020 26
Color CIE x CIE y CIE z
Red 0.6400 0.3300 0.0300
Green 0.3000 0.6000 0.1000
Blue 0.1500 0.0600 0.7900
White 0.3127 0.3290 0.3583
Recommendation ITU-R BT.709
Color CIE x CIE y CIE z
Red 0.6400 0.3300 0.0300
Green 0.2900 0.6000 0.1100
Blue 0.1500 0.0600 0.7900
White 0.3127 0.3290 0.3583
Recommendation ITU-R BT.601625 lines
Color CIE x CIE y CIE z
Red 0.6300 0.3400 0.0300
Green 0.3100 0.5950 0.0950
Blue 0.1550 0.0700 0.7750
White 0.3127 0.3290 0.3583
Recommendation ITU-R BT.601525 lines
Color CIE x CIE y CIE z
Red 0.6300 0.3400 0.0300
Green 0.3100 0.5950 0.0950
Blue 0.1550 0.0700 0.7750
White 0.3127 0.3290 0.3583
SMPTE standard 240M
D65
D65 D65
D65
Chromaticities of common image standards
CSE 291, Spring 2020 27
Color CIE x CIE y CIE z
Red 0.6400 0.3300 0.0300
Green 0.3000 0.6000 0.1000
Blue 0.1500 0.0600 0.7900
White 0.3127 0.3290 0.3583
sRGB
Color CIE x CIE y CIE z
Red 0.7347 0.2653 0.0000
Green 0.1152 0.8264 0.0584
Blue 0.1566 0.0177 0.8257
White 0.3457 0.3585 0.2958
Wide gamut RGB(or Adobe Wide Gamut RGB)
Color CIE x CIE y CIE z
Red 0.7347 0.2653 0.0000
Green 0.1596 0.8404 0.0000
Blue 0.0366 0.0001 0.9633
White 0.3457 0.3585 0.2958
ProPhoto RGB(or reference output media metric (ROMM RGB))
Color CIE x CIE y CIE z
Red 0.6400 0.3300 0.0300
Green 0.2100 0.7100 0.0800
Blue 0.1500 0.0600 0.7900
White 0.3127 0.3290 0.3583
Adobe RGB
D65
D50 D50
D65
Chromaticity diagrams
Rec. 709 and sRGB35.9% of visible colors
Adobe RGB52.1% of visible colors
CSE 291, Spring 2020 28
Chromaticity diagrams
Rec. 709 and sRGB35.9% of visible colors
Wide gamut RGB77.6% of visible colors
CSE 291, Spring 2020 29
Chromaticity diagrams
Rec. 709 and sRGB35.9% of visible colors
ProPhoto RGB90% of visible colors
CSE 291, Spring 2020 30
Academy Color Encoding Specification (ACES)
Color CIE x CIE y CIE z
Red 0.73470 0.26530 0.00000
Green 0.00000 1.00000 0.00000
Blue 0.00010 -0.07700 1.07690
White 0.32168 0.33767 0.34065
ACES
Approximately D60
100% of visible colors
CSE 291, Spring 2020 31
Nonlinear encoding
โข All of these standards use nonlinear encoding (gamma encoding)โ Video
โข Recommendation ITU-R BT.601 (standard-definition television (SDTV))
โข SMPTE standard 240M (precursor to Rec. 709)
โข Recommendation ITU-R BT.709 (high-definition television (HDTV))
โ Imageโข sRGB
โข Adobe RGB
โข Wide gamut RGB (or Adobe Wide Gamut RGB)
โข ProPhoto RGB (or reference output medium metric (ROMM) RGB)
CSE 291, Spring 2020 32
Nonlinear encoding and conversion to linear
โข Typical CRT monitors have a transfer function of gamma = 2.2
โข Image and video standards were designed to be directly displayed on CRTs
โ Pixel values are encoded to approximate gamma = 2.2
โข Nonlinear to linear (floating-point) using a lookup table
โข Linear to nonlinear calculation
CSE 291, Spring 2020 33
LinearNonlinear
Nonlinear RโGโBโ color space and linear RGB color space
โข Example: sRGB
CSE 291, Spring 2020 34
RGB linear
sRGB nonlinear
Slope of sRGB nonlinearin log-log space
XYZ color space
โข Encompasses all color sensations the average person can experience
โข Standard referenceโ Many other color space definitions are based on XYZ
โข Y is luminanceโข Z is quasi-equal to blue stimulationโข X is a linear combination of cone response curves chosen to be
nonnegativeโข The plane parallel to the XZ plane and that Y lies on contains all
possible chromaticities at that luminance
CSE 291, Spring 2020 35
RGB color space to XYZ color space
๐๐๐
=
๐๐ ๐๐ ๐๐๐๐ ๐๐ ๐๐๐๐ ๐๐ ๐๐
๐ ๐บ๐ต
๐๐๐
=
๐๐ฅ(๐๐ + ๐๐ + ๐๐) ๐๐ฅ(๐๐ + ๐๐ + ๐๐) ๐๐ฅ(๐๐ + ๐๐ + ๐๐)๐๐ฆ(๐๐ + ๐๐ + ๐๐) ๐๐ฆ(๐๐ + ๐๐ + ๐๐) ๐๐ฆ(๐๐ + ๐๐ + ๐๐)
๐๐ง(๐๐ + ๐๐ + ๐๐) ๐๐ง(๐๐ + ๐๐ + ๐๐) ๐๐ง(๐๐ + ๐๐ + ๐๐)
๐ ๐บ๐ต
๐๐๐
=
๐๐ฅ ๐๐ฅ ๐๐ฅ๐๐ฆ ๐๐ฆ ๐๐ฆ๐๐ง ๐๐ง ๐๐ง
๐๐ + ๐๐ + ๐๐ 0 00 ๐๐ + ๐๐ + ๐๐ 00 0 ๐๐ + ๐๐ + ๐๐
๐ ๐บ๐ต
CSE 291, Spring 2020 36
RGB color space to XYZ color space
โข Substitute in RGB white and XYZ white
CSE 291, Spring 2020 37
๐๐ ๐บ๐ต =111
๐๐๐๐ =
๐ค๐ฅ๐ค๐ฆ1๐ค๐ง๐ค๐ฆ
๐ค๐ฅ๐ค๐ฆ1๐ค๐ง๐ค๐ฆ
=
๐๐ฅ ๐๐ฅ ๐๐ฅ๐๐ฆ ๐๐ฆ ๐๐ฆ๐๐ง ๐๐ง ๐๐ง
๐๐ + ๐๐ + ๐๐ 0 00 ๐๐ + ๐๐ + ๐๐ 00 0 ๐๐ + ๐๐ + ๐๐
111
๐ค๐ฅ
๐ค๐ฆ
1๐ค๐ง
๐ค๐ฆ
=
๐๐ฅ ๐๐ฅ ๐๐ฅ๐๐ฆ ๐๐ฆ ๐๐ฆ๐๐ง ๐๐ง ๐๐ง
๐๐ + ๐๐ + ๐๐๐๐ + ๐๐ + ๐๐๐๐ + ๐๐ + ๐๐
, solve for
๐๐ + ๐๐ + ๐๐๐๐ + ๐๐ + ๐๐๐๐ + ๐๐ + ๐๐
M_RGB_to_XYZ =
๐๐ฅ ๐๐ฅ ๐๐ฅ๐๐ฆ ๐๐ฆ ๐๐ฆ๐๐ง ๐๐ง ๐๐ง
๐๐ + ๐๐ + ๐๐ 0 00 ๐๐ + ๐๐ + ๐๐ 00 0 ๐๐ + ๐๐ + ๐๐
Chromatic adaptation
โข Estimating the appearance of a sample under a different illuminantโ Convert between different
white points
โข LMS color spaceโ Response of the three types of
cones in the human eyeโข Long, medium, and short
wavelengths
โข XYZ to LMSโ Bradford transformation
matrix
โข Chromatic adaptationโ Adaptation matrix
CSE 291, Spring 2020 38
Color mapping (using chromaticities)
โข Image 1 to image 2โ Calculate transformation matrix from image 1 RGB to XYZ
(M_RGB1_to_XYZ1) and transformation matrix from image 2 RGB to XYZ (M_RGB2_to_XYZ2)โข White points in XYZ are the same as in RGB
โ If white points are the sameโข M_RGB1_to_RGB2 = inv(M_RGB2_to_XYZ2) * M_RGB1_to_XYZ1
โ Else (white points are different, include chromatic adaptation)โข Bradford transformation matrix (M_XYZ_to_LMS)โข Map white points to LMS and calculate adaptation matrix
(M_LMS1_to_LMS2)โข Compose transformation matrices
โ M_RGB1_to_RGB2 = inv(M_RGB2_to_XYZ2) * inv(M_XYZ_to_LMS) * M_LMS1_to_LMS2 * M_XYZ_to_LMS * M_RGB1_to_XYZ1
CSE 291, Spring 2020 39
Note: luminance Y and luma Yโ
โข Luminance is calculated from linear RGB
โ Y coordinate of XYZ
โข Luma is calculated from nonlinear RโGโBโ
โข Example: sRGBY = 0.21263903 * R + 0.71516871 * G + 0.072192319 * B
Yโ = 0.21263903 * Rโ + 0.71516871 * Gโ + 0.072192319 * Bโ
(Coefficients are the middle row of M_RGB_to_XYZ)
CSE 291, Spring 2020 40
Remember
โข Common video and image standards use nonlinear encoding and may use different chromaticities
โข Chromaticities define gamuts that contain a percent of the visible colorsโ Some contain more than others
โข Mapping pixel data between these standards requires mapping to the XYZ color space and possibly the LMS color space
โข Luminance is different than luma
CSE 291, Spring 2020 41