![Page 1: Lecture 02 Image Formation - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2017/02_image_formation_1.pdf · 2017-10-05 · • First photo: Joseph Nicephore Niepce (1822) • Daguerréotypes](https://reader034.vdocuments.site/reader034/viewer/2022050302/5f6b81c7bd906434df1961d9/html5/thumbnails/1.jpg)
Lecture 02Image Formation
Davide Scaramuzza
Institute of Informatics – Institute of Neuroinformatics
![Page 2: Lecture 02 Image Formation - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2017/02_image_formation_1.pdf · 2017-10-05 · • First photo: Joseph Nicephore Niepce (1822) • Daguerréotypes](https://reader034.vdocuments.site/reader034/viewer/2022050302/5f6b81c7bd906434df1961d9/html5/thumbnails/2.jpg)
Outline of this lecture
• Image Formation
• Other camera parameters
• Digital camera
• Perspective camera model
• Lens distortion
![Page 3: Lecture 02 Image Formation - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2017/02_image_formation_1.pdf · 2017-10-05 · • First photo: Joseph Nicephore Niepce (1822) • Daguerréotypes](https://reader034.vdocuments.site/reader034/viewer/2022050302/5f6b81c7bd906434df1961d9/html5/thumbnails/3.jpg)
Historical context• Pinhole model: Mozi (470-390 BCE),
Aristotle (384-322 BCE)
• Principles of optics (including lenses):
Alhacen (965-1039 CE)
• Camera obscura: Leonardo da Vinci
(1452-1519), Johann Zahn (1631-1707)
• First photo: Joseph Nicephore Niepce (1822)
• Daguerréotypes (1839)
• Photographic film (Eastman, 1889)
• Cinema (Lumière Brothers, 1895)
• Color Photography (Lumière Brothers, 1908)
• Television (Baird, Farnsworth, Zworykin, 1920s)
• First consumer camera with CCD:
Sony Mavica (1981)
• First fully digital camera: Kodak DCS100 (1990)
Niepce, “La Table Servie,” 1822
CCD chip
Alhacen’s notes
![Page 4: Lecture 02 Image Formation - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2017/02_image_formation_1.pdf · 2017-10-05 · • First photo: Joseph Nicephore Niepce (1822) • Daguerréotypes](https://reader034.vdocuments.site/reader034/viewer/2022050302/5f6b81c7bd906434df1961d9/html5/thumbnails/4.jpg)
Image formation
• How are objects in the world captured in an image?
![Page 5: Lecture 02 Image Formation - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2017/02_image_formation_1.pdf · 2017-10-05 · • First photo: Joseph Nicephore Niepce (1822) • Daguerréotypes](https://reader034.vdocuments.site/reader034/viewer/2022050302/5f6b81c7bd906434df1961d9/html5/thumbnails/5.jpg)
filmobject
• Place a piece of film in front of an object Do we get a reasonable image?
How to form an image
![Page 6: Lecture 02 Image Formation - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2017/02_image_formation_1.pdf · 2017-10-05 · • First photo: Joseph Nicephore Niepce (1822) • Daguerréotypes](https://reader034.vdocuments.site/reader034/viewer/2022050302/5f6b81c7bd906434df1961d9/html5/thumbnails/6.jpg)
filmobject
Pinhole camera
• Add a barrier to block off most of the rays
– This reduces blurring
– The opening is known as the aperture
barrier
![Page 7: Lecture 02 Image Formation - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2017/02_image_formation_1.pdf · 2017-10-05 · • First photo: Joseph Nicephore Niepce (1822) • Daguerréotypes](https://reader034.vdocuments.site/reader034/viewer/2022050302/5f6b81c7bd906434df1961d9/html5/thumbnails/7.jpg)
Camera obscura
• Basic principle known to Mozi (470-390 BC), Aristotle (384-322 BC)
• Drawing aid for artists: described by Leonardo da Vinci (1452-1519)
• Image is inverted
• Depth of the room (box) is the effectivefocal length
"Reinerus Gemma-Frisius, observed an eclipse of the sun at Louvain on January 24, 1544, and later he used this illustration of the event in his book De Radio Astronomica et Geometrica, 1545. It is thought to be the first published illustration of a camera obscura..." Hammond, John H., The Camera Obscura, A Chronicle
In Latin, means ‘dark room’
![Page 8: Lecture 02 Image Formation - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2017/02_image_formation_1.pdf · 2017-10-05 · • First photo: Joseph Nicephore Niepce (1822) • Daguerréotypes](https://reader034.vdocuments.site/reader034/viewer/2022050302/5f6b81c7bd906434df1961d9/html5/thumbnails/8.jpg)
Camera obscura at home
Sketch from http://www.funsci.com/fun3_en/sky/sky.htmhttp://www.youtube.com/watch?v=B2aOs8RWntg
![Page 9: Lecture 02 Image Formation - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2017/02_image_formation_1.pdf · 2017-10-05 · • First photo: Joseph Nicephore Niepce (1822) • Daguerréotypes](https://reader034.vdocuments.site/reader034/viewer/2022050302/5f6b81c7bd906434df1961d9/html5/thumbnails/9.jpg)
Home-made pinhole camera
What can we do to reduce the blur?
![Page 10: Lecture 02 Image Formation - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2017/02_image_formation_1.pdf · 2017-10-05 · • First photo: Joseph Nicephore Niepce (1822) • Daguerréotypes](https://reader034.vdocuments.site/reader034/viewer/2022050302/5f6b81c7bd906434df1961d9/html5/thumbnails/10.jpg)
Effects of the Aperture Size
• In an ideal pinhole, only one ray of light reaches each point on the film the image can be very dim
• Making aperture bigger makes the image blurry
![Page 11: Lecture 02 Image Formation - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2017/02_image_formation_1.pdf · 2017-10-05 · • First photo: Joseph Nicephore Niepce (1822) • Daguerréotypes](https://reader034.vdocuments.site/reader034/viewer/2022050302/5f6b81c7bd906434df1961d9/html5/thumbnails/11.jpg)
Shrinking the aperture
Why not make the aperture as small as possible?
![Page 12: Lecture 02 Image Formation - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2017/02_image_formation_1.pdf · 2017-10-05 · • First photo: Joseph Nicephore Niepce (1822) • Daguerréotypes](https://reader034.vdocuments.site/reader034/viewer/2022050302/5f6b81c7bd906434df1961d9/html5/thumbnails/12.jpg)
Shrinking the aperture
Why not make the aperture as small as possible?
• Less light gets through (must increase the exposure)
• Diffraction effects…
![Page 13: Lecture 02 Image Formation - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2017/02_image_formation_1.pdf · 2017-10-05 · • First photo: Joseph Nicephore Niepce (1822) • Daguerréotypes](https://reader034.vdocuments.site/reader034/viewer/2022050302/5f6b81c7bd906434df1961d9/html5/thumbnails/13.jpg)
filmobject Lens
Image formation using a converging lens
• A lens focuses light onto the film
• Rays passing through the Optical Center are not deviated
![Page 14: Lecture 02 Image Formation - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2017/02_image_formation_1.pdf · 2017-10-05 · • First photo: Joseph Nicephore Niepce (1822) • Daguerréotypes](https://reader034.vdocuments.site/reader034/viewer/2022050302/5f6b81c7bd906434df1961d9/html5/thumbnails/14.jpg)
• All rays parallel to the Optical Axis converge at the Focal Point
Lensobject
Focal Point
f
Optical Axis
Focal Length:
Image formation using a converging lens
![Page 15: Lecture 02 Image Formation - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2017/02_image_formation_1.pdf · 2017-10-05 · • First photo: Joseph Nicephore Niepce (1822) • Daguerréotypes](https://reader034.vdocuments.site/reader034/viewer/2022050302/5f6b81c7bd906434df1961d9/html5/thumbnails/15.jpg)
Thin lens equation
Object Lens
Focal Point
Image
B
A
z e
f
• Similar Triangles:
z
e
A
B
Find a relationship between f, z, and e
![Page 16: Lecture 02 Image Formation - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2017/02_image_formation_1.pdf · 2017-10-05 · • First photo: Joseph Nicephore Niepce (1822) • Daguerréotypes](https://reader034.vdocuments.site/reader034/viewer/2022050302/5f6b81c7bd906434df1961d9/html5/thumbnails/16.jpg)
Thin lens equation
• Similar Triangles:
Object Lens
Focal Point
Image
f
fe
A
B
B
A
z e
f
A
1f
e
z
e
A
B
ezfz
e
f
e 1111
“Thin lens equation”
Any object point satisfying this equation is in focus
![Page 17: Lecture 02 Image Formation - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2017/02_image_formation_1.pdf · 2017-10-05 · • First photo: Joseph Nicephore Niepce (1822) • Daguerréotypes](https://reader034.vdocuments.site/reader034/viewer/2022050302/5f6b81c7bd906434df1961d9/html5/thumbnails/17.jpg)
“In focus”
• There is a specific distance from the lens, at which world points are “in focus” in the image
• Other points project to a “blur circle” in the image
Lensobject
f
Optical Axis Focal Point
“Circle of Confusion”or
“Blur Circle”
film
![Page 18: Lecture 02 Image Formation - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2017/02_image_formation_1.pdf · 2017-10-05 · • First photo: Joseph Nicephore Niepce (1822) • Daguerréotypes](https://reader034.vdocuments.site/reader034/viewer/2022050302/5f6b81c7bd906434df1961d9/html5/thumbnails/18.jpg)
Blur Circle
• Object is out of focus Blur Circle has radius: – A minimal L (pinhole) gives minimal R
– To capture a ‘good’ image:adjust camera settings, such that R remains smaller than the image resolution
e
LR
2
Object
Lens
R
Image Plane
Blur Circle of radius
L
Focal Plane
f
z e
![Page 19: Lecture 02 Image Formation - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2017/02_image_formation_1.pdf · 2017-10-05 · • First photo: Joseph Nicephore Niepce (1822) • Daguerréotypes](https://reader034.vdocuments.site/reader034/viewer/2022050302/5f6b81c7bd906434df1961d9/html5/thumbnails/19.jpg)
The Pin-hole approximation
• What happens if 𝑧 ≫ 𝑓 ?
• We need to adjust the image plane such that objects at infinity are in focus
Lens
Focal Point
Image
B
Object
A
z e
f
A
ezf
111 ef
ef
11
0
![Page 20: Lecture 02 Image Formation - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2017/02_image_formation_1.pdf · 2017-10-05 · • First photo: Joseph Nicephore Niepce (1822) • Daguerréotypes](https://reader034.vdocuments.site/reader034/viewer/2022050302/5f6b81c7bd906434df1961d9/html5/thumbnails/20.jpg)
The Pin-hole approximation
• What happens if 𝑧 ≫ 𝑓 ?
• We need to adjust the image plane such that objects at infinity are in focus
ezf
111 ef
ef
11
0
Lens
Focal Point
'h
Object
h
z
f
COptical Center
orCenter of Projection
Focal Plane
![Page 21: Lecture 02 Image Formation - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2017/02_image_formation_1.pdf · 2017-10-05 · • First photo: Joseph Nicephore Niepce (1822) • Daguerréotypes](https://reader034.vdocuments.site/reader034/viewer/2022050302/5f6b81c7bd906434df1961d9/html5/thumbnails/21.jpg)
The Pin-hole approximation
• What happens if 𝑧 ≫ 𝑓 ?
• We need to adjust the image plane such that objects at infinity are in focus
• The dependence of the apparent size of an object on its depth (i.e. distance from the camera) is known as perspective
Lens
Focal Point
'h
Object
h
z
f
COptical Center
orCenter or Projection
hz
fh
z
f
h
h '
'
![Page 22: Lecture 02 Image Formation - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2017/02_image_formation_1.pdf · 2017-10-05 · • First photo: Joseph Nicephore Niepce (1822) • Daguerréotypes](https://reader034.vdocuments.site/reader034/viewer/2022050302/5f6b81c7bd906434df1961d9/html5/thumbnails/22.jpg)
Perspective effects
• Far away objects appear smaller
![Page 23: Lecture 02 Image Formation - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2017/02_image_formation_1.pdf · 2017-10-05 · • First photo: Joseph Nicephore Niepce (1822) • Daguerréotypes](https://reader034.vdocuments.site/reader034/viewer/2022050302/5f6b81c7bd906434df1961d9/html5/thumbnails/23.jpg)
Perspective effects
![Page 24: Lecture 02 Image Formation - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2017/02_image_formation_1.pdf · 2017-10-05 · • First photo: Joseph Nicephore Niepce (1822) • Daguerréotypes](https://reader034.vdocuments.site/reader034/viewer/2022050302/5f6b81c7bd906434df1961d9/html5/thumbnails/24.jpg)
Perspective and art• Use of correct perspective projection indicated in 1st century BCE
frescoes
• During Renaissance time, artists developped systematic methods to determine perspective projection (around 1480-1515)
DurerRaphael
![Page 25: Lecture 02 Image Formation - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2017/02_image_formation_1.pdf · 2017-10-05 · • First photo: Joseph Nicephore Niepce (1822) • Daguerréotypes](https://reader034.vdocuments.site/reader034/viewer/2022050302/5f6b81c7bd906434df1961d9/html5/thumbnails/25.jpg)
Playing with Perspective• Perspective gives us very strong depth cues
hence we can perceive a 3D scene by viewing its 2D representation (i.e. image)
• An example where perception of 3D scenes is misleading is the Ames room
A clip from "The computer that ate Hollywood" documentary.
Dr. Vilayanur S. Ramachandran.
“Ames room”
![Page 26: Lecture 02 Image Formation - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2017/02_image_formation_1.pdf · 2017-10-05 · • First photo: Joseph Nicephore Niepce (1822) • Daguerréotypes](https://reader034.vdocuments.site/reader034/viewer/2022050302/5f6b81c7bd906434df1961d9/html5/thumbnails/26.jpg)
Projective Geometry
What is preserved?
• Straight lines are still straight
![Page 27: Lecture 02 Image Formation - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2017/02_image_formation_1.pdf · 2017-10-05 · • First photo: Joseph Nicephore Niepce (1822) • Daguerréotypes](https://reader034.vdocuments.site/reader034/viewer/2022050302/5f6b81c7bd906434df1961d9/html5/thumbnails/27.jpg)
Projective Geometry
What is lost?
• Length
• Angles
Perpendicular?
Parallel?
![Page 28: Lecture 02 Image Formation - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2017/02_image_formation_1.pdf · 2017-10-05 · • First photo: Joseph Nicephore Niepce (1822) • Daguerréotypes](https://reader034.vdocuments.site/reader034/viewer/2022050302/5f6b81c7bd906434df1961d9/html5/thumbnails/28.jpg)
Vanishing points and lines
Parallel lines in the world intersect in the image at a “vanishing point”
![Page 29: Lecture 02 Image Formation - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2017/02_image_formation_1.pdf · 2017-10-05 · • First photo: Joseph Nicephore Niepce (1822) • Daguerréotypes](https://reader034.vdocuments.site/reader034/viewer/2022050302/5f6b81c7bd906434df1961d9/html5/thumbnails/29.jpg)
Vanishing points and lines
Vanishingpoint
Vanishingline
Vanishingpoint
Vertical vanishingpoint
(at infinity)
Parallel lines in the world intersect in the image at a “vanishing point”
![Page 30: Lecture 02 Image Formation - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2017/02_image_formation_1.pdf · 2017-10-05 · • First photo: Joseph Nicephore Niepce (1822) • Daguerréotypes](https://reader034.vdocuments.site/reader034/viewer/2022050302/5f6b81c7bd906434df1961d9/html5/thumbnails/30.jpg)
Vanishing points and lines
oVanishing Point o
Vanishing Point
Vanishing Line
Parallel planes in the world intersect in the image at a “vanishing line”
![Page 31: Lecture 02 Image Formation - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2017/02_image_formation_1.pdf · 2017-10-05 · • First photo: Joseph Nicephore Niepce (1822) • Daguerréotypes](https://reader034.vdocuments.site/reader034/viewer/2022050302/5f6b81c7bd906434df1961d9/html5/thumbnails/31.jpg)
Outline of this lecture
• Image Formation
• Other camera parameters
• Digital camera
• Perspective camera model
• Lens distortion
![Page 32: Lecture 02 Image Formation - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2017/02_image_formation_1.pdf · 2017-10-05 · • First photo: Joseph Nicephore Niepce (1822) • Daguerréotypes](https://reader034.vdocuments.site/reader034/viewer/2022050302/5f6b81c7bd906434df1961d9/html5/thumbnails/32.jpg)
Focus and depth of field
• Depth of field (DOF) is the distance between the nearest and farthest objects in a scene that appear acceptably sharp in an image.
• Although a lens can precisely focus at only one distance at a time, the decrease in sharpness is gradual on each side of the focused distance, so that within the DOF, the unsharpness is imperceptible under normal viewing conditions
Depth of field
![Page 33: Lecture 02 Image Formation - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2017/02_image_formation_1.pdf · 2017-10-05 · • First photo: Joseph Nicephore Niepce (1822) • Daguerréotypes](https://reader034.vdocuments.site/reader034/viewer/2022050302/5f6b81c7bd906434df1961d9/html5/thumbnails/33.jpg)
Focus and depth of field• How does the aperture affect the depth of field?
• A smaller aperture increases the range in which the object appears approximately in focus but reduces the amount of light into the camera
![Page 34: Lecture 02 Image Formation - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2017/02_image_formation_1.pdf · 2017-10-05 · • First photo: Joseph Nicephore Niepce (1822) • Daguerréotypes](https://reader034.vdocuments.site/reader034/viewer/2022050302/5f6b81c7bd906434df1961d9/html5/thumbnails/34.jpg)
• As f gets smaller, image becomes more wide angle
– more world points project onto the finite image plane
• As f gets larger, image becomes more narrow angle
– smaller part of the world projects onto the finite image plane
Field of view depends on focal length
![Page 35: Lecture 02 Image Formation - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2017/02_image_formation_1.pdf · 2017-10-05 · • First photo: Joseph Nicephore Niepce (1822) • Daguerréotypes](https://reader034.vdocuments.site/reader034/viewer/2022050302/5f6b81c7bd906434df1961d9/html5/thumbnails/35.jpg)
Field of view
Smaller FOV = larger Focal Length
![Page 36: Lecture 02 Image Formation - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2017/02_image_formation_1.pdf · 2017-10-05 · • First photo: Joseph Nicephore Niepce (1822) • Daguerréotypes](https://reader034.vdocuments.site/reader034/viewer/2022050302/5f6b81c7bd906434df1961d9/html5/thumbnails/36.jpg)
Field of view
Angular measure of portion of 3D space seen by the camera
![Page 37: Lecture 02 Image Formation - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2017/02_image_formation_1.pdf · 2017-10-05 · • First photo: Joseph Nicephore Niepce (1822) • Daguerréotypes](https://reader034.vdocuments.site/reader034/viewer/2022050302/5f6b81c7bd906434df1961d9/html5/thumbnails/37.jpg)
Outline of this lecture
• Image Formation
• Other camera parameters
• Digital camera
• Perspective camera model
• Lens distortion
![Page 38: Lecture 02 Image Formation - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2017/02_image_formation_1.pdf · 2017-10-05 · • First photo: Joseph Nicephore Niepce (1822) • Daguerréotypes](https://reader034.vdocuments.site/reader034/viewer/2022050302/5f6b81c7bd906434df1961d9/html5/thumbnails/38.jpg)
Digital cameras
• Film sensor array
• Often an array of charge coupled devices
• Each CCD/CMOS is light sensitive diode that converts photons (light energy) to electrons
![Page 39: Lecture 02 Image Formation - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2017/02_image_formation_1.pdf · 2017-10-05 · • First photo: Joseph Nicephore Niepce (1822) • Daguerréotypes](https://reader034.vdocuments.site/reader034/viewer/2022050302/5f6b81c7bd906434df1961d9/html5/thumbnails/39.jpg)
im[176][201] has value 164 im[194][203] has value 37
width 500j=1
height300
i=1Pixel Intensity with 8 bits ranges between [0,255]
Digital images
NB. Matlab coordinates: [rows, cols]; C/C++ [cols, rows]
![Page 40: Lecture 02 Image Formation - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2017/02_image_formation_1.pdf · 2017-10-05 · • First photo: Joseph Nicephore Niepce (1822) • Daguerréotypes](https://reader034.vdocuments.site/reader034/viewer/2022050302/5f6b81c7bd906434df1961d9/html5/thumbnails/40.jpg)
Color sensing in digital cameras
Bayer grid • The Bayer pattern (Bayer 1976) places green filters over half of the sensors (in a checkerboard pattern), and red and blue filters over the remaining ones.
• This is because the luminance signal is mostly determined by green values and the human visual system is much more sensitive to high frequency detail in luminance than in chrominance.
![Page 41: Lecture 02 Image Formation - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2017/02_image_formation_1.pdf · 2017-10-05 · • First photo: Joseph Nicephore Niepce (1822) • Daguerréotypes](https://reader034.vdocuments.site/reader034/viewer/2022050302/5f6b81c7bd906434df1961d9/html5/thumbnails/41.jpg)
Color sensing in digital cameras
Estimate missing components from neighboring values(demosaicing)
Bayer grid
Foveon chip design (http://www.foveon.com) stacks the red, green, and blue sensors beneath each other but has not gained widespread adoption.
![Page 42: Lecture 02 Image Formation - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2017/02_image_formation_1.pdf · 2017-10-05 · • First photo: Joseph Nicephore Niepce (1822) • Daguerréotypes](https://reader034.vdocuments.site/reader034/viewer/2022050302/5f6b81c7bd906434df1961d9/html5/thumbnails/42.jpg)
R G B
Color images:
RGB color space
… but there are also many other color spaces… (e.g., YUV)
![Page 43: Lecture 02 Image Formation - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2017/02_image_formation_1.pdf · 2017-10-05 · • First photo: Joseph Nicephore Niepce (1822) • Daguerréotypes](https://reader034.vdocuments.site/reader034/viewer/2022050302/5f6b81c7bd906434df1961d9/html5/thumbnails/43.jpg)
An example camera datasheet
![Page 44: Lecture 02 Image Formation - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2017/02_image_formation_1.pdf · 2017-10-05 · • First photo: Joseph Nicephore Niepce (1822) • Daguerréotypes](https://reader034.vdocuments.site/reader034/viewer/2022050302/5f6b81c7bd906434df1961d9/html5/thumbnails/44.jpg)
Outline of this lecture
• Image Formation
• Other camera parameters
• Digital camera
• Perspective camera model
• Lens distortion
![Page 45: Lecture 02 Image Formation - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2017/02_image_formation_1.pdf · 2017-10-05 · • First photo: Joseph Nicephore Niepce (1822) • Daguerréotypes](https://reader034.vdocuments.site/reader034/viewer/2022050302/5f6b81c7bd906434df1961d9/html5/thumbnails/45.jpg)
Perspective Camera
For convenience, the image plane is usually represented in front of C such that the image preserves the same orientation (i.e. not flipped)
Note: a camera does not measure distances but angles! a camera is a “bearing sensor”
C = optical center = center of the lens
Image plane
Pc
C
O
u
v
p
Zc
f
O = principal point
Zc = optical axis
Xc
Yc
![Page 46: Lecture 02 Image Formation - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2017/02_image_formation_1.pdf · 2017-10-05 · • First photo: Joseph Nicephore Niepce (1822) • Daguerréotypes](https://reader034.vdocuments.site/reader034/viewer/2022050302/5f6b81c7bd906434df1961d9/html5/thumbnails/46.jpg)
Find pixel coordinates (u,v) of point Pcin the camera frame:
1. Convert Pc to image-plane
coordinates (x,y)
2. Convert Pc to (discretised) pixel
coordinates (u,v)
0. Convert world point Pw to
camera point Pc
Pc
O
u
v
p
Xc
y
x
C
Zc
Yc
Find pixel coordinates (u,v) of point Pwin the world frame:
[R|T]
W
Zw
Yw
Xw
Pw
From World to Pixel coordinates
![Page 47: Lecture 02 Image Formation - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2017/02_image_formation_1.pdf · 2017-10-05 · • First photo: Joseph Nicephore Niepce (1822) • Daguerréotypes](https://reader034.vdocuments.site/reader034/viewer/2022050302/5f6b81c7bd906434df1961d9/html5/thumbnails/47.jpg)
Perspective Projection (1)
The Camera point Pc=( Xc , 0 , Zc )T projects to p=(x, y) onto the image plane
From similar triangles:
Similarly, in the general case:
c
c
c
c
Z
fXx
Z
X
f
x
Image Planef
Pc=( Xc , 0 , Zc )T
C
p
xXc
Xc
Zc
O
c
c
c
c
Z
fYy
Z
Y
f
y
1. Convert Pc to image-plane
coordinates (x,y)
2. Convert Pc to (discretised) pixel
coordinates (u,v)
From the Camera frame to the image plane
![Page 48: Lecture 02 Image Formation - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2017/02_image_formation_1.pdf · 2017-10-05 · • First photo: Joseph Nicephore Niepce (1822) • Daguerréotypes](https://reader034.vdocuments.site/reader034/viewer/2022050302/5f6b81c7bd906434df1961d9/html5/thumbnails/48.jpg)
Perspective Projection (2)
To convert p from the local image plane coords (x,y) to the pixel coords (u,v), we need to account for:
the pixel coords of the camera optical center
Scale factors for the pixel-size in both dimensions
So:
Use Homogeneous Coordinates for linear mapping from 3D to 2D, by introducing an extra element (scale):
),( 00 vuO
From the Camera frame to pixel coordinates
O
v
(0,0)u
(u0,v0) x
y p
Image plane
vu kk ,
c
cvv
c
cuu
Z
fYkvvykvv
Z
fXkuuxkuu
00
00
v
up
1~
~
~
~ v
u
w
v
u
p
![Page 49: Lecture 02 Image Formation - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2017/02_image_formation_1.pdf · 2017-10-05 · • First photo: Joseph Nicephore Niepce (1822) • Daguerréotypes](https://reader034.vdocuments.site/reader034/viewer/2022050302/5f6b81c7bd906434df1961d9/html5/thumbnails/49.jpg)
So:
Expressed in matrix form and homogenerous coordinates:
c
c
c
v
u
Z
Y
X
vfk
ufk
v
u
100
0
0
0
0
c
c
c
c
c
c
v
u
Z
Y
X
K
Z
Y
X
v
u
v
u
100
0
0
0
0
Or alternatively
c
cv
c
cu
Z
fYkvv
Z
fXkuu
0
0
Image plane (CCD)
Pc
C
O
u
v
p
Zc
f
Xc
Yc
Focal length in pixels
K is called “Calibration matrix” or “Matrix of Intrinsic Parameters”
Sometimes, it is common to assume a skew factor (𝐾12 ≠ 0) to account for possible misalignments between CCD and lens. However, the camera manufacturing process today is so good that we can safely assume 𝐾12 = 0 and 𝛼𝑢= 𝛼𝑣.
Perspective Projection (3)
![Page 50: Lecture 02 Image Formation - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2017/02_image_formation_1.pdf · 2017-10-05 · • First photo: Joseph Nicephore Niepce (1822) • Daguerréotypes](https://reader034.vdocuments.site/reader034/viewer/2022050302/5f6b81c7bd906434df1961d9/html5/thumbnails/50.jpg)
Exercise 1
• Determine the Intrinsic Parameter Matrix (K) for a digital camera with image size 640 × 480 pixels and horizontal field of view equal to 90°
• Assume the principal point in the center of the image and squared pixels
• What is the vertical field of view?
![Page 51: Lecture 02 Image Formation - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2017/02_image_formation_1.pdf · 2017-10-05 · • First photo: Joseph Nicephore Niepce (1822) • Daguerréotypes](https://reader034.vdocuments.site/reader034/viewer/2022050302/5f6b81c7bd906434df1961d9/html5/thumbnails/51.jpg)
Exercise 1
• Determine the Intrinsic Parameter Matrix (K) for a digital camera with image size 640 × 480 pixels and horizontal field of view equal to 90°
• Assume the principal point in the center of the image and squared pixels
• What is the vertical field of view?
𝑓 =640
2 tan𝜃2
= 320 𝒑𝒊𝒙𝒆𝒍𝒔
100
2403200
3200320
K
𝜃𝑉 = 2 tan−1𝐻
2𝑓= 2 tan−1
480
2 ∙ 320= 73.74°
![Page 52: Lecture 02 Image Formation - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2017/02_image_formation_1.pdf · 2017-10-05 · • First photo: Joseph Nicephore Niepce (1822) • Daguerréotypes](https://reader034.vdocuments.site/reader034/viewer/2022050302/5f6b81c7bd906434df1961d9/html5/thumbnails/52.jpg)
Exercise 2
• Prove that world’s parallel lines intersect at a vanishing point in the camera image
Vanishingpoint
Vanishingline
Vanishingpoint
Vertical vanishingpoint
(at infinity)
Slide from Efros, Photo from Criminisi
![Page 53: Lecture 02 Image Formation - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2017/02_image_formation_1.pdf · 2017-10-05 · • First photo: Joseph Nicephore Niepce (1822) • Daguerréotypes](https://reader034.vdocuments.site/reader034/viewer/2022050302/5f6b81c7bd906434df1961d9/html5/thumbnails/53.jpg)
Exercise 2
• Prove that world’s parallel lines intersect at a vanishing point in the camera image
• Let’s consider the perspective projection equation in calibrated coordinates:
• Two parallel 3D lines have parametric equations:
• Now substitute this into the camera perspective projection equation and compute the limit for 𝑠 → ∞
• The result solely depends on the direction vector of the line. These are the image coordinates of the vanishing point (VP).
• What is the intuitive interpretation of this?
n
m
l
s
Z
Y
X
Z
Y
X
0
0
0
Z
Yy
Z
Xx ,
n
m
l
s
Z
Y
X
Z
Y
X
1
1
1
VP
i
i
sVP
i
i
sy
n
m
snZ
smYx
n
l
snZ
slX
lim,lim
![Page 54: Lecture 02 Image Formation - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2017/02_image_formation_1.pdf · 2017-10-05 · • First photo: Joseph Nicephore Niepce (1822) • Daguerréotypes](https://reader034.vdocuments.site/reader034/viewer/2022050302/5f6b81c7bd906434df1961d9/html5/thumbnails/54.jpg)
Perspective Projection (4)
c
c
c
Z
Y
X
Kv
u
1
From the World frame to the Camera frame
Projection Matrix (M)
11
w
w
w
Z
Y
X
TRKv
u
3
2
1
333231
232221
131211
t
t
t
Z
Y
X
rrr
rrr
rrr
Z
Y
X
w
w
w
c
c
c
113333231
2232221
1131211
w
w
w
w
w
w
c
c
c
Z
Y
X
TRZ
Y
X
trrr
trrr
trrr
Z
Y
X
Pc
O
u
v
p
XcCZc
Yc[R|T]
Extrinsic Parameters
W
Zw
Yw
Xw
= Pw
Perspective Projection Equation
![Page 55: Lecture 02 Image Formation - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2017/02_image_formation_1.pdf · 2017-10-05 · • First photo: Joseph Nicephore Niepce (1822) • Daguerréotypes](https://reader034.vdocuments.site/reader034/viewer/2022050302/5f6b81c7bd906434df1961d9/html5/thumbnails/55.jpg)
Outline of this lecture
• Image Formation
• Other camera parameters
• Digital camera
• Perspective camera model
• Lens distortion
![Page 56: Lecture 02 Image Formation - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2017/02_image_formation_1.pdf · 2017-10-05 · • First photo: Joseph Nicephore Niepce (1822) • Daguerréotypes](https://reader034.vdocuments.site/reader034/viewer/2022050302/5f6b81c7bd906434df1961d9/html5/thumbnails/56.jpg)
Radial Distortion
PincushionBarrel distortionNo distortion
![Page 57: Lecture 02 Image Formation - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2017/02_image_formation_1.pdf · 2017-10-05 · • First photo: Joseph Nicephore Niepce (1822) • Daguerréotypes](https://reader034.vdocuments.site/reader034/viewer/2022050302/5f6b81c7bd906434df1961d9/html5/thumbnails/57.jpg)
Radial Distortion
• The standard model of radial distortion is a transformation from the ideal coordinates (𝑢, 𝑣) (i.e., undistorted) to the real observable coordinates (distorted) (𝑢𝑑 , 𝑣𝑑)
• The amount of distortion of the coordinates of the observed image is a nonlinear function of their radial distance . For most lenses, a simple quadratic model of distortion produces good results
where
• Depending on the amount of distortion (an thus on the camera field of view), higher order terms can be introduced:
𝑢𝑑𝑣𝑑
= 1 + 𝑘1𝑟2
𝑢 − 𝑢0𝑣 − 𝑣0
+𝑢0𝑣0
𝑟2= 𝑢 − 𝑢02+ 𝑣 − 𝑣0
2
𝑢𝑑𝑣𝑑
= 1 + 𝑘1𝑟2 + 𝑘2𝑟
4 + 𝑘3𝑟6
𝑢 − 𝑢0𝑣 − 𝑣0
+𝑢0𝑣0
𝑟2= 𝑢 − 𝑢02+ 𝑣 − 𝑣0
2
![Page 58: Lecture 02 Image Formation - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2017/02_image_formation_1.pdf · 2017-10-05 · • First photo: Joseph Nicephore Niepce (1822) • Daguerréotypes](https://reader034.vdocuments.site/reader034/viewer/2022050302/5f6b81c7bd906434df1961d9/html5/thumbnails/58.jpg)
• To recap, a 3D world point 𝑃 = 𝑋𝑤, 𝑌𝑤, 𝑍𝑤 projects into the image point 𝑝 =𝑢, 𝑣
where
and λ is the depth (λ = 𝑍𝐶) of the scene point
• If we want to take into account the radial distortion, then the distorted coordinates 𝑢𝑑, 𝑣𝑑 (in pixels) can be obtained as
where
• See also the OpenCV documentation:
Summary: Perspective projection equations
1
1~
~
~
~
w
w
w
Z
Y
X
TRKv
u
w
v
u
p
100
0
0
0
0
v
u
K
𝑢𝑑𝑣𝑑
= 1 + 𝑘1𝑟2
𝑢 − 𝑢0𝑣 − 𝑣0
+𝑢0𝑣0
𝑟2= 𝑢 − 𝑢02+ 𝑣 − 𝑣0
2
http://docs.opencv.org/2.4.13.3/modules/calib3d/doc/camera_calibration_and_3d_reconstruction.html
![Page 59: Lecture 02 Image Formation - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2017/02_image_formation_1.pdf · 2017-10-05 · • First photo: Joseph Nicephore Niepce (1822) • Daguerréotypes](https://reader034.vdocuments.site/reader034/viewer/2022050302/5f6b81c7bd906434df1961d9/html5/thumbnails/59.jpg)
Summary (things to remember)• Perspective Projection Equation
• Intrinsic and extrinsic parameters (K, R, t)
• Homogeneous coordinates
• Normalized image coordinates
• Image formation equations (including radial distortion)
• Chapter 4 of Autonomous Mobile Robot book