computational photography •image formatoin - portal · 1 computational photography matthias...
TRANSCRIPT
1
Computational Photography
Matthias ZwickerUniversity of Bern
Fall 2009
Today• Course organization
• Course overview
• Image formation
Course organizationInstructor
• Matthias Zwicker ([email protected])
Teaching Assistant
• To be announcedTo be announced
Course organizationLecture
• Mondays, 14:00-16:00, Engehaldenstr. 8, Hörsaal 001
Exercises
• Mondays, 16:00-17:00 , Engehaldenstr. 8, Hörsaal 001
Class web page• Schedule, slides, reading, project
descriptions, etc.http://www.cgg.unibe.ch/teaching/courses/herbstsemester-2009/computational-photography/
Web-based forum• On ILIAS
https://ilias.unibe.ch/ilias3/repository.php?cmd=frameset&ref_id=66993
• Use your campus account to log in
• Join group “IAM Computational Photography” with password Photography with password “cggcompphoto”
• Any questions and discussions related to class material and exercises
2
Exercises• Programming projects
– Matlab• Exercise series on paper
• Contribute 50% to final grade
• Late penalty• Late penalty– 50% of original score– Exceptions for military service, illness
• Collaboration– Discussion among student is encouraged– Each student must write up his/her own solution
Prerequisites• Familiarity with
– Linear algebra– Programming
Today• Course organization
• Course overview
• Image formation
PhotographyTraditionally
• „Measuring light“
• Optics focuses light on sensor
• Sensor records image
• Sensors
– Digital– Film
http://en.wikipedia.org/wiki/Single-lens_reflex_camera http://en.wikipedia.org/wiki/Digital_single-lens_reflex_camera
Computational photography• More than digital photography
• Arbitrary computation between light measurement and final image– Enhance and extend capabilities of digital
photographyh d f l – Light measured on sensor is not final image
• Two „types“ of computation– Post-process after traditional imaging– Design of new camera devices that require
computation to form an image• Overview of recent research
http://en.wikipedia.org/wiki/Computational_photography
Removing imaging artifacts• Denoising & deblurring
http://www.cs.ust.hk/~quan/publications/yuan-deblur-siggraph07.pdf
Blurry OutputNoisy
3
Removing imaging artifacts• High dynamic range images & tone mapping
Image manipulation• Panoramas
http://en.wikipedia.org/wiki/Image_stitching
Computational optics
Coded aperture
Captured image
Computational optics
Recovered depth
Refocused image
Focus of class• Algorithms and computational techniques
with potential applications in the consumer domain
– Mostly software, less hardware
• Recent research• Recent research
What you will learn• Basic understanding of photography, light, and
color
• Practical experience with implementation of algorithms for image processing & computational photography
• Cool and creative applications of mathematical Cool and creative applications of mathematical tools– Fourier transforms– Linear and non linear filtering– Optimization techniques (least squares, iteratively re-
weighted least squares, graph cuts)– Probabilistic models
• Lots of applications beyond processing images!
4
Related areas, not covered• Image processing for scientific applications
– Physics, biology, etc.
• Optics, lens design
• Sensors, sensor design
• Computational imaging
– Tomography, radar, microscopy
• 3D imaging
• Using photo processing tools, e.g. Photoshop
• Artistical aspects of photography
Syllabus1. Introduction, image formation
2. Color & color processing3. Dynamic range & contrast4. Sampling, reconstruction, & the frequency domain5. Image restoration: denoising & deblurring6. Image manipulation using optimization7. Gradient domain image manipulation7. Gradient domain image manipulation
8. Warping & morphing9. Panoramas10. Automatic alignment11. Probabilistic image models12. Light fields13. Capturing light transport
http://www.cgg.unibe.ch/teaching/courses/herbstsemester-2009/computational-photography
• Cameras, image artifacts
Image formation Color• Color perception, color spaces, color
measurement, color processing
Dynamic range & contrast• HDR imaging
http://en.wikipedia.org/wiki/High_dynamic_range_imaginghttp://en.wikipedia.org/wiki/Tone_mapping
Sampling, reconstruction• Sampling artifacts
• Frequency domainanalysis
Spatial Domain Frequency Domain
5
Image restoration• Denoising & deblurring
Blurry input Deblurred output
Estimated blur kernel (scaled)http://vision.ucsd.edu/kriegman-grp/research/psf_estimation/
Image manipulation using optimization• Photomontage, matting, colorization
http://grail.cs.washington.edu/projects/photomontage/
http://www.cs.huji.ac.il/~yweiss/Colorization/http://grail.cs.washington.edu/projects/digital-matting/image-matting/
Gradient domain manipulation• Poisson equation
http://portal.acm.org/citation.cfm?id=882269
Warping & morphing
Panoramas• Automatic alignment, stitching
http://www.cs.cmu.edu/afs/andrew/scs/cs/15-463/f07/proj4/www/wwedler/
Probabilistic models• Faces, textures
http://web4.cs.ucl.ac.uk/staff/j.kautz/publications/Visio_SIG09.pdf
6
• Beyond 2D images
Light fields
http://www-graphics.stanford.edu/papers/fourierphoto/
Capturing light transport• Dual photography
http://www-graphics.stanford.edu/papers/dual_photography/
Today• Course organization
• Course overview
• Image formation
Question• Why is there no image on a white piece of
paper?
Question• Why is there no image on a white piece of
paper?
• Receives all light rays
– Images from all viewpoints
• Need to select lightrays for specificeimage, viewpoint
• How?
• Invented by Alhazen, 10th centuryhttp://en.wikipedia.org/wiki/Pinhole_camera
Pinhole camera
7
Limitations• Small pinhole: sharper image, longer exposure
• Larger pinhole: blurrier image, shorter exposure
Camera model• Thin lens, aperture, shutter, film
Lenses• Gather more light
• Use refraction
• Need to be focused
Lens
http://en.wikipedia.org/wiki/Lens_(optics)
Scene point(emits or
reflects light)Image of
scene point
LensesPinhole Lens
6 sec. exposure 0.01 sec exposure
Thin lens model• Theoretical model for well-behaved lenses
• Properties
– All parallel rays converge at focal lengthlength
– Rays through the center are not deflectedSame perspective imageas pinhole at centerof lens
Thin lens model• How are arbitrary rays deflected when
passing through a thin lens?
8
Thin lens model Thin lens model• Similar triangles
Thin lens model• More similar triangles
Thin lens model• Thin lens formula
• All rays passing through a single point on a plane at distance in front of the lens will pass through a single point at distance behind the lensbehind the lens
Thin lens model• Focus at infinity:
Film plane
• Closest focusing distance:
Object
Thin lens model• Out of focus film plane results in spherical
blur
Out of focus film planes
Spherical blur
9
Properties of real lenses• Mostly undesired!
• Aberrations
– Spherical aberration– Chromatic aberration
• Distortion
– Barrel distortion– Pincushion distortion
• Etc.
Barrel & pincushion distortion
Camera model• Thin lens, aperture, shutter, film
Aperture• Blurriness of out of focus objects depends
on aperture size
• Aperture size determines depth of field:depth range that is sharp in image
Aperture
Depth of field
Circle of confusion• Also called „blur circle“
• Calculation of radius c
– Lens focused at S1
– Object at S2
– Aperture A– Focal length f sensor
http://en.wikipedia.org/wiki/Circle_of_confusion
Proportional to A
f-number• Fraction of focal length over diameter of
aperture
• Large aperture means small f-number
• Practice: f-stops increase by factors of√2
– f/2.0, f/2.8, f/4, f/5.6, f/8– Aperture area gets halved in each step
10
Camera model• Thin lens, aperture, shutter, film
Shutter• Determines time the film is exposed to
light
• Amount of light captured is proportional to exposure time
• Long exposure leads to motion blur• Long exposure leads to motion blur
Reciprocity• Amount of light captured stays
same if exposure is doubled and aperture area is halved (or vice versa)
Reciprocity• Which exposure/aperture combination?
Film• Film/sensor responds roughly linearly to light
– „Double the amount of light leads to double the recorded value“
• Film speed: sensitivity of film to light– Digital photography analog: sensor gain– Scaling factor
• Measured using ISO scale– Linear: sensitivity is proportional to ISO value– „Double ISO value, halve the exposure time“
Film• Trade-off: higher gain, more noise
ISO 100 ISO 3200
11
Film• Underexposure
– Not enough light, image too dark• Overexposure
– Film or sensor is saturated– Clipping of highlight details
Good exposure OverexposureUnderexposure
Conclusions• Simple camera model
– Thin lens, aperture, shutter, film
• Photographs often have undesired artifacts
– Distortions, color artifacts, blur, noise, d /under/overexposure
Goal
• Develop algorithms to remove artifacts after image is captured
References• „Photography“, by London, Upton, Stone
Next time• Color, color processing