computational photography - computer action teamweb.cecs.pdx.edu/~lusi/cs510/slides/lecture2.pdf ·...

61
Computational Photography Si Lu Spring 2018 http://web.cecs.pdx.edu/~lusi/CS510/CS510_Computati onal_Photography.htm 04/05/2018

Upload: others

Post on 20-May-2020

76 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Computational Photography - Computer Action Teamweb.cecs.pdx.edu/~lusi/CS510/slides/Lecture2.pdf · n Computational Photography 2. Today o Digital Camera n History of Camera n Controlling

Computational Photography

Si Lu

Spring 2018

http://web.cecs.pdx.edu/~lusi/CS510/CS510_Computational_Photography.htm

04/05/2018

Page 2: Computational Photography - Computer Action Teamweb.cecs.pdx.edu/~lusi/CS510/slides/Lecture2.pdf · n Computational Photography 2. Today o Digital Camera n History of Camera n Controlling

Last Time

o Course overviewn Admin. Infon Computational Photography

2

Page 3: Computational Photography - Computer Action Teamweb.cecs.pdx.edu/~lusi/CS510/slides/Lecture2.pdf · n Computational Photography 2. Today o Digital Camera n History of Camera n Controlling

Today

o Digital Cameran History of Cameran Controlling Camera

o Photography Concepts

3

Page 4: Computational Photography - Computer Action Teamweb.cecs.pdx.edu/~lusi/CS510/slides/Lecture2.pdf · n Computational Photography 2. Today o Digital Camera n History of Camera n Controlling

Pinhole-Camera

4

o The first cameran 5th B.C. Aristotle, Mozi (Chinese: 墨子) n How does the aperture size affect the image?

http://en.wikipedia.org/wiki/Pinhole_cameraModified from slides provided by L. Zhang

aperture

Page 5: Computational Photography - Computer Action Teamweb.cecs.pdx.edu/~lusi/CS510/slides/Lecture2.pdf · n Computational Photography 2. Today o Digital Camera n History of Camera n Controlling

Shrinking the aperture

5Slide credit: L. Zhang

Page 6: Computational Photography - Computer Action Teamweb.cecs.pdx.edu/~lusi/CS510/slides/Lecture2.pdf · n Computational Photography 2. Today o Digital Camera n History of Camera n Controlling

Shrinking the aperture

6

o Why not make the aperture as small as possible o Less light gets through o Diffraction effects

Slide credit: L. Zhang

Page 7: Computational Photography - Computer Action Teamweb.cecs.pdx.edu/~lusi/CS510/slides/Lecture2.pdf · n Computational Photography 2. Today o Digital Camera n History of Camera n Controlling

Shrinking the aperture

7Slide credit: L. Zhang

Page 8: Computational Photography - Computer Action Teamweb.cecs.pdx.edu/~lusi/CS510/slides/Lecture2.pdf · n Computational Photography 2. Today o Digital Camera n History of Camera n Controlling

First production camera?

o 1839. Daguerrotype

Slide credit: F. Durand

Page 9: Computational Photography - Computer Action Teamweb.cecs.pdx.edu/~lusi/CS510/slides/Lecture2.pdf · n Computational Photography 2. Today o Digital Camera n History of Camera n Controlling

Beginning of hobby photography?

o 1900 Kodak Brownie

Slide credit: F. Durand

Page 10: Computational Photography - Computer Action Teamweb.cecs.pdx.edu/~lusi/CS510/slides/Lecture2.pdf · n Computational Photography 2. Today o Digital Camera n History of Camera n Controlling

Color photographyo Who did the first color photography? n Maxwell

(yes, the same from the EM equations)

o When? 1861o Oldest color photos still preserved:

Prokudin-Gorskii http://www.loc.gov/exhibits/empire/

Slide credit: F. Durand

Page 11: Computational Photography - Computer Action Teamweb.cecs.pdx.edu/~lusi/CS510/slides/Lecture2.pdf · n Computational Photography 2. Today o Digital Camera n History of Camera n Controlling

Prokudin-Gorskii

o Digital restoration

http://www.loc.gov/exhibits/empire/

Slide credit: F. Durand

Page 12: Computational Photography - Computer Action Teamweb.cecs.pdx.edu/~lusi/CS510/slides/Lecture2.pdf · n Computational Photography 2. Today o Digital Camera n History of Camera n Controlling

Prokudin-Gorskii

Slide credit: F. Durand

Page 13: Computational Photography - Computer Action Teamweb.cecs.pdx.edu/~lusi/CS510/slides/Lecture2.pdf · n Computational Photography 2. Today o Digital Camera n History of Camera n Controlling

Prokudin-Gorskii

Slide credit: F. Durand

Page 14: Computational Photography - Computer Action Teamweb.cecs.pdx.edu/~lusi/CS510/slides/Lecture2.pdf · n Computational Photography 2. Today o Digital Camera n History of Camera n Controlling

Deep Learning?

14

Zhang, Richard, Phillip Isola, and Alexei A. Efros. "Colorful image colorization." European Conference on Computer Vision. Springer, Cham, 2016.

Page 15: Computational Photography - Computer Action Teamweb.cecs.pdx.edu/~lusi/CS510/slides/Lecture2.pdf · n Computational Photography 2. Today o Digital Camera n History of Camera n Controlling

Flash bulb?

o As opposed to powder systemso Boutan-Chauffour - 1893 o For underwater photography

Slide credit: F. Durand

Page 16: Computational Photography - Computer Action Teamweb.cecs.pdx.edu/~lusi/CS510/slides/Lecture2.pdf · n Computational Photography 2. Today o Digital Camera n History of Camera n Controlling

Instant photography?

o 1947, Edwin Land (Polaroid founder)

Slide credit: F. Durand

Page 17: Computational Photography - Computer Action Teamweb.cecs.pdx.edu/~lusi/CS510/slides/Lecture2.pdf · n Computational Photography 2. Today o Digital Camera n History of Camera n Controlling

Autofocus

o 1978, Konica

o 1981 Pentax ME-F.

o Canon T80 1985n Canon AL1 had focus assist but no actuator

o Minolta Maxxum 1985 (AF in body)

Slide credit: F. Durand

Page 18: Computational Photography - Computer Action Teamweb.cecs.pdx.edu/~lusi/CS510/slides/Lecture2.pdf · n Computational Photography 2. Today o Digital Camera n History of Camera n Controlling

First microprocessor in a camera

o Canon AE-1 1976

Slide credit: F. Durand

Page 19: Computational Photography - Computer Action Teamweb.cecs.pdx.edu/~lusi/CS510/slides/Lecture2.pdf · n Computational Photography 2. Today o Digital Camera n History of Camera n Controlling

First scanned photo?

o 1957, Russell A. Kirsch of the National Bureau of Standards, 176x176

Slide credit: F. Durand

Page 20: Computational Photography - Computer Action Teamweb.cecs.pdx.edu/~lusi/CS510/slides/Lecture2.pdf · n Computational Photography 2. Today o Digital Camera n History of Camera n Controlling

CCD technology?

o 1969, Willard S. Boyle and George E. Smith, Bell Laboratories

Slide credit: F. Durand

Page 21: Computational Photography - Computer Action Teamweb.cecs.pdx.edu/~lusi/CS510/slides/Lecture2.pdf · n Computational Photography 2. Today o Digital Camera n History of Camera n Controlling

CCD in astronomy

o 1979, 1-meter telescope at Kitt Peak National Observatory, 320x512, great for dim light

o Nitrogen cooled

Slide credit: F. Durand

Page 22: Computational Photography - Computer Action Teamweb.cecs.pdx.edu/~lusi/CS510/slides/Lecture2.pdf · n Computational Photography 2. Today o Digital Camera n History of Camera n Controlling

Computer Graphics?

Computers to create imageo Sketchpad, 1961, Ivan Sutherland

Slide credit: F. Durand

Turing Awardeeat PSU now

Page 23: Computational Photography - Computer Action Teamweb.cecs.pdx.edu/~lusi/CS510/slides/Lecture2.pdf · n Computational Photography 2. Today o Digital Camera n History of Camera n Controlling

Paint programo Dick Shoup: SuperPaint [1972-73]

n 8 bitsn http://www.rgshoup.com/prof/SuperPaint/

o Alvy Ray Smith (Pixar co-founder): Paint [1975-77] n 8 bits then 24 bitsn http://www.alvyray.com/Awards/

AwardsMain.htmn http://www.alvyray.com/Bio/BioMain.htm

o Tom Porter: Paint

Slide credit: F. Durand

Page 24: Computational Photography - Computer Action Teamweb.cecs.pdx.edu/~lusi/CS510/slides/Lecture2.pdf · n Computational Photography 2. Today o Digital Camera n History of Camera n Controlling

Photoshop

o Thomas Knoll and John Knoll began development in 1987

o Version 1.0 on Mac: 1990o http://en.wikipedia.org/wiki/Photoshop#Developmento http://www.storyphoto.com/multimedia/multimedia_photoshop.html

Slide credit: F. Durand

Page 25: Computational Photography - Computer Action Teamweb.cecs.pdx.edu/~lusi/CS510/slides/Lecture2.pdf · n Computational Photography 2. Today o Digital Camera n History of Camera n Controlling

First digital camera?o 1975, Steve Sasson, Kodako Uses ccd from Fairchild semiconductor, A/D from Motorola, .01

megapixels, 23 second exposure, recorded on digital cassette

Slide credit: F. Durand

Page 26: Computational Photography - Computer Action Teamweb.cecs.pdx.edu/~lusi/CS510/slides/Lecture2.pdf · n Computational Photography 2. Today o Digital Camera n History of Camera n Controlling

Completely Digital Commercial camera

o 1991 first completely digital Logitech Dycam 376x240

http://www.g4tv.com/

Slide credit: F. Durand

Page 27: Computational Photography - Computer Action Teamweb.cecs.pdx.edu/~lusi/CS510/slides/Lecture2.pdf · n Computational Photography 2. Today o Digital Camera n History of Camera n Controlling

Digital

o 1994 Apple quicktake, first mass-market color digital camera, 640 x 480 (commercial failure)

Slide credit: F. Durand

Page 28: Computational Photography - Computer Action Teamweb.cecs.pdx.edu/~lusi/CS510/slides/Lecture2.pdf · n Computational Photography 2. Today o Digital Camera n History of Camera n Controlling

Digital SLR?

o 1992 Kodak DCS 200, 1.5 Mpixels, based on Nikon body

Slide credit: F. Durand

Page 29: Computational Photography - Computer Action Teamweb.cecs.pdx.edu/~lusi/CS510/slides/Lecture2.pdf · n Computational Photography 2. Today o Digital Camera n History of Camera n Controlling

Consumer digital SLR?

o Canon D30, 2000 3MPixels

Slide credit: F. Durand

Page 30: Computational Photography - Computer Action Teamweb.cecs.pdx.edu/~lusi/CS510/slides/Lecture2.pdf · n Computational Photography 2. Today o Digital Camera n History of Camera n Controlling

Camera phone?

o In November 2000 Sharp and J-Phone introduced the first camera-phone in Japan

Slide credit: F. Durand

Page 31: Computational Photography - Computer Action Teamweb.cecs.pdx.edu/~lusi/CS510/slides/Lecture2.pdf · n Computational Photography 2. Today o Digital Camera n History of Camera n Controlling

Outline

o History of Camerao Controlling Camera

31

Page 32: Computational Photography - Computer Action Teamweb.cecs.pdx.edu/~lusi/CS510/slides/Lecture2.pdf · n Computational Photography 2. Today o Digital Camera n History of Camera n Controlling

Camera specifics

o Focal lengtho Shuttero Apertureo Reciprocityo Depth of field (focal)o Motiono ISOo Metering

32

Page 33: Computational Photography - Computer Action Teamweb.cecs.pdx.edu/~lusi/CS510/slides/Lecture2.pdf · n Computational Photography 2. Today o Digital Camera n History of Camera n Controlling

Pinhole imaging

Slide credit: F. Durand

Page 34: Computational Photography - Computer Action Teamweb.cecs.pdx.edu/~lusi/CS510/slides/Lecture2.pdf · n Computational Photography 2. Today o Digital Camera n History of Camera n Controlling

Focal length: pinhole optics

pinhole

Film/sensor

pinhole

Film/sensor scene

f d

s

Slide credit: F. Durand

Page 35: Computational Photography - Computer Action Teamweb.cecs.pdx.edu/~lusi/CS510/slides/Lecture2.pdf · n Computational Photography 2. Today o Digital Camera n History of Camera n Controlling

Field of View 24mm

50mm

135mm

Slide credit: F. Durand

Page 36: Computational Photography - Computer Action Teamweb.cecs.pdx.edu/~lusi/CS510/slides/Lecture2.pdf · n Computational Photography 2. Today o Digital Camera n History of Camera n Controlling

Focal length: pinhole optics

o What happens when the focal length is doubled?n Projected object sizen Amount of light gathered

pinhole

Film/sensor scene

f d

s

2f

Slide credit: F. Durand

Page 37: Computational Photography - Computer Action Teamweb.cecs.pdx.edu/~lusi/CS510/slides/Lecture2.pdf · n Computational Photography 2. Today o Digital Camera n History of Camera n Controlling

Perspective vs. viewpointo Focal lens does NOT ONLY change subject sizeo Same size by moving the viewpointo Different perspective (e.g. background)

Slide credit: F. Durand

Page 38: Computational Photography - Computer Action Teamweb.cecs.pdx.edu/~lusi/CS510/slides/Lecture2.pdf · n Computational Photography 2. Today o Digital Camera n History of Camera n Controlling

Perspective vs. viewpointo Telephoto makes it easier to select

background (a small change in viewpoint is a big change in background.)

Slide credit: F. Durand

Page 39: Computational Photography - Computer Action Teamweb.cecs.pdx.edu/~lusi/CS510/slides/Lecture2.pdf · n Computational Photography 2. Today o Digital Camera n History of Camera n Controlling

Perspective vs. viewpointo Portrait: distortion with wide angleo Why?

Wide angle Standard Telephoto

Slide credit: F. Durand

Page 40: Computational Photography - Computer Action Teamweb.cecs.pdx.edu/~lusi/CS510/slides/Lecture2.pdf · n Computational Photography 2. Today o Digital Camera n History of Camera n Controlling

Shuttero Most of the time, the film/sensor is protected from lighto When we take a picture, the shutter opens and closes, thereby

exposing the film. o Exposure is proportional to

the time the shutter is openo Expressed in fraction of a second

(1/60s, 1/125s, 1/250s, 1/500s, etc.)shutter

Slide credit: F. Durand

Page 41: Computational Photography - Computer Action Teamweb.cecs.pdx.edu/~lusi/CS510/slides/Lecture2.pdf · n Computational Photography 2. Today o Digital Camera n History of Camera n Controlling

Effect of shutter speedo Longer shutter speed => more light, but more motion bluro Faster shutter speed freezes motion

Slide credit: F. Durand

Page 42: Computational Photography - Computer Action Teamweb.cecs.pdx.edu/~lusi/CS510/slides/Lecture2.pdf · n Computational Photography 2. Today o Digital Camera n History of Camera n Controlling

Effect of shutter speedo Longer shutter speed => more light, but more motion bluro Faster shutter speed freezes motion

Slide credit: F. Durand

Page 43: Computational Photography - Computer Action Teamweb.cecs.pdx.edu/~lusi/CS510/slides/Lecture2.pdf · n Computational Photography 2. Today o Digital Camera n History of Camera n Controlling

Effect of shutter speed

o Freezing motion

1/125 1/250 1/500 1/1000

Walking people Running people Car Fast train

Slide credit: F. Durand

Page 44: Computational Photography - Computer Action Teamweb.cecs.pdx.edu/~lusi/CS510/slides/Lecture2.pdf · n Computational Photography 2. Today o Digital Camera n History of Camera n Controlling

Shutter speed and focal length

o Because telephoto “magnify”, they also magnify your hand shaking

o Telephotos therefore require faster shutter speedo Rule of thumb: n The slowest shutter speed where normal human can

hand-hold and get a sharp picture is 1/fn E.g., a 500mm requires 1/500 s or higher.

o Solution: Image stabilization n mechanically compensates for vibrationn Can gain 2 or 3 shutter speeds (1/125 or 1/60 for a

500mm)

Slide credit: F. Durand

Page 45: Computational Photography - Computer Action Teamweb.cecs.pdx.edu/~lusi/CS510/slides/Lecture2.pdf · n Computational Photography 2. Today o Digital Camera n History of Camera n Controlling

Your best friend

o Use a tripod! It will always enhance sharpness

Slide credit: F. Durand

Page 46: Computational Photography - Computer Action Teamweb.cecs.pdx.edu/~lusi/CS510/slides/Lecture2.pdf · n Computational Photography 2. Today o Digital Camera n History of Camera n Controlling

Exposure

o Two main parameters: n Shutter speedn Aperture (in f stop)

n Shutter speed (in fraction of a second)

Slide credit: F. Durand

Page 47: Computational Photography - Computer Action Teamweb.cecs.pdx.edu/~lusi/CS510/slides/Lecture2.pdf · n Computational Photography 2. Today o Digital Camera n History of Camera n Controlling

Apertureo Diameter of the lens opening (controlled by diaphragm)o Expressed as a fraction of focal length, in f-number

n f/2.0 on a 50mm means that the aperture is 25mmn f/2.0 on a 100mm means that the aperture is 50mm

o Disconcerting: small f number = big apertureo What happens to the area of the aperture when going from

f/2.0 to f/4.0?o Typical f numbers are

f/2.0, f/2.8, f/4, f/5.6, f/8, f/11, f/16, f/22, f/32

Slide credit: F. Durand

Page 48: Computational Photography - Computer Action Teamweb.cecs.pdx.edu/~lusi/CS510/slides/Lecture2.pdf · n Computational Photography 2. Today o Digital Camera n History of Camera n Controlling

Exposureo Two main parameters: n Aperture (in f stop)n Shutter speed (in fraction of a second)

o Reciprocityn The same exposure is obtained with an exposure

twice as long and an aperture area half as bign Reciprocity can fail for very long exposures

Slide credit: F. Durand

Page 49: Computational Photography - Computer Action Teamweb.cecs.pdx.edu/~lusi/CS510/slides/Lecture2.pdf · n Computational Photography 2. Today o Digital Camera n History of Camera n Controlling

Exposure & meteringo The camera metering system measures how bright the

scene iso In Aperture priority mode, the photographer sets the

aperture, the camera sets the shutter speedo In Shutter-speed priority mode, the photographers sets

the shutter speed and the camera deduces the aperturen In both cases, reciprocity is exploited

o In Program mode, the camera decides both exposure and shutter speed (middle value more or less)

o In Manual, the user decides everything (but can get feedback)

Slide credit: F. Durand

Page 50: Computational Photography - Computer Action Teamweb.cecs.pdx.edu/~lusi/CS510/slides/Lecture2.pdf · n Computational Photography 2. Today o Digital Camera n History of Camera n Controlling

Pros and cons of various modeso Aperture priority

n Direct depth of field controln Cons: can require impossible shutter speed (e.g. with f/1.4 for a

bright scene)

o Shutter speed priorityn Direct motion blur controln Cons: can require impossible aperture (e.g. when requesting a

1/1000 speed for a dark scene)o Note that aperture is somewhat more restricted

o Programn Almost no control, but no need for neurons

o Manualn Full control, but takes more time and thinking

Slide credit: F. Durand

Page 51: Computational Photography - Computer Action Teamweb.cecs.pdx.edu/~lusi/CS510/slides/Lecture2.pdf · n Computational Photography 2. Today o Digital Camera n History of Camera n Controlling

Meteringo Photosensitive sensors measure scene luminanceo Most cameras then use a center-weighted average

n Can fail if scenes are very white or very blackn Nikon has a more advanced system (3D matrix)

Slide credit: F. Durand

Page 52: Computational Photography - Computer Action Teamweb.cecs.pdx.edu/~lusi/CS510/slides/Lecture2.pdf · n Computational Photography 2. Today o Digital Camera n History of Camera n Controlling

Main effect of aperture

o Depth of field

From Photography, London et al. Slide credit: F. Durand

Page 53: Computational Photography - Computer Action Teamweb.cecs.pdx.edu/~lusi/CS510/slides/Lecture2.pdf · n Computational Photography 2. Today o Digital Camera n History of Camera n Controlling

Depth of field

lenssensor

Point in focus

Object with texture

Slide credit: F. Durand

Page 54: Computational Photography - Computer Action Teamweb.cecs.pdx.edu/~lusi/CS510/slides/Lecture2.pdf · n Computational Photography 2. Today o Digital Camera n History of Camera n Controlling

Depth of field

lenssensor

Point in focus

Object with texture

o We allow for some tolerance

lenssensor

Point in focus

Object with texture

Max acceptable circle of confusion

Depth of field

Depth of focus

Slide credit: F. Durand

Page 55: Computational Photography - Computer Action Teamweb.cecs.pdx.edu/~lusi/CS510/slides/Lecture2.pdf · n Computational Photography 2. Today o Digital Camera n History of Camera n Controlling

Slide credit: F. Durand

Page 56: Computational Photography - Computer Action Teamweb.cecs.pdx.edu/~lusi/CS510/slides/Lecture2.pdf · n Computational Photography 2. Today o Digital Camera n History of Camera n Controlling

Depth of fieldo What happens when we close the aperture by two stop?

n Aperture diameter is divided by twon Depth of field is doubled

lenssensor

Point in focus

Object with texture

Diaphragm

Slide credit: F. Durand

Page 57: Computational Photography - Computer Action Teamweb.cecs.pdx.edu/~lusi/CS510/slides/Lecture2.pdf · n Computational Photography 2. Today o Digital Camera n History of Camera n Controlling

Depth of field

From Photography, London et al. Slide credit: F. Durand

Page 58: Computational Photography - Computer Action Teamweb.cecs.pdx.edu/~lusi/CS510/slides/Lecture2.pdf · n Computational Photography 2. Today o Digital Camera n History of Camera n Controlling

o What happens when we divide focusing distance by two?n Similar triangles => divided by two as well

lenssensor

Point in focus

Half depth of field

Half depth of field

Depth of field & focusing distance

Slide credit: F. Durand

Page 59: Computational Photography - Computer Action Teamweb.cecs.pdx.edu/~lusi/CS510/slides/Lecture2.pdf · n Computational Photography 2. Today o Digital Camera n History of Camera n Controlling

o What happens when we divide focusing distance by two?n Similar triangles => divided by two as well

From Photography, London et al.

Depth of field & focusing distance

Slide credit: F. Durand

Page 60: Computational Photography - Computer Action Teamweb.cecs.pdx.edu/~lusi/CS510/slides/Lecture2.pdf · n Computational Photography 2. Today o Digital Camera n History of Camera n Controlling

Sensitivity (ISO)o Third variable for exposureo Linear effect (200 ISO needs half the light as 100 ISO)o Film photography: trade sensitivity for grain

o Digital photography: trade sensitivity for noise From dpreview.com

Slide credit: Y. Chuang

Page 61: Computational Photography - Computer Action Teamweb.cecs.pdx.edu/~lusi/CS510/slides/Lecture2.pdf · n Computational Photography 2. Today o Digital Camera n History of Camera n Controlling

Next Time

o TBD

61