computational photography research at stanford - photography... · computational photography...

Download Computational photography research at Stanford - Photography... · Computational photography research

Post on 25-Aug-2018

222 views

Category:

Documents

1 download

Embed Size (px)

TRANSCRIPT

  • Computational photography research at Stanford

    Marc LevoyComputer Science DepartmentStanford University

    January 2012

  • ! Marc Levoy

    Outline

    ! Light field photography

    ! FCam and the Stanford Frankencamera

    ! User interfaces for computational photography

    ! Ongoing projects FrankenSLR burst-mode photography languages and architectures for programmable cameras computational videography and cinematography multi-bucket pixels

    2

  • Light field photography using a handheld plenoptic camera

    Ren Ng, Marc Levoy, Mathieu Brdif,Gene Duval, Mark Horowitz and Pat Hanrahan

    (Proc. SIGGRAPH 2005and TR 2005-02)

  • ! Marc Levoy

    Light field photography[Ng SIGGRAPH 2005]

  • Prototype camera

    4000 ! 4000 pixels 292 ! 292 lenses = 14 ! 14 pixels per lens

    Contax medium format camera Kodak 16-megapixel sensor

    Adaptive Optics microlens array 125 square-sided microlenses

  • ! 2010 Marc Levoy

    Digital refocusing

    !

    !

  • ! 2010 Marc Levoy

    Example of digital refocusing

  • ! 2010 Marc Levoy

    Refocusing portraits

  • ! 2010 Marc Levoy

    Light field photography

    (Flash demo)

  • trades off (excess) spatial resolution forability to refocus and adjust the perspective

    sensor pixels should be made even smaller, subject to the diffraction limit, for example: 36mm ! 24mm 2.5 pixels = 266 Mpix 20K ! 13K pixels 4000 ! 2666 pixels ! 20 ! 20 rays per pixelor 2000 ! 1500 pixels ! 3 ! 3 rays per pixel = 27 Mpix

    ! Marc Levoy

    Commercialization

  • Camera 2.0Marc Levoy

    Computer Science DepartmentStanford University

  • ! 2010 Marc Levoy

    ! facilitate research in experimental computational photography

    ! for students in computational photography courses worldwide

    ! proving ground for plugins and apps for future cameras

    The Stanford Frankencameras

    14

    Frankencamera F2 Nokia N900 F

  • ! 2010 Marc Levoy

    CS 448A - Computational Photography

    15

  • ! 2010 Marc Levoy

    CS 448 projectsM. Culbreth, A. Davis!! ! ! A phone-to-phone 4D light field fax machine

    J. Koberstein, A. Mattos!! ! Motion blur detection as a game controller N. Sakunkoo! ! ! ! ! ! ! Camera-assisted self portraits on mobile devices C. DuHadway!! ! ! ! ! ! Driving/parking assistive mosaics using car cameras C.-H. Chen, S.H. Park! ! ! High dynamic range (HDR) viewfinding V. Chandrasekhar, G. Takacs!High dynamic range imaging on the Nokia N95 L. Kim, S. Que ! ! ! ! User interfaces for adjusting focus and tone A.Y.L. Chan, S. Meador!! !Photography using circular/hemispherical ring flash P. Huang, Z. Zheng !! ! Image deblurring from multiple images M. Smith !! ! ! ! ! ! ! Representative (non-real) thumbnails for small screens A. Medearis ! ! ! ! ! ! Multi-shot noise reduction for sports photography D. Johnson !! ! ! ! ! ! Synthetic aperture photography using a painted aperture D. Chan !! ! ! ! ! ! Synthetic aperture rear view mirror for safer driving

    16

  • ! 2010 Marc Levoy

    Frankencamera architecture

    17

    Image Sensor

    Lens

    Metadata

    Actions

    Flash

    ...

    ApplicationProcessor

    Con!gure1

    Expose2

    Readout3

    Image Processing

    StatisticsCollection

    Imaging Processor

    4

    +

    Devices

    Requests

    Images andStatistics

  • ! 2010 Marc Levoy18

    Sensor sensor;Flash flash;vector burst(2);

    burst[0].exposure = 1/200.;burst[1].exposure = 1/30.;

    Flash::FireAction fire(&flash);fire.time = burst[0].exposure/2;burst[0].actions.insert(fire);

    sensor.stream(burst);

    while (1) { Frame flashFrame = sensor.getFrame(); Frame noflashFrame = sensor.getFrame();}

    Frankencamera software: the FCAM API

  • ! 2010 Marc Levoy

    Demonstration applications

    19

    Canon 430EX (smaller flash) strobed continuously

    Canon 580EX (larger flash) fired once at end of exposure

  • ! 2010 Marc Levoy

    Fcam is being actively ported to...

    ! Android (at least by NVIDIA)

    ! iOS?

    ! Nokia/Microsoft?

    ! Android point-and-shoot camera?

    20

  • ! Marc Levoy

    Prospects for computational photography

    ! the megapixel wars are over (and its about time)

    ! computational photography is the next battlegroundin the camera industry (its already starting)

    21

  • ! Marc Levoy

    ! this headroom permits more computation per pixel,or more frames per second, or less custom hardware

    22

    Premise: available computing power in cameras is rising faster than megapixels

    0

    3

    6

    9

    12

    2005 2006 2007 2008 2009 20100

    17.5

    35

    52.5

    70

    billi

    ons o

    f pix

    els /

    seco

    nd

    meg

    apix

    els

    Avg megapixels in new cameras, CAGR = 1.2NVIDIA GTX texture fill rate, CAGR = 1.8(CAGR for Moores law = 1.5)

  • ! Marc Levoy

    Interfaces for computational photography

    ! the megapixel wars are over (long overdue)

    ! computational photography is the next battlegroundin the camera industry (its already starting)

    ! how will these features appear to consumers? standard and invisible standard and visible (and disable-able) aftermarket plugins and apps for your camera

    23

  • ! Marc Levoy

    Interfaces for computational photography

    ! the megapixel wars are over (long overdue)

    ! computational photography is the next battlegroundin the camera industry (its already starting)

    ! how will these features appear to consumers? standard and invisible standard and visible (and disable-able) aftermarket plugins and apps for your camera

    ! but arent cameras too complicated already? wont computational photography make this worse?

    24

  • ! Marc Levoy

    Sony NEX-5! manual is 159 pages long

    and full of grammatical mistakes

    ! scene modes are not explained well Sunset: Shoots the red of the sunset beautifully. (page 43)

    ! user interface is difficult to navigate

    25

    1234567

    89

    10111213

  • ! Marc Levoy

    ProHDR for the iPhone 4S

    ! most popular HDR capture method on the market

    ! app developers are making photography easy again!26

    " 1 " " 2 "

  • 27

    Free

  • Free

    28

  • Free

    29

  • 30

    Free

    circa 500,000 downloads in

    2011

  • 31

    iPhone 4, single HD video frame

  • 32

    SynthCam, align & average ~30 frames

    SNR increases as sqrt(# of frames)

  • SynthCam

  • Canon 5DII, 1/400 sec

  • Canon 5DII, average of 15 shots of 1/6 sec

  • (Marc Levoy)

  • (Marc Levoy)

  • ! 2010 Marc Levoy

    Removing foreground objectsby translating the camera

    align the shots match histograms apply median filter

  • ! 2010 Marc Levoy

    Ongoing projects

    ! Frankencamera F3 (FrankenSLR) SLR-quality sensor, but big, heavy, and expensive ($10K)

    39

    Micron MT9001 cell phone quality $150

    Cypress LUPA 4000 DSLR quality $1500 arbitrary ROIs and non-destructive readout

  • ! 2010 Marc Levoy

    Ongoing projects

    ! Frankencamera F3 (FrankenSLR) SLR-quality sensor, but big, heavy, and expensive ($10K)

    ! high-speed burst-mode photography all still cameras should capture at (up to) 500 fps capture while aiming the camera - no shutter half-press frameless photography - ROIs, MOIs (M = Moment)

    40

  • ! 2010 Marc Levoy

    Burst-mode photography

    41

  • ! 2010 Marc Levoy

    Ongoing projects

    ! Frankencamera F3 (FrankenSLR) SLR-quality sensor, but big, heavy, and expensive ($10K)

    ! high-speed burst-mode photography all still cameras should capture at (up to) 500 fps capture while aiming the camera - no shutter half-press frameless photography - ROIs, MOIs (M = Moment)

    ! languages and architectures for programmable cameras FCam was an API for camera control now we need APIs for photographic processing and computer vision

    42

  • ! 2010 Marc Levoy

    Ongoing projects

    ! Frankencamera F3 (FrankenSLR) SLR-quality sensor, but big, heavy, and expensive ($10K)

    ! high-speed burst-mode photography all still cameras should capture at (up to) 500 fps capture while aiming the camera - no shutter half-press frameless photography - ROIs, MOIs (M = Moment)

    ! languages and architectures for programmable cameras FCam was an API for camera control now we need APIs for photographic processing and computer vision

    ! computational videography & cinematography stabilization, stereo, view interpolation, free-viewpoint video

    43

  • ! Marc Levoy

    3D video stabilization[Agarwala 2011]

    44 (http://web.cecs.pdx.edu/~fliu/project/subspace_stabilization/demo.mp4)

  • ! 2009 Marc Levoy

    Camera 3.0:multi-bucket pixels

    ! a CMOS pixel architecturewith multiple accumulation nodes per pixel

    ! can flip between TX1 and TX2 very fast all pixels flip at once

    45

    storage nodes

    photodetector

    TX2TX1

    transfer gates

  • ! 2009 Marc Levoy

    Flash/noflash photography[Agrawal SIGGRAPH 2005]

    46

    TX2TX1put flash exposure

    in middle of ambient exposure

    another app:HDR capture

    (with no differences in object motion

    between exposures)

  • !"#$%&'()*$+,-'.+"/

    Hig

Recommended

View more >