EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 1

Computational Plenoptic ImagingGordon Wetzstein1 Ivo Ihrke2 Douglas Lanman3 Wolfgang Heidrich1

1University of British Columbia 2Saarland University 3MIT Media Lab

Eurographics 2011 – State of the Art Report

VI. Multiplexing Time

EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 2

History – Eadweard Muybridge 1830-1904

http://en.wikipedia.org/wiki/Eadweard_Muybridge

EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 3

History – Étienne-Jules Marey 1830-1904

www.wikipedia.org

EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 4

VI.I Time Lapse Photography

EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 5

BBC Time Lapse – Look It!

ND 3.0 filter, f22, 1 minute exposure

• Long exposures to avoid temporal aliasing

EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 6

VI.II High-Speed Imaging

EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 7

High-Speed Cameras

Vision Research Phantom Flex (CMOS)2570 fps at HD resolution

Photron FASTCAM SA5 (CMOS)7500 fps at megapixel resolutionone million fps at 64x64 pixels

Casio Exilim Series (consumer cam)1000 fps at reduced resolution

Shimadzu HyperVision HPV-2 (CCD)one million fps at 312x260 pixels

EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 8

Assorted Pixels

[Narasimhan & Nayar 05]

EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 9

Temporal Mosaic with DMD

[Bub et al. 10]

DMD aligned with CCD in microscope

EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 10

Non-Destructive Sensor Readout & Pixim

Cypress Semiconductor LUPA 30003 megapixels, 485 fps

www.pixim.com

EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 11

Coded Rolling Shutter

[Gu et al. 10]

EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 12

Reinterpretable Imager

• Moving pinhole over time in aperture

• Capture with light field camera

[Agrawal et al. 10]

EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 13

Bullet Time Effect

from ‘The Matrix’

EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 14

Stanford Multi-Camera Array

[Wilburn et al. 04]

EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 15

Coded Temporal Sampling

[Agrawal et al. 10]

EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 16

High-Speed Illumination – Electronic Strobes

Harold ‘Doc’ Edgerton1903-1990

EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 17

Temporal Dithering with DLP Illumination

[Narasimhan et al. 08]

EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 18

Coded Strobing Photography

[Reddy et al. 11]

EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 19

Streak Cameras

www.hamamatsu.com

C5680$200K

EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 20

VI.I Motion Deblurring

EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 21

Motion Deblurring Overview

• Motion blur is velocity-dependent

• Can be described as convolution, where– Kernel shape is motion trajectory– Trajectory is modulated by exposure function

http://en.wikipedia.org/wiki/Motion_blur

EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 22

Deconvolution is Still Hard

• Again – problems:

– Camera noise

– Spatially varying kernel (velocity-dependent)

– Unknown motion trajectory

– Ill-posed problem, kernel of box integration function is not invertible (optical cancellation of image frequencies)

EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 23

Approaches to Improve Motion Deblurring

• Make PSF invertible coded exposure

• Make PSF velocity-invariant shift-invariant deconvolution

• Automatize PSF estimation

EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 24

Flutter Shutter

[Raskar et al. 06]

EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 25

Optimal Motion PSFs

• Optimality criteria PSF invertibility & estimation

[Agrawal & Xu 07]

EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 26

Motion Invariant Photography

• Engineer PSF to be motion invariant

• Only for 1D motion

[Levin 08]

EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 27

Hybrid Cameras

• Combined high-speed low-quality & low-speed high-quality camera

Input images Computed PSF

Deblurred Result Ground Truth

[Ben-Ezra & Nayar 04]

EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 28

Motion Blur in Video

• Coded exposure &

super-resolution in

successive video frames

[Agrawal et al. 09]

EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 29

Next: Further Light Properties

EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 30

Flexible Voxels

• Flexible space-time resolution as post-processing

[Gupta et al. 10]

EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 31

Synthetic Shutter Speed Imaging

• Combine multiple short exposures to reduce noise

• Align with optical flow

[Tel

leen

07]

EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 32

Hybrid Cameras

• Motion deblurring & super-resolution

[Tai

et a

l. 08

]

EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 33

Hybrid Cameras

• Motion deblurring & depth from two low-resolution high-speed camers

[Li e

t al.

08]

Input images

Deblurred result Recovered Depth

EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 34

Analysis

• Analysis of optimal coded, single image deblurring

• MIP becomes worse when velocities exceed expectations

[Agrawal & Raskar 09]

Coded Exposure Motion Invariant Photography