raskar cosi invited talk oct 2009

Raskar, Camera Culture, MIT Media Lab

Camera Culture

Ramesh Raskar

Camera CultureAssociate Professor, MIT Media Lab

Computational Photography

http://raskar.info

Invertible Motion Blur in Video

Photo 1 Photo 2 Photo 3 Deblurring

Agrawal, Xu, Raskar, Siggraph 2009

Traditional Exposure Video

DFTMotion PSF (Box Filter)

Information is lost

Exposure Time

Coded Exposure (Flutter Shutter) Raskar et al. 2006

Single Photo Deblurred Image

Varying Exposure Video

DFTExposure Time


DFTExposure Time

Exposure Time


DFT

Exposure Time

Exposure Time

Exposure Time

Varying Exposure Video == PSF Null-Filling

DFT

Joint Frequency Spectrum

Preserves High Frequencies

Varying Exposure Video: Exploit auto-exposure mode

Completely automatic: (i) Segmentation, (ii) PSF estimation, (iii) deblurring

Blurred Photo

Deblurred Result Ground Truth

Input Photos

SynthesisLow Level Mid Level High

LevelHyper

realism

Raw

Angle, spectrum

aware

Non-visual Data, GPS

Metadata

Priors

Comprehensive

8D reflectance field

Digital

Epsilon

Coded

Essence

Computational Photography aims to make progress on

both axis

Camera ArrayHDR, FoV Focal stack

Decomposition problems

Depth

Spectrum

LightFields

Human Stereo Vision

Transient Imaging

Virtual Object Insertion

Relighting

Augmented Human

Experience

Material editing from single photo

Scene completion from photos

Motion Magnification

Phototourism

Resolution

Computational Photography (vs Imaging)

MIT Media Lab Camera Culture

Lots of glass; Heavy; Bulky; Expensive

Enhanced Defocus Blur


Image Destabilization

Lens Sensor

Camera

Static Scene


Image Destabilization

Static Scene

Lens Motion Sensor Motion

Camera

Mohan, Lanman,Hiura, Raskar ICCP 2009


“Time Lens”

Lens Equation:

Virtual Focal Length:

Virtual F-Number:

Ratio of speeds


Adjusting the Focus Plane

all-in-focus image



focused in the front using destabilization



focused in the middle using destabilization



focused in the back using destabilization


Bokode


• Smart Barcode size : 3mm x 3mm• Ordinary Camera: Distance 3 meter

Long Distance Barcodes

MIT media lab camera culture bokode.com mohan, woo, hiura, smithwick, raskar


Defocus blur of Bokode


Coding in Angle

Mohan, Woo, Smithwick, Hiura, Raskar [Siggraph 2009]


camera

sensorBokode(angle)

Encoding in Angle, not space, time or wavelength


camera

sensor

circle of confusion circle of information

Bokode(angle)

Quote suggested by Kurt Akeley

Encoding in Angle, not space, time or wavelength


Bokode

camera

fb

fc

magnification = fc /fb (microscope);

focus always at infinity

‘long-distance microscope’

MIT media lab camera culture

Product labels

Street-view Tagging


cell-phone cameraclose to the Bokode

(10,000+ bytes of data)

capturing Bokodes

Augmenting Plenoptic Function

38

Wigner Distribution Function

TraditionaTraditional Light l Light FieldField

WDF

TraditionaTraditional Light l Light FieldField

Augmented LF

Interference & DiffractionInteraction w/ optical elements

ray optics basedsimple and powerful

wave optics basedrigorous but cumbersome

Light Fields

Referenceplane

position angle

Goal: Representing propagation, interaction and image formation of light using purely position and angle parameters

LF propagation

(diffractive)optical

element

LF LF LF LF

LF propagation

light field transformer

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

40

41

LF Transformer: input to output LF Thin Elements: 6D

General Case: 8D

Angle Shift Invariant: 4D

Augmented LF framework

42

1. LF propagation

(diffractive)

optical element

LF LF LF LF

LF propagation

2. light field transformer

3. negative radiance

Tech report, S. B. Oh et al. http://web.media.mit.edu/~raskar/RayWavefront/

4. interference

Cubic phase plate

43

Pure ray bendingPositive only

radiance

DiffractionInterference of +/-

rays

Rotating PSF

Interference received on Complex Geometry

44

Can you ‘see’ around a corner ?

Femto-Photography: Higher Dimensional LF

FemtoFlash

UltraFast Detector

Computational OpticsSerious Sync

Important DatesSubmission: November 2, 2009Notification : February 2, 2010

Topics

Computational CamerasMultiple Images and Camera ArraysComputational IlluminationAdvanced Image and Video ProcessingScientific Photography and VideographyOrganizing and Exploiting Photo & Video Collections

Program ChairsKyros Kutulakos, U. TorontoRafael Piestun, U. ColoradoRamesh Raskar, MIT

International Conference on Computational Photography (ICCP)

March 29-30, 2010 MIT, Cambridge MA

http://cameraculture.media.mit.edu/ iccp10 /

Beyond Multi-touch: Mobile

Laptops

Mobile

Converting LCD Screen = large Camera for 3D Interactive HCI and Video Conferencing

Matthew Hirsch, Henry HoltzmanDoug Lanman, Ramesh Raskar

Siggraph Asia 2009

BiDi Screen

Overview: Sensing Depth from Array of Virtual Cameras in

LCD

Camera Culture Group, MIT Media Lab Ramesh Raskar http://raskar.info

• Beyond Traditional Imaging• Invertible motion blur in video• Looking around a corner• LCDs as virtual cameras• Computational probes (bokode)• Image destabilization

• Augmented Light Field• Rays for diffraction+interference

Computational PhotographyDigital

Epsilon

Coded

Essence

Computational Photography aims to make progress on both axis

Camera Array

HDR, FoV Focal stack

Decomposition problems

Depth

Spectrum

LightFields

Human Stereo Vision

Transient Imaging

Virtual Object Insertion

Relighting

Augmented Human

Experience

Material editing from single

photo

Scene completion from

photos

Motion Magnification

Phototourism

WDF

Light Light FieldField

Augmented LF

LF propagati

on

(diffractive)

optical element

LF

LF

LF

LF

LF propagati

on

light field transformer

raskar cosi invited talk oct 2009

Technology