active light time-of-flight imagingmanao.inria.fr/.../uploads/2015/09/active_3d_tof.pdf · 2015. 9....

Ivo Ihrke / Autumn 2015

Active Light – Time-of-Flight Imaging


Time-of-Flight – Pulse-based

time-of-flight scanners [Gvili03]

NOT triangulation based

short infrared laser pulse is sent from camera

reflection is recorded in a very short time frame (ps)

results in depth profile (intensity image)


Time-of-Flight – Pulse-based

time-of-flight scanner – examples

accuracy 1-2 cm in a range of 4 – 7 m

applications:

"depth keying" replaces chroma keying

3D interaction

large scale 3D scanning (LIDAR – light detection and ranging)


Canesta/3DV – Kinect 2 (?)


PMDs – Photonic Mixer Devices

[Luan’01]

Chip layout Schematic view

Electric symbol

Fast on-chip modulation

Also called “multi-bucket sensors”


PMDs – Photonic Mixer Devices

Working principle: Measure phase difference of emitted, modulated signal and received one

Current hardware: ~20MHz modulation frequency – in practice square wave

[Metrilus]


dt

P

PMD Sensor

Photonic Mixer Device (PMD) sensor:

[slid

e b

y F

elix

Heid

e]


dt

P

PMD Sensor

Single-path Time-of-Flight depth imaging:

[slid

e b

y F

elix

Heid

e]


dt

P

PMD Sensor

Single-path Time-of-Flight depth imaging:

distance-dependent correlation

[slid

e b

y F

elix

Heid

e]


NASAUniversity of Washington

LID

AR

Tim

e-o

f-F

light C

am

era

s

See [Kolb et al. 10], EG STAR for more details

PanasonicPMD

Fotonic ARTTS MESA


Active Light –Transient Imaging


What is Transient Imaging ?

a.k.a. Light-in-Flight Imaging

• Ultrafast imaging of non-

stationary light distribution

in scenes

• Tracking of wavefronts of

light as they propagate in

the scene

• Equivalent to imaging

ultrashort pulses of light

t = 0t = 1t = 2t = 3t = 4t = 5t = 6t = 7t = 8

[slid

e b

y F

elix

Heid

e]


Pandharkar et al´2011

Many applications by analyzing a transient image!

Velten et al´2013

Understanding

light transport:

Naik et al´2011

Surface

reflectance

capture:

Wu et al´2012

Reconstructing hidden

object geometry (and

motion):

Velten et al´2012

Decomposing light

transport:

Applications of Transient Imaging

[slid

e b

y F

elix

Heid

e]


Visualizing Light in Motion

Repetitive Event, 1012 FPS, 672 x 1000 pixel resolution, 1 ns+ capture time

Light moves 0.6 mm per frame

Ability to see light transport

[Velten et al. 13]


Streak Cameras

©

Hamamatsu Picosecond time resolution

1D: 1x672 pixels

Result as 2D image (“Streak Photo”)

[Velten et al. 13]


Object

Experimental Setup

Ti:Sapphire Laser

Camera

Synchronization

[Velten et al. 13]

Scanning Unit


Actual Setup

[Velten et al. 13]


Streak Camera Picture

Space

Tim

e

[Ve

lten

et a

l. 13

]


[Velten et al. 13]


Example Result [Velten et al. 13]


[Velten et al. 13]


Scattering in Real Scenes

[Velten et al. 13]


[Velten et al. 13]


With PMD Devices – linked to the “multi-path” or “mixed pixel” problem

PanasonicPMD

Fotonic ARTTS MESA

Transient Imaging


dt

PMD Sensor

P2

P1

Multi-path contributions:

[slid

es b

y F

elix

Heid

e]


dt

PMD Sensor

P2

P1

Recovering multi-path contributions:

[slid

es b

y F

elix

Heid

e]


dt

PMD Sensor

P2

P1

…

Recovering multi-path contributions:

[slid

es b

y F

elix

Heid

e]


• Relation of Transient Image to PMD measurement:

• Linear system is ill-posed:

• assume model for temporal response (signal sparsity)

Mixture Model: Gaussian + Exponential

• and spatial smoothness

• solve non-linear system

Image formation model:

multiple measurements

[slid

es b

y F

elix

Heid

e]

Spatial coherence Temporal model


PMDTechnologies CamBoard nano

FPGA

LED unit

PMD sensor

PLL

25MHz

ADC

Control

2”

Too slow

Fixed frequency

[slid

es b

y F

elix

Heid

e]


Novatech

DDS9m

FPGA

Modifications to the hardware

PMD sensor ADC

Control

Control

Laser unit

iC-HG +

LPC826

TRIG

Prototype: Max mod. frequency 180 MHz, stable up to 110 MHz

2-ch.

function

generator

0-180 MHz

MOD

[slid

es b

y F

elix

Heid

e]


Prototype setup

Used for results in paper

Laser

unit

Function

generator

Power

Camera

MOD in

TRIG

out


Color coding of strongest component:

Result ‘Discoball’:

[slid

es b

y F

elix

Heid

e]


Different time-steps:

Results ‘Discoball’:

[slid

es b

y F

elix

Heid

e]


Color coding of strongest component:

Result ‘Bottles’:

[slid

es b

y F

elix

Heid

e]


Camera unit

FPGA

Emerging Technologies prototype

PMD sensor ADC

Control

Control

Light

source

TRIGMOD

AD9958

DDS (function

generator)

0-180 MHz

[slid

es b

y F

elix

Heid

e]


Light

source

Camera

[slid

es b

y F

elix

Heid

e]


Results: People

[slid

es b

y F

elix

Heid

e]


Curless, Levoy, “Better optical triangulation through spacetime

analysis“, CVPR 1995

Levoy et al., “The digital Michelangelo project: 3D scanning of

large statues”, SIGGRAPH 2000

[Gvili03] R. Gvili, A. Kaplan, E. Ofek, G. Yahav, "Depth Keying", SPIE ISOE 5006, 2003, pp. 564-574

Luan, “Experimental Investigation of Photonic Mixer Device and Development of TOF 3D Ranging Systems Based on PMD Technology”, PhD Thesis, Siegen University, 2001

Velten et al. “Femto-photography: capturing and visualizing the

propagation of light”, SIGGRAPH 2013

Heide et al. “Low-budget Transient Imaging using Photonic Mixer Devices.”, SIGGRAPH 2013

References

active light time-of-flight imagingmanao.inria.fr/.../uploads/2015/09/active_3d_tof.pdf · 2015. 9....

Documents