Real-Time Phase-Stamp Range Finder with Improved Accuracy

Akira KimachiOsaka Electro-Communication University

Neyagawa, Osaka 572-8530, Japan

1August 2, 2009 Optical Engineering + Applications, San Diego Convention Center

Outline

• Introduction• Phase-stamp range finder (PSRF)

– Time-domain correlation image sensor (CIS)– Phase-stamp imaging (PSI)– Problem of artifacts

• Accuracy improvement by calibrating the CIS for PSI– Experiment #1: CIS output behavior in PSI– CIS output model for PSI– Compensation for phase stamp errors– Experiment #2: accuracy evaluation

• Conclusions

2

Real-time range imaging

• Demands– Assembly/inspection of industrial products– Environment recognition for robots/vehicles– Observation of mobile objects– Assistance in human workspace

• Active vs. passive range finders– Active methods are preferable in terms of accuracy/reliability

3

# of sensors

Active illumination

Accuracy/reliability

Methods

Active 1 Yes High

Light sectioningStructured lightTime-of-flightTime-stampPhase-stamp

Passive > 1 No LowBinocular stereoMultiple-camera

Real-time active range finders• Robustness vs. depth resolution

– Difficult to establish both

4

Method SensorDepth

resolution

Robustness to disturbance

Ambient illuminationSurface textureTemporal

variationSpatial

variation

Light sectioning

High-speed camera

< 1 mm Low Low Low

VLSI sensor < 1 mm Low High Low

Structured lightHigh-speed camera

< 1 mm Low High High

Time-of-flight (TOF)

VLSI sensor > 1 mm High High High

Axi-Vision Camera

> 1 mm High High High

Time-stamp VLSI sensor < 1 mm Low Low Low

Phase-stampCorrelation image sensor

< 1 mm High High High

flat board @ 400 mm

Objective

• Phase-stamp range finder (PSRF)Kimachi and Ando, Electronic Imaging (2007)

– Time-domain correlation image sensor (CIS)– Frame-rate 3D capture based on “phase stamp” imaging (PSI)

• Problem of artifacts

• Solution– Analyze the behavior of CIS outputs in PSI– Model the CIS outputs for PSI– Calibrate the PSRF by compensating for CIS output errors

5

undulation

~ 3.5 mm rmsrandomnoisepattern

~ 1 mm rms

Correlation image sensor (CIS)

6

Ando and Kimachi, Trans. IEEE ED (2003)

Output images

Average intensity

200x200-pixel CMOS camera

Temporal correlation

: frame integral

frame

Phase-stamp imaging (PSI)

7

Light pulseenergy image

Phase stampimage

Correlation images

uncorrelated

noiseHigher temporal resolutionthan the frame period T

Phase-stamp range finder (PSRF)

8

SOLenergy

ambientillumination

surfacereflectance

(SOL)

Incident lightintensity

Referencesignals

SOLangle

Phasestamp

Correlation images

Range image

: pixel spacing in x

•One SOL scan in one frame•Based on PSI

Kimachi and Ando,Electronic Imaging (2007)

•Range image from single frame •Ambient illumination removed•Surface reflectane canceled

( )ijB tijR

ijz ( , )kQ i j

Experimental PSRF system

9

CIS 200x200-pixel

Frame rate 12.5 fps

Reference signal frequency

50 Hz

Mirror scanning rate

25 Hz

Laser DPSS ， 40 mW ， 658 nm

Camera lens 25 mm F1.4

PSRF artifacts

• Artifacts in range images– Undulation– Random noise pattern

• Considered to be caused by errors in detected phase stamps

• Temporal correlations may not follow the ideal characteristics with respect to SOL incidence time

10

ijz

ij

( , )kQ i jijt

phase stamp range mapaverage intensity SOL energy

(flat board @ 400 mm)

0

/2

/2

Investigating CIS outputs in PSI

• Capture a sequence of correlation images while shifting pulse occurrence time

11

( , , )kQ i j 0 50 Hz ( 20 ms)f T

Pulse occurrence time 0~20 ms (1 ms step)

Pulse height 0~8, 9 levels

Pulse width 0.2~2 ms (0.2 ms step)

Reference signal amplitude 0.05~0.5 V (0.05 step)0V

ht

0t0t

Results: image average behavior

12

Distortions in from sine functions become severer as increases( )kQ 0V

Image averages of temporal correlations increase monotonicallywith , , and

( )kQ 0Vt h

( , , )kQ i j

Pulse width variedt Pulse height variedh Amplitude varied0V

Undulations in the phase stamp become severer as increases0V

Results: pixel-wise deviation behavior

13

0

/2

/2

Pulse width variedt Pulse height variedh Amplitude varied0V

Pixel-wise deviations of temporal correlations increasemonotonically as , , and 0Vt h

( , , )k i j ( , , )kQ i j

Pixel-wise deviations of computed phase stamps oscillate with 0tij

( , ) (100,100)i j

oscillate with in arbitrary waveforms, randomly with respect to pixel and channel

( , , )k i j 0tk( , )i j

• Ideal characteristic of CIS temporal correlation outputs

• Experiment-based CIS output model for PSI

– Undulation/distortion→ Harmonics , ,

– Pixel-/channel-wise random deviation→ Coefficients , , , ,

– Dependence on light pulse energy → Multiplication by

• Coefficients are estimated from a sequence of PSI images by least-squares fitting

CIS output model for PSI

14

( , )ka i j ( , )kb i j ( , )kc i j ( , )kd i j ( , )ke i j

sin k cos 2 k sin 2 k

,ijA t h

noise

(for fixed )0V

Compensation for phase stamp errorsFor a single-frame set of temporal correlation images

, , and ,

1. Compute the phase stamps

2. Approximate the PSI CIS model by regarding as containing a small error

3. Estimate and pixel-wise by least-squares fitting to the model with and the pre-estimated coefficients , , , ,

4. Obtain the phase stamp estimates15

1( , )Q i j 3( , )Q i j

( , )kQ i j

2 ( , )Q i j

ij1ij

ijij

ij ijA

( , )ka i j ( , )kb i j ( , )kc i j ( , )kd i j ( , )ke i j

Results: CIS calibration for PSI (1)

16

( , ) (100,108)i j

test data = calibration data

•Fitting — over a sequence of •Compensation — on a single frame of

( , , )kQ i j kQ

Results: CIS calibration for PSI (2)

17

108j

test data = calibration data

0 4 mst

0 10 mst

0 17 mst

test data ≠ calibration data

Results: CIS calibration in PSI (3)

18

test data ≠ calibration data

Before compensation

After compensation

Results: artifacts removal in PSRF

19

SOL intensity phase stamp

@ 400 mm

compensateduncompensated

flat board

paper box

can bottle

Compensation Before After

Offsetphase [rad] −2.50 0.00

range [mm] 5.37 0.70

Undulation (rms)

phase [rad] 7.44 1.16

range [mm] 3.45 0.48

Random noise pattern (rms)

phase [rad] 1.81 0.92

range [mm] 1.09 0.55

Conclusions• Artifacts in PSRF outputs have been removed

– CIS outputs were modeled based on PSI experiments– A method for compensating for phase stamp errors in CIS

outputs was proposed– Confirmed in experiments

• Accuracy improved on a PSRF system– Undulation — 3.45 mm → 0.48 mm– Random noise pattern — 1.09 mm → 0.55 mm

20