Dynamic Refraction Stereo

7. Contributions• Refractive disparity optimization gives stable reconstructions regardless of surface shape• Require no geometric assumptions other than single light refraction• Produces a full resolution height map and a separate, full-resolution normal map• Highly-detailed reconstructions for a variety of complex, deforming liquid surfaces

Kiriakos N. Kutulakoskyros @ cs.toronto.edu

Nigel J. W. Morrisnmorris @ dgp.toronto.edu

8. Experimental results

Experiments with dynamic water surfaces

t=0.6s

Hei

ght m

apT

ilt m

ap

Instantaneous 3D reconstructionst=0.92s t=1.32s

Simulations vs. Ground-truth Experiments Refractive index estimation

Observed results (solid red) vs simulated results (dotted blue) for 0.08 pixel localization error for set of planar refractive surfaces at various heights

Total reconstruction error as function of refractive index (liquid was water)

Hei

ght m

apT

ilt m

apH

eigh

t map

Tilt

map

t=0.92s

t=0.3s

t=0.91s

4. Refractive Stereo Algorithm

2. Related WorkSingle-view methods (require extra assumptions/optics):

• Shape from Distortion (Murase, PAMI 1992) assumes constant mean distance

• Shape from Refraction (Jähne et al., JOSA 1994)

uses collimating lens & lighting gradient

Multi-view methods:

• Sanderson et al. (PAMI 1998), Bonfort & Sturm (ICCV 2003)

static mirror scenes (known refractive index), optimization degrades for shallow liquid heights

• Multi-media photogrammetry (Flach & Maas, IAPRS 2000)

known parametric shape model

3. The Refraction Stereo Constraint

Key insight: For a generic refractive surface, knowledge of pixel q & pattern point C(q) defines a constraint curve in the 3D space of possible normals & refractive index values

• Height hypothesis in Camera 1

correspondence in Camera 2

• q & C(q) in Camera 1

1st constraint curve

• q’ & C(q’) in Camera 2

2nd constraint curve

• Generically, curves do not intersect for incorrect height hypotheses

can resolve height, normal, refractive index

Goal: Reconstruct instantaneous height map & normal map of

time-varying refractive surface projecting to 2 cameras

• Calibrated & synchronized cameras

• Arbitrary, time-varying surface shape (i.e. no prior shape models)

• Refractive index unknown

• Calibrated pattern at known 3D position, visible through liquid

• Known 1-1 mapping from pixels q to points C(q) on pattern

1. Dynamic Refraction Stereo Problem

p

n

Cam1 (time t)

Cam2(time t)

q

C(q)

• Discretize interval of possible refractive index values.For each value, do Steps 1-4:

Step 1: Initialize correspondence function C(.) for time 0

Step 2: For each time t and each pixel q in Camera 1

2a: (Refractive disparity optimization) 1D optimization along ray of q, searching for height hypothesis consistent with both viewpoints

2b: (Bundle adjustment) 5D optimization of p and n using reprojection error

Step 3: Fuse depth & normal map to obtain 3D surface

Step 4: Update correspondence function for next frame

• Choose refractive index value minimizing total reconstruction error (across all frames and pixels)

5. 1D Optimization: Refractive Disparity

• Large liquid heights: both criteria are 0 when p on ‘true’ surface

• As height → 0

estimation of n1, n2 unstable ||d1||2 + ||d2||2 remains stable (reduces to standard stereo)

minimize angle(n1,n2)

p

n1 n2

minimize ||d1||2 + ||d2||2

Bonfort & Sturm Refractive Disparity

n1n2n2n1

d1 d2

n1n2

p

nq

C(q)

q’

C(q’)

6. Computing Correspondence C(q)

Iteratively compute flow between un-refracted & refractedviews using Lucas-Kanade (Baker & Matthews, IJCV 2004)

Refracted view (time t)Un-refracted view (reference)

t=0.3s t=0.35s t=0.4s

t=0.75s t=0.83s t=0.91s