A Theory of Locally Low Dimensional Light Transport

Dhruv Mahajan (Columbia University)

Ira Kemelmacher-Shlizerman (Weizmann Institute)

Ravi Ramamoorthi (Columbia University)

Peter Belhumeur (Columbia University)

Image Relighting

Ng et al 2003

Relighting – Linear Combination

1l 2l Nl .......=

Images lit by directional light sources

Lighting Intensities

Nimeroff et al 94

Dorsey 95

Hallinan 94

Relighting – Matrix Vector Multiply

.......=

...

1l 2l Nl

1T 2T NT L

=

Input Lighting(Unfolded Cubemap)

Output ImageVector Transport Matrix

T L

1b

2b

Mb

...B

Light transport matrix dimensions

512 x 512 images

6 x 32 x 32 = 6144 cubemap lighting

Multiplication / Relighting cost

Approximately 1010 computations per frame

Multiplication intractable in real time

Need to compress the light transport

Light Transport – Computational Cost

Light Transport – SVD

1T 2T NTTransport Matrix

....... .......

....

12

3

KU S

12

3

K

....1w 2w 3w Kw

L

LightingVector

Relit Image

KP KK NKEigenvalues

1N

Hallinan 94

VT

VT

Basis Images

Projection Weights

Light Transport – SVD

- Global DimensionalityK

K

99

Large

....... .......

....

12

3

K

V

T

1T 2T NTTransport Matrix

KP KK NKEigenvalues

En

erg

y (i

n %

)

No. of Eigenvalues

Computation still intractable

Global Dimensionality

32326700 K6144

Locally Low Dimensional Light Transport

.......

p pixels

p rows

SVD

1 2 3 n....1w 2w 3w nw

Locally Low Dimensional Transport Npn ,4096p 6144N 50n Lighting Resolution

Dimensionality of the patchn

1T 2T NTTransport Matrix

Previous Work

Blockwise PCA – Nayar et al. 04 Image divided in to fixed size

square patches

Each patch compressed using PCA

Clustered PCA – Sloan et al. 03 Object divided in to fixed

number of clusters

Each cluster compressed using PCA

Previous Work

Surface light fields Nishino et al. 01 Chen et al. 02

General reflectance fields Matusik et al. 02 Garg et al. 06

Compression JPEG, MPEG

No Theoretical Analysis

Dimensionality vs Patch Size?

Dimensionality vs Material Properties?

Dimensionality vs Global Effects ?

Local Light Transport Dimensionality

Analysis of local light transport dimensionality

P

Dimensionality

Co

st

Patch Area1

Local Light Transport Dimensionality

Analysis of local light transport dimensionality

Dimensionality

Co

st

Patch Area

2 x 2

Local Light Transport Dimensionality

Analysis of local light transport dimensionality

Dimensionality

Co

st

Patch Area

Local Light Transport Dimensionality

Analysis of local light transport dimensionality

Dimensionality

Co

st

Patch Area

Local Light Transport Dimensionality

Analysis of local light transport dimensionality

Dimensionality

Co

st

Patch Area

Local Light Transport Dimensionality

Analysis of local light transport dimensionality

Dimensionality

Co

st

Patch Area

Rendering Cost

Theoretical analysis of rendering cost

Co

st

Patch Area

Overhead cost for rendering

Dimensionality

Overhead Cost

....1w 2w 3w nw

Global Lighting

Dimensionality cost = number of bases

Overhead Cost = Projection Weights

Co

st

Patch Area

Rendering Cost

Theoretical analysis of rendering cost

Co

st

Patch Area

Overhead cost for rendering

P

Rendering Cost

Theoretical analysis of rendering cost

Co

st

Patch Area

Overhead cost for rendering

Rendering Cost

Theoretical analysis of rendering cost

Co

st

Patch Area

Overhead cost for rendering

Rendering Cost

Theoretical analysis of rendering cost

Co

st

Patch Area

Overhead cost for rendering

Rendering Cost

Theoretical analysis of rendering cost

Co

st

Patch Area

Overhead cost for rendering

Rendering Cost

Theoretical analysis of rendering cost

Co

st

Patch Area

Overhead cost for rendering

Rendering Cost

Theoretical analysis of rendering cost

Co

st

Patch Area

Overhead cost for rendering

Patch SizeOptimal

Rendering cost = Dimensionality + Overhead

Contributions

Analysis of dimensionality of local light transport Change of dimensionality with size

Diffuse and glossy reflections Shadows

Analyzing rendering cost Analytical formula for optimal patch size

Practical Applications Fine tuning parameters of existing methods Scale images to very high resolutions Develop adaptive clustering algorithm

Local Light Transport Dimensionality

Analysis of local light transport dimensionality

Dimensionality

Co

st

Patch Area

Dimensionality vs. Patch Size

Large Area : linear relationship

slope = 1

slope - rate of change of dimensionality

Independent of material properties np,

log

(D

ime

nsi

ona

lity)

log (Patch Area)

pixels dimensionality

2,2 np 2,2 np

Diffuse/Specular BRDF

Dimensionality Patch Area

Dimensionality vs. Patch Size

Small Area : sub - linear relationship

np,

log

(D

ime

nsi

ona

lity)

log (Patch Area)

pixels dimensionality

slope < 1

np ,2np ,2

Diffuse/Specular BRDF

Mathematical Tools for Analysis

Convolution formula for glossy reflections and shadows

Ramamoorthi and Hanrahan 01

Basri and Jacobs 01

Ramamoorthi et al 04

Szego’s Eigenvalue Distribution Theorem

Eigenvalues of the light transport matrix of the patch

Fourier Scale and Convolution Theorems

Dimensionality as a function of patch size

Bandwidth of BRDF

Central Result

PatchDimensionality

PatchArea

ConstantBandwidth of BRDF

PatchDimensionality

PatchArea

Constant

Lighting

BRDF

low pass filter

Material property

Fourier

Transform

x

)(xf F

BRDF/ Material Properties

Bandwidth of BRDF

Central Result

PatchDimensionality

PatchArea

Constant

99% Energy

low frequency

highfrequency

Bandwidth

Central Result

log

Large Arealo

g (D

imen

sion

ality

)

log (Patch area)

log

Diffuse/Specular BRDF

Bandwidth of BRDF

PatchDimensionality

PatchArea ( ( )) Bandwidth

of BRDFPatchArea ConstantConstant

Large Area

log

(Dim

ensi

onal

ity)

log (Patch area)

Diffuse/Specular BRDF

log log Bandwidth of BRDF

PatchDimensionality

PatchArea ( () )

Large Area

Bandwidth of BRDF

)(loglo

g (D

imen

sion

ality

)

log (Patch area)

Diffuse/Specular BRDF

log logPatchDimensionality

PatchArea( ( ))

log log Bandwidth of BRDF

PatchDimensionality

PatchArea ( () )

Large Area

Bandwidth of BRDF

)(loglo

g (D

imen

sion

ality

)

log (Patch area)

Diffuse/Specular BRDF

linear relationship

slope = 1

log logPatchDimensionality

PatchArea( ( ))

Small Area

log

(Dim

ensi

onal

ity)

log (Patch area)

Diffuse/Specular BRDF

slope < 1

sublinear relationship

Bandwidth of BRDF

)(loglog logPatchDimensionality

PatchArea( ( ))

Contributions

Analysis of dimensionality of local light transport Change of dimensionality with size

Glossy reflections Shadows

Analyzing rendering cost Analytical formula for optimal patch size

Practical Applications Fine tuning parameters of existing methods Scale images to very high resolutions Develop adaptive clustering algorithm

Visibility Function

Blocker

Visibility Function = 0

Visibility Function = 1Visibility Function = 1

P

Lighting Directions

Shadows

Dimensionality changes slowly in presence of shadows

Diffuse and Specular BRDF Shadows

slope = .5

slope = 1lo

g (

Dim

en

sion

alit

y)

log (Patch area)

Light Transport = Visibility Function

Shadows – Step Blocker

x

y

z

1p2p

3p

Step Blocker

Dimensionality √Patch Area

Same Visibility Function

Dimensionality changes only along one dimension

log (Dimensionality) .5 log(Patch Area)

Different Visibility Function

Light Transport = Visibility Function

21, pp

x

z

3p

Contributions

Analysis of dimensionality of local light transport Change of dimensionality with size

Glossy reflections Shadows

Analyzing rendering cost Analytical formula for optimal patch size

Practical Applications Fine tuning parameters of existing methods Scale images to very high resolutions Develop adaptive clustering algorithm

Contributions

Analysis of dimensionality of local light transport Change of dimensionality with size

Glossy reflections Shadows

Analyzing rendering cost Analytical formula for optimal patch size

Practical Applications Fine tuning parameters of existing methods Scale images to very high resolutions Develop adaptive clustering algorithm

Overhead CostC

ost

Patch Area

Dimensionality

Overhead CostC

ost

Patch Area

P

Overhead

Dimensionality

Overhead CostC

ost

Patch Area

Dimensionality

Overhead

Overhead CostC

ost

Patch Area

Dimensionality

Overhead

Overhead CostC

ost

Patch Area

Dimensionality

Overhead

Overhead CostC

ost

Patch Area

Dimensionality

Overhead

Overhead CostC

ost

Patch Area

Dimensionality

Overhead

Rendering CostC

ost

Patch Area

Rendering Cost

Dimensionality

Overhead

Rendering Cost vs. Patch Size

Intermediate size :

Rate of increase in dimensionality

Rate of decrease in overhead=

Total cost minimum

Co

st

Patch Area

Rendering Cost

Dimensionality

Overhead

Minimum

Optimal Patch Size

1

12* KpOptimal Patch Size

- Global DimensionalityK

Optimal Patch Size

- Global DimensionalityK

1

12* KpOptimal Patch Size

- Function of slope of dimensionality curve

Dimensionality Curve

- From our theoretical analysis- Empirically from the given dataset

Optimal Patch Size – CPCA Example

7.168* p

1

12* KpOptimal Patch Size

Total cost

Face dataset across lighting

170~*p

110 220 330 440 550average cluster size

cost

per

pix

el

- Global DimensionalityK - Function of slope of

dimensionality curve

Glossy Reflections

1

12* KpOptimal Patch Size - Global DimensionalityK - Function of slope of

dimensionality curve

Kp ~*

Number of pixels in the patch increases with glossiness

Independent of material properties

Contributions

Analysis of dimensionality of local light transport Change of dimensionality with size

Glossy reflections Shadows

Analyzing rendering cost Analytical formula for optimal patch size

Practical Applications Fine tuning parameters of existing methods Scale images to very high resolutions Develop adaptive clustering algorithm

Setting Optimal Patch Size – CPCA

Setting Optimal Patch Size – CPCA

24000 vertices

57.Estimated 220 114.78

cost per pixel

]6.53[.

clusters

130-600 114.78-130

11 310.7large

6 X 32 X 32Cube Map45.0 Hz.

Contributions

Analysis of dimensionality of local light transport Change of dimensionality with size

Glossy reflections Shadows

Analyzing rendering cost Analytical formula for optimal patch size

Practical Applications Fine tuning parameters of existing methods Scale images to very high resolutions Develop adaptive clustering algorithm

Scaling of Cost With Resolution

Subdivide More

p

ppnew 4new resolutionIndependent of patch resolution

170~*p 140~*p

Optimal patch size same for both resolutions

1

12* Kp

- Global Dimensionality

- Function of slope of dimensionality curve

K

Scaling of Cost With Resolution

Sub-linear increase in cost with resolution

Increase in resolution - x0.4

24Increase in cost -

68.0

1.85

p

ppnew 4new resolution

Sublinear increase in cost with resolution

1024 1024 800 x 600

Scaling of Cost With Resolution

Scaling of Cost With Resolution

Summary

Analysis of dimensionality of local light transport

Diffuse and Glossy reflections, dimensionality area

Shadows, dimensionality √area

Analysis of rendering cost

Optimal patch size

Scaling of cost with resolution

Practical Applications

Setting optimal parameters in existing methods

Adaptive clustering algorithms

Future Work

More solid theoretical foundation High dimensional appearance compression

Representation

ECCV 2006, PAMI 2007

Analysis of light transport in frequency domain

TOG, Jan. 2007

Analysis of light transport in gradient domain

Siggraph 2007

Analysis of general local light transport for patches