polarization-based inverse rendering from single view daisuke miyazaki robby t. tan kenji hara...

Polarization-based Inverse Polarization-based Inverse Rendering from Single ViewRendering from Single View

Daisuke MiyazakiRobby T. Tan

Kenji HaraKatsushi Ikeuchi

2

Modeling cultural assetsModeling cultural assets

Integrated framework for obtaining 3 types of information

Geometrical Photometrical Environmental

3

Related workRelated work

Geometry Photometry Environment

Tominaga et.al. 2000

Zheng et.al. 1991

Nayar et.al. 1996

Sato et.al. 1999

Ramamoorthi et.al. 2001

Nishino et.al. 2001

Hara et.al. 2002

Proposed method

4

OutlineOutline

1. Reflection componentsseparation

2. Shape from polarizationusing diffuse light

3. Light source estimationfrom intensity peak

4. Reflection parametersestimation by l.s.m. Minimize

Ks, σ

renderedimage

realimage

2

1. Reflection components separation1. Reflection components separation

6

Dichromatic reflection modelDichromatic reflection model

Incident lightSpecularly

reflected lightDiffusely

reflected light

Air

Object

Surfacenormal

7

Reflection components separationReflection components separation

DiffuseInput Specular

[Tan2002]

•Shape•Illumination•Reflection parameters

2. Shape from polarization2. Shape from polarization

9

Related workRelated work

Object Reflection View

Koshikawa 1979 Opaque Specular 1

Wolff 1990 Opaque Diffuse 2

Rahmann et.al. 2001 Opaque Diffuse 2~5

Miyazaki et.al. 2002 Transparent Specular 2

Proposed method Opaque Diffuse 1

10

PolarizationPolarization

Incident lightSpecularly

reflected lightDiffusely

reflected light

Air

Object

11

Surface normalSurface normal

Object

Surface normal

Polarizer

Camera

Zenith angle

Azimuth angle

12

Azimuth angleφ and intensity differenceAzimuth angleφ and intensity difference

Rotationangle

ofpolarizer

Inten

sity

255

0

Imax

3601 2

-ambiguity

Imin

Azimuth angle

13

PropagationPropagation

[Ikeuchi&Horn1981]

Determination of azimuth angle Propagate φ from occluding boundary to inner part of object area (Assumption: smooth surface)

object

Cannot apply to “dimples”(=perfect concave)

14

Zenith angleθ and DOPρZenith angleθ and DOPρ

0

1

90°

Zenith angle θ

DOP ρ

DegreeOfPolarization

ρ

θ

minmax

minmax

II

II

22222

22

sincos4sin122

sin1

nnnn

nn

15

ModificationModification

0

0.5

90°

Zenithangle

θ

DOP ρ

DegreeOfPolarization

22222

22

sincos4sin122

sin1

nnnn

nn

uII

II

minmax

minmax

minmax

minmax

II

II

u: modification factor•Raises DOP•Assumption

•Closed smooth object•“u” is constant

Definition of DOP:

Modified DOP:

u

16

Surface normalSurface normal

φ

θSurface normal

17

HeightHeight

• Relaxation method

dxdyqy

Hp

x

H22

Minimize: where,

x

Hp

y

Hq

Gradient Height H

y

q

x

pyxHyxH ii

4

1),(),( )()1(Iteratively update:

[Ikeuchi1984]

221

1

qp

qp T

nSurface

normal

3. Illumination estimation3. Illumination estimation

19

Illumination sphereIllumination sphere

θ=0°

θ=90°θ=90°

θ=180°

Object

Light source is represented in polar coordinate system (θ, φ)

φ=0°

φ=90°

φ=180°

φ=270°

L1=(θ1, φ1)L2=(θ2, φ2)

L3=(θ3, φ3)

20

Illumination estimationIllumination estimation

Detect position of intensity peakDetermine light source orientation from the peak

1.Project to (θ, φ)-space 2.Thresholding 3.Detect intensity peak

4. Reflection parameters estimation4. Reflection parameters estimation

22

Torrance-Sparrow reflection modelTorrance-Sparrow reflection model

Specular reflection

2

2

2

cos

1cos

α

θθ eKKI

rsid

Diffuse reflection

Incidentlight

Surfacenormal

ViewBisector

Object surface

αθi θr

Known: θi, θr, α

Unknown:•Diffuse reflection scale; Kd

•Specular reflection scale; Ks

•Surface roughness; σ

23

Reflection parameters estimationReflection parameters estimation

Solve the following least-square problemby steepest-descent method

MinimizeKs, σ

renderedimage

realimage

2

2

2

2

cos

1 α

θeK

rs

Experimental resultExperimental result

25

InputInput

Intensity I

Azimuth angleφ

DOPρ

26

Result of shape estimationResult of shape estimation

27

Result of illumination estimationResult of illumination estimation

Actual illumination distribution Estimated illumination distribution

28

Rendering resultRendering result

Input

Synthesizedimage Rendered image under

different illumination & view

29

Result for another objectResult for another object

Input

Synthesizedimage

Estimatedshape

Rendered image underdifferent illumination & view

30

ConclusionsConclusions

• Estimated geometrical, photometrical, environmental information in one integrated framework– Shape from polarization– Surface reflection parameters from iterative

computation– Illumination from intensity peak

31

Application to digital archiving projectApplication to digital archiving project• Multiple View

• Modeling a statue in a room– IBR with

• surface normal• reflection parameters

Photorealistic preservation

FinFin

(c) Daisuke Miyazaki 2003(c) Daisuke Miyazaki 2003All rights reserved.All rights reserved.

http://www.cvl.iis.u-tokyo.ac.jp/D. Miyazaki, R. T. Tan, K. Hara, K. Ikeuchi,

"Polarization-based Inverse Rendering from Single View," in Proceedings of International Symposium on the CREST Digital Archiving Project, pp.51-65, Tokyo,

Japan, 2003.05