![Page 1: Analysis of Lighting Effects Outline: The problem Lighting models Shape from shading Photometric stereo Harmonic analysis of lighting](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d485503460f94a2349a/html5/thumbnails/1.jpg)
Analysis of Lighting Effects
Outline: The problem Lighting models Shape from shading Photometric stereo Harmonic analysis of lighting
![Page 2: Analysis of Lighting Effects Outline: The problem Lighting models Shape from shading Photometric stereo Harmonic analysis of lighting](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d485503460f94a2349a/html5/thumbnails/2.jpg)
![Page 3: Analysis of Lighting Effects Outline: The problem Lighting models Shape from shading Photometric stereo Harmonic analysis of lighting](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d485503460f94a2349a/html5/thumbnails/3.jpg)
![Page 4: Analysis of Lighting Effects Outline: The problem Lighting models Shape from shading Photometric stereo Harmonic analysis of lighting](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d485503460f94a2349a/html5/thumbnails/4.jpg)
![Page 5: Analysis of Lighting Effects Outline: The problem Lighting models Shape from shading Photometric stereo Harmonic analysis of lighting](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d485503460f94a2349a/html5/thumbnails/5.jpg)
Applications
Modeling the effect of lighting can be used for Recognition – particularly face recognition Shape reconstruction Motion estimation Re-rendering …
![Page 6: Analysis of Lighting Effects Outline: The problem Lighting models Shape from shading Photometric stereo Harmonic analysis of lighting](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d485503460f94a2349a/html5/thumbnails/6.jpg)
Lighting is Complex
Lighting can come from any direction and at any strength
Infinite degree of freedom
![Page 7: Analysis of Lighting Effects Outline: The problem Lighting models Shape from shading Photometric stereo Harmonic analysis of lighting](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d485503460f94a2349a/html5/thumbnails/7.jpg)
Issues in Lighting
Single light source (point, extended) vs. multiple light sources
Far light vs. near light Matt surfaces vs. specular surfaces Cast shadows Inter-reflections
![Page 8: Analysis of Lighting Effects Outline: The problem Lighting models Shape from shading Photometric stereo Harmonic analysis of lighting](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d485503460f94a2349a/html5/thumbnails/8.jpg)
Lighting
From a source – travels in straight lines Energy decreases with r2 (r – distance from
source) When light rays reach an object
Part of the energy is absorbed Part is reflected (possibly different amounts
in different directions) Part may continue traveling into the object,
if object is transparent / translucent
![Page 9: Analysis of Lighting Effects Outline: The problem Lighting models Shape from shading Photometric stereo Harmonic analysis of lighting](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d485503460f94a2349a/html5/thumbnails/9.jpg)
Specular Reflectance
When a surface is smooth light reflects in the opposite direction of the surface normal
![Page 10: Analysis of Lighting Effects Outline: The problem Lighting models Shape from shading Photometric stereo Harmonic analysis of lighting](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d485503460f94a2349a/html5/thumbnails/10.jpg)
Specular Reflectance
When a surface is slightly rough the reflected light will fall off around the specular direction
![Page 11: Analysis of Lighting Effects Outline: The problem Lighting models Shape from shading Photometric stereo Harmonic analysis of lighting](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d485503460f94a2349a/html5/thumbnails/11.jpg)
Lambertian Reflectance
When the surface is very rough light may be reflected equally in all directions
![Page 12: Analysis of Lighting Effects Outline: The problem Lighting models Shape from shading Photometric stereo Harmonic analysis of lighting](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d485503460f94a2349a/html5/thumbnails/12.jpg)
Lambertian Reflectance
When the surface is very rough light may be reflected equally in all directions
![Page 13: Analysis of Lighting Effects Outline: The problem Lighting models Shape from shading Photometric stereo Harmonic analysis of lighting](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d485503460f94a2349a/html5/thumbnails/13.jpg)
Lambertian Reflectance
![Page 14: Analysis of Lighting Effects Outline: The problem Lighting models Shape from shading Photometric stereo Harmonic analysis of lighting](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d485503460f94a2349a/html5/thumbnails/14.jpg)
Lambert Law
n̂
cos ( 90 !)
ˆ ˆ( , )
o
T
I E
I l n l E l n n
or
l̂
![Page 15: Analysis of Lighting Effects Outline: The problem Lighting models Shape from shading Photometric stereo Harmonic analysis of lighting](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d485503460f94a2349a/html5/thumbnails/15.jpg)
BRDF
A general description of how opaque objects reflect light is given by the Bidirectional Reflectance Distribution Function (BRDF)
BRDF specifies for a unit of incoming light in a direction (θi,Φi) how much light will be reflected in a direction (θe,Φe) . BRDF is a function of 4 variables f(θi,Φi;θe,Φe).
(0,0) denotes the direction of the surface normal. Most surfaces are isotropic, i.e., reflectance in any
direction depends on the relative direction with respect to the incoming direction (leaving 3 parameters)
n̂
![Page 16: Analysis of Lighting Effects Outline: The problem Lighting models Shape from shading Photometric stereo Harmonic analysis of lighting](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d485503460f94a2349a/html5/thumbnails/16.jpg)
Why BRDF is Needed?
Light from front Light from back
![Page 17: Analysis of Lighting Effects Outline: The problem Lighting models Shape from shading Photometric stereo Harmonic analysis of lighting](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d485503460f94a2349a/html5/thumbnails/17.jpg)
Most Existing Algorithms
Assume a single, distant point source All normals visible to the source (θ<90°) Plus, maybe, ambient light (constant lighting
from all directions)
![Page 18: Analysis of Lighting Effects Outline: The problem Lighting models Shape from shading Photometric stereo Harmonic analysis of lighting](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d485503460f94a2349a/html5/thumbnails/18.jpg)
Shape from Shading
Input: a single image Output: 3D shape Problem is ill-posed, many different shapes can
give rise to same image Common assumptions:
Lighting is known Reflectance properties are completely known –
For Lambertian surfaces albedo is known (usually uniform)
![Page 19: Analysis of Lighting Effects Outline: The problem Lighting models Shape from shading Photometric stereo Harmonic analysis of lighting](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d485503460f94a2349a/html5/thumbnails/19.jpg)
convex
concave
![Page 20: Analysis of Lighting Effects Outline: The problem Lighting models Shape from shading Photometric stereo Harmonic analysis of lighting](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d485503460f94a2349a/html5/thumbnails/20.jpg)
convex
concave
![Page 21: Analysis of Lighting Effects Outline: The problem Lighting models Shape from shading Photometric stereo Harmonic analysis of lighting](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d485503460f94a2349a/html5/thumbnails/21.jpg)
HVS Assumes Light from Above
![Page 22: Analysis of Lighting Effects Outline: The problem Lighting models Shape from shading Photometric stereo Harmonic analysis of lighting](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d485503460f94a2349a/html5/thumbnails/22.jpg)
HVS Assumes Light from Above
![Page 23: Analysis of Lighting Effects Outline: The problem Lighting models Shape from shading Photometric stereo Harmonic analysis of lighting](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d485503460f94a2349a/html5/thumbnails/23.jpg)
Lambertian Shape from Shading (SFS) Image irradiance equation
Image intensity depends on surface orientation It also depends on lighting and albedo, but those
assumed to be known
ˆ( , ) ( ( , ))I x y R n x y
![Page 24: Analysis of Lighting Effects Outline: The problem Lighting models Shape from shading Photometric stereo Harmonic analysis of lighting](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d485503460f94a2349a/html5/thumbnails/24.jpg)
Surface Normal
A surface z(x,y) A point on the surface: (x,y,z(x,y))T
Tangent directions tx=(1,0,p)T, ty=(0,1,q)T with p=zx, q=zy
2 2
1ˆ ( , ,1)
. 1
x y Tt tn p q
p q
ˆ ˆ( , , ) , 1Tx y zn n n n n
![Page 25: Analysis of Lighting Effects Outline: The problem Lighting models Shape from shading Photometric stereo Harmonic analysis of lighting](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d485503460f94a2349a/html5/thumbnails/25.jpg)
Lambertian SFS
We obtain
Proportionality – because albedo is known up to scale For each point one differential equation in two unknowns, p and q
But both come from an integrable surface z(x,y) Thus, py= qx (zxy=zyx). Therefore, one differential equation in one unknowns
2 2ˆ( , ) ( , )
1
x y zT l p l q lI x y R p q l n
p q
![Page 26: Analysis of Lighting Effects Outline: The problem Lighting models Shape from shading Photometric stereo Harmonic analysis of lighting](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d485503460f94a2349a/html5/thumbnails/26.jpg)
Lambertian SFS
cos 0.5
60
I
![Page 27: Analysis of Lighting Effects Outline: The problem Lighting models Shape from shading Photometric stereo Harmonic analysis of lighting](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d485503460f94a2349a/html5/thumbnails/27.jpg)
SFS with Fast Marching
Suppose lighting coincides with viewing direction l=(0,0,1)T, then
Therefore
For general l we can rotate the camera
2 2
1
1I
p q
2 2 1
x y zl p l q lI
p q
2 22
11p q z
I
![Page 28: Analysis of Lighting Effects Outline: The problem Lighting models Shape from shading Photometric stereo Harmonic analysis of lighting](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d485503460f94a2349a/html5/thumbnails/28.jpg)
Distance Transform
is called Eikonal equation Consider d(x) s.t. |dx|=1
Assume x0=0
2
11z
I
x0
d
x
![Page 29: Analysis of Lighting Effects Outline: The problem Lighting models Shape from shading Photometric stereo Harmonic analysis of lighting](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d485503460f94a2349a/html5/thumbnails/29.jpg)
Distance Transform
is called Eikonal equation Consider d(x) s.t. |dx|=1
Assume x0=0 and x0=1
2
11z
I
x0
d
xx1
![Page 30: Analysis of Lighting Effects Outline: The problem Lighting models Shape from shading Photometric stereo Harmonic analysis of lighting](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d485503460f94a2349a/html5/thumbnails/30.jpg)
SFS with Fast Marching
- Some places are more difficult to walk than others
Solution to Eikonal equations –using a variation of Dijkstra’s algorithm
Initial condition: we need to know z at extrema Starting from lowest points, we propagate a
wave front, where we gradually compute new values of z from old ones
( , )z F x y
![Page 31: Analysis of Lighting Effects Outline: The problem Lighting models Shape from shading Photometric stereo Harmonic analysis of lighting](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d485503460f94a2349a/html5/thumbnails/31.jpg)
Results
![Page 32: Analysis of Lighting Effects Outline: The problem Lighting models Shape from shading Photometric stereo Harmonic analysis of lighting](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d485503460f94a2349a/html5/thumbnails/32.jpg)
Photometric Stereo
Fewer assumptions are needed if we have several images of the same object under different lightings
In this case we can solve for both lighting, albedo, and shape
This can be done by Factorization Recall that
Ignore the case θ>90°
s.t. 0T TI l n l n
![Page 33: Analysis of Lighting Effects Outline: The problem Lighting models Shape from shading Photometric stereo Harmonic analysis of lighting](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d485503460f94a2349a/html5/thumbnails/33.jpg)
Photometric Stereo - FactorizationGoal: given M, find L and S
M L S
1 1 111 1
1 2
1 2
1 2
3
1 3
...
. ....
. ....
. ....
p x y z
x x
y y
z z pf f f
f fp x y zf p f
I I l l l
n n
n n
n n
I I l l l
What should rank(M) be?
![Page 34: Analysis of Lighting Effects Outline: The problem Lighting models Shape from shading Photometric stereo Harmonic analysis of lighting](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d485503460f94a2349a/html5/thumbnails/34.jpg)
Photometric Stereo - Factorization Use SVD to find a rank 3 approximation
Define So Factorization is not unique, since
, A invertible
To reduce ambiguity we impose integrability
, 3 3TM U V
, TL U S V
M L S
1 ˆˆ( ) ( )M LA AS L S 1 ˆL̂ LA S AS
![Page 35: Analysis of Lighting Effects Outline: The problem Lighting models Shape from shading Photometric stereo Harmonic analysis of lighting](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d485503460f94a2349a/html5/thumbnails/35.jpg)
Reducing Ambiguity
Assume We want to enforce integrability Notice that
Denote by the three rows of A, then
From which we obtain
n An
y xp q
, yx
z z
nnp q
n n
1 2
3 3
,a n a n
p qa n a n
1 2
3 3
a n a n
y a n x a n
1 2 3, ,a a a
![Page 36: Analysis of Lighting Effects Outline: The problem Lighting models Shape from shading Photometric stereo Harmonic analysis of lighting](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d485503460f94a2349a/html5/thumbnails/36.jpg)
Reducing Ambiguity
Linear transformations of a surface
It can be shown that this is the only transformation that maintains integrability
Such transformations are called “generalized bas relief transformations” (GBR)
Thus, by imposing integrability the surface is reconstructed up to GBR
( , ) ( , )
x
y
y y x x
z x y x y z x y
p z p
q z q
p p q q
![Page 37: Analysis of Lighting Effects Outline: The problem Lighting models Shape from shading Photometric stereo Harmonic analysis of lighting](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d485503460f94a2349a/html5/thumbnails/37.jpg)
Relief Sculptures
![Page 38: Analysis of Lighting Effects Outline: The problem Lighting models Shape from shading Photometric stereo Harmonic analysis of lighting](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d485503460f94a2349a/html5/thumbnails/38.jpg)
Illumination Cone Due to additivity, the set of images of an object
under different lighting forms a convex cone in RN
This characterization is generic, holds also with specularities, shadows and inter-reflections
Unfortunately, representing the cone is complicated (infinite degree of freedom)
= 0.5* +0.2* +0.3*
![Page 39: Analysis of Lighting Effects Outline: The problem Lighting models Shape from shading Photometric stereo Harmonic analysis of lighting](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d485503460f94a2349a/html5/thumbnails/39.jpg)
Eigenfaces
Photobook/Eigenfaces (MIT Media Lab)
![Page 40: Analysis of Lighting Effects Outline: The problem Lighting models Shape from shading Photometric stereo Harmonic analysis of lighting](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d485503460f94a2349a/html5/thumbnails/40.jpg)
Recognition with PCA
Amano, Hiura, Yamaguti, and Inokuchi; Atick and Redlich; Bakry, Abo-Elsoud, and Kamel; Belhumeur, Hespanha, and Kriegman; Bhatnagar, Shaw, and Williams; Black and Jepson; Brennan and Principe; Campbell and Flynn; Casasent, Sipe and Talukder; Chan, Nasrabadi and Torrieri; Chung, Kee and Kim; Cootes, Taylor, Cooper and Graham; Covell; Cui and Weng; Daily and Cottrell; Demir, Akarun, and Alpaydin; Duta, Jain and Dubuisson-Jolly; Hallinan; Han and Tewfik; Jebara and Pentland; Kagesawa, Ueno, Kasushi, and Kashiwagi; King and Xu; Kalocsai, Zhao, and Elagin; Lee, Jung, Kwon and Hong; Liu and Wechsler; Menser and Muller; Moghaddam; Moon and Philips; Murase and Nayar; Nishino, Sato, and Ikeuchi; Novak, and Owirka; Nishino, Sato, and Ikeuchi; Ohta, Kohtaro and Ikeuchi; Ong and Gong; Penev and Atick; Penev and Sirivitch; Lorente and Torres; Pentland, Moghaddam, and Starner; Ramanathan, Sum, and Soon; Reiter and Matas; Romdhani, Gong and Psarrou; Shan, Gao, Chen, and Ma; Shen, Fu, Xu, Hsu, Chang, and Meng; Sirivitch and Kirby; Song, Chang, and Shaowei; Torres, Reutter, and Lorente; Turk and Pentland; Watta, Gandhi, and Lakshmanan; Weng and Chen; Yuela, Dai, and Feng; Yuille, Snow, Epstein, and Belhumeur; Zhao, Chellappa, and Krishnaswamy; Zhao and Yang…
![Page 41: Analysis of Lighting Effects Outline: The problem Lighting models Shape from shading Photometric stereo Harmonic analysis of lighting](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d485503460f94a2349a/html5/thumbnails/41.jpg)
Ball Face Phone Parrot
#1 48.2 53.7 67.9 42.8
#2 84.4 75.2 83.2 69.7
#3 94.4 90.2 88.2 76.3
#4 96.5 92.1 92.0 81.5
#5 97.9 93.5 94.1 84.7
#6 98.9 94.5 95.2 87.2
#7 99.1 95.3 96.3 88.5
#8 99.3 95.8 96.8 89.7
#9 99.5 96.3 97.2 90.7
#10 99.6 96.6 97.5 91.7
Empirical Study
(Yuille et al.)
![Page 42: Analysis of Lighting Effects Outline: The problem Lighting models Shape from shading Photometric stereo Harmonic analysis of lighting](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d485503460f94a2349a/html5/thumbnails/42.jpg)
0 1 2 30
0.5
1
0 1 2 30
0.5
1
1.5
2
lighting
reflectance
Intuition
![Page 43: Analysis of Lighting Effects Outline: The problem Lighting models Shape from shading Photometric stereo Harmonic analysis of lighting](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d485503460f94a2349a/html5/thumbnails/43.jpg)
(Light → Reflectance) = Convolution
![Page 44: Analysis of Lighting Effects Outline: The problem Lighting models Shape from shading Photometric stereo Harmonic analysis of lighting](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d485503460f94a2349a/html5/thumbnails/44.jpg)
(Light → Reflectance) = Convolution
![Page 45: Analysis of Lighting Effects Outline: The problem Lighting models Shape from shading Photometric stereo Harmonic analysis of lighting](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d485503460f94a2349a/html5/thumbnails/45.jpg)
Spherical Harmonics
Z YX
23 1Z XZ YZ22 YX XY
2 2 2 1X Y Z Positive values
Negative values
![Page 46: Analysis of Lighting Effects Outline: The problem Lighting models Shape from shading Photometric stereo Harmonic analysis of lighting](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d485503460f94a2349a/html5/thumbnails/46.jpg)
Harmonic Transform of Kernel
1.023
0.495
-0.111
0.05
-0.029
0.886
-0.5
0
0.5
1
1.5
0 1 2 3 4 5 6 7 8
00
( ) max(cos ,0) n nn
k k Y
nk
n
![Page 47: Analysis of Lighting Effects Outline: The problem Lighting models Shape from shading Photometric stereo Harmonic analysis of lighting](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d485503460f94a2349a/html5/thumbnails/47.jpg)
37.5
87.599.22 99.81 99.93 99.97
0
20
40
60
80
100
0 1 2 3 4 5 6 7 8
Cumulative Energy
N
1
N
nn
E
(percents)
![Page 48: Analysis of Lighting Effects Outline: The problem Lighting models Shape from shading Photometric stereo Harmonic analysis of lighting](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d485503460f94a2349a/html5/thumbnails/48.jpg)
Second Order Approximation
21 1 15( ) cos cos 2
64 2 64k
![Page 49: Analysis of Lighting Effects Outline: The problem Lighting models Shape from shading Photometric stereo Harmonic analysis of lighting](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d485503460f94a2349a/html5/thumbnails/49.jpg)
Reflectance Near 9D
0
4( )
2 1
n
n nm nmn m n
r k k l Yn
Yields 9D linear subspace.
4D approximation (first order) can also be used = point source + ambient
0
2 4( )
2 1
n
n nm nmn m n
k l Yn
![Page 50: Analysis of Lighting Effects Outline: The problem Lighting models Shape from shading Photometric stereo Harmonic analysis of lighting](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d485503460f94a2349a/html5/thumbnails/50.jpg)
Harmonic Representations
2(3 1)zn 2 2( )x yn n x yn n x zn n y zn n
zn xn yn
Positive values
Negative values( , , )x y zn n n n
ρ Albedo
n Surface normal
![Page 51: Analysis of Lighting Effects Outline: The problem Lighting models Shape from shading Photometric stereo Harmonic analysis of lighting](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d485503460f94a2349a/html5/thumbnails/51.jpg)
Photometric Stereo
LM
S
Image n
:
Image 1
Light n
:
Light 1
*
SVD recovers L and S up to an ambiguity
nz
nz
ny
nz2-1)
nx2-ny
2)
nxny
nxnz
nynz
r r
![Page 52: Analysis of Lighting Effects Outline: The problem Lighting models Shape from shading Photometric stereo Harmonic analysis of lighting](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d485503460f94a2349a/html5/thumbnails/52.jpg)
Photometric Stereo
![Page 53: Analysis of Lighting Effects Outline: The problem Lighting models Shape from shading Photometric stereo Harmonic analysis of lighting](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d485503460f94a2349a/html5/thumbnails/53.jpg)
Photometric Stereo
![Page 54: Analysis of Lighting Effects Outline: The problem Lighting models Shape from shading Photometric stereo Harmonic analysis of lighting](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d485503460f94a2349a/html5/thumbnails/54.jpg)
Summary
Lighting effects are complex Algorithms for SFS and photometric stereo for
Lambertian object illuminated by a single light source
Harmonic analysis extends this to multiple light sources
Handling specularities, shadows, and inter-reflections is difficult