![Page 1: Light, Surface and Feature in Color Images Lilong Shi Postdoc at Caltech Computational Vision Lab, Simon Fraser University](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649d035503460f949d639b/html5/thumbnails/1.jpg)
Light, Surface and Feature
in Color Images
Lilong ShiPostdoc at Caltech
Computational Vision Lab, Simon Fraser University
![Page 2: Light, Surface and Feature in Color Images Lilong Shi Postdoc at Caltech Computational Vision Lab, Simon Fraser University](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649d035503460f949d639b/html5/thumbnails/2.jpg)
Topics
Color Constancy
Surface Reflectance Model
Feature Analysis
![Page 3: Light, Surface and Feature in Color Images Lilong Shi Postdoc at Caltech Computational Vision Lab, Simon Fraser University](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649d035503460f949d639b/html5/thumbnails/3.jpg)
Color Formation
reflectance spectral
Illum. power distribution BGRkdSEp kk ,,)()()(
camera response
sensor sensitivity
Light
Surface
Feature
![Page 4: Light, Surface and Feature in Color Images Lilong Shi Postdoc at Caltech Computational Vision Lab, Simon Fraser University](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649d035503460f949d639b/html5/thumbnails/4.jpg)
Color Constancy
Light
Surface
Feature
![Page 5: Light, Surface and Feature in Color Images Lilong Shi Postdoc at Caltech Computational Vision Lab, Simon Fraser University](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649d035503460f949d639b/html5/thumbnails/5.jpg)
Automatic White Balance
AWB
Light
Surface
Feature
Canonical
![Page 6: Light, Surface and Feature in Color Images Lilong Shi Postdoc at Caltech Computational Vision Lab, Simon Fraser University](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649d035503460f949d639b/html5/thumbnails/6.jpg)
Color Constancy Methods
Retinex Theory (McCann64)
MaxRGB/White-Patch (Land77): max(R)
Gray-World (Buchsbaum80): mean(R)
Shades-of-Gray (Finlayson04): [mean(Rp)]1/p
Gray-Edge Hypothesis (Weijer07): mean(edge(R))
Non-Negative Matrix Factorization (Shi07)
Light
Surface
Feature
=
![Page 7: Light, Surface and Feature in Color Images Lilong Shi Postdoc at Caltech Computational Vision Lab, Simon Fraser University](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649d035503460f949d639b/html5/thumbnails/7.jpg)
Color Constancy Methods
Gamut Mapping (Forsyth90)
Color by Correlation (Finlayson01)
Neural Network (Cardei02)
Support Vector Regression (Xiong06)
Thin Plate Spline (Shi11)
Light
Surface
Feature
![Page 8: Light, Surface and Feature in Color Images Lilong Shi Postdoc at Caltech Computational Vision Lab, Simon Fraser University](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649d035503460f949d639b/html5/thumbnails/8.jpg)
Color Constancy Methods
Classification-based (Bianco09)
Scene-based (Gijsenij11)
Light
Surface
Feature
![Page 9: Light, Surface and Feature in Color Images Lilong Shi Postdoc at Caltech Computational Vision Lab, Simon Fraser University](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649d035503460f949d639b/html5/thumbnails/9.jpg)
Color Constancy Evaluation
MethodInpu
tTrain speed Para.
Relative Performance
Assumptions
Max-RGB img no very fast none poor white surface
Gray-World img no very fast none poor average gray
Shades-of-Gray img no moderate one moderate/good average gray
Edge-based Hyp. img no fast none moderate average gray
Color-by-Corre. hist yes fast a few moderate candidate illums
Neural-Network hist yes moderate some good none
Sup. Vector Reg. hist yes dep. train some moderate/good none
Thin-Plate-Spline thum yes dep. train a few good none
Light
Surface
Feature
![Page 10: Light, Surface and Feature in Color Images Lilong Shi Postdoc at Caltech Computational Vision Lab, Simon Fraser University](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649d035503460f949d639b/html5/thumbnails/10.jpg)
Blackbody Radiator Lights
Tungsten lamps, sunrise/sunset, sky light
Planckian locus
Narrowband sensors
Light
Surface
Feature
T
c
eIcTE2
51),(
2
51( ) ( ) ( ) , { , , }i
c
Ti i i i ip S E S Ic e i R G B
![Page 11: Light, Surface and Feature in Color Images Lilong Shi Postdoc at Caltech Computational Vision Lab, Simon Fraser University](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649d035503460f949d639b/html5/thumbnails/11.jpg)
Surface Reflectance Model
LIS Coordinate (Finlayson 01)
Light
Surface
Feature
![Page 12: Light, Surface and Feature in Color Images Lilong Shi Postdoc at Caltech Computational Vision Lab, Simon Fraser University](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649d035503460f949d639b/html5/thumbnails/12.jpg)
Achromatic Surface
Detection in LIS
Light
Surface
Feature
GraySurface
![Page 13: Light, Surface and Feature in Color Images Lilong Shi Postdoc at Caltech Computational Vision Lab, Simon Fraser University](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649d035503460f949d639b/html5/thumbnails/13.jpg)
Skin Color Model
Skin: melanin + hemoglobin
Skin Reflectance (Hiraoka et al 93)
Under blackbody illumination
Light
Surface
Feature
)]()()()()(exp[)( hhhmmm llS
pigment density
absorbance length in epidermis/dermi
s
absorbance of other material
c1ωσσΠ
bτb
SEp
hhmmhm ),,,(
![Page 14: Light, Surface and Feature in Color Images Lilong Shi Postdoc at Caltech Computational Vision Lab, Simon Fraser University](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649d035503460f949d639b/html5/thumbnails/14.jpg)
Skin Color Locus
Linear model
m is melanin basis,
h is hemoglobin basis,
is blackbody radiator basis,
c is a constant vector
Light
Surface
Feature
![Page 15: Light, Surface and Feature in Color Images Lilong Shi Postdoc at Caltech Computational Vision Lab, Simon Fraser University](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649d035503460f949d639b/html5/thumbnails/15.jpg)
Skin Tone Correction
Even simpler model:
Tone correction
Preserve melanin
16 different illum + camera
calibrations
( , )m m m τ Π σ ω
Light
Surface
Feature
![Page 16: Light, Surface and Feature in Color Images Lilong Shi Postdoc at Caltech Computational Vision Lab, Simon Fraser University](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649d035503460f949d639b/html5/thumbnails/16.jpg)
Features
Textures, edges, corner, blobs, etc..
Colors
Integrated by Quaternion
Light
Surface
Feature
![Page 17: Light, Surface and Feature in Color Images Lilong Shi Postdoc at Caltech Computational Vision Lab, Simon Fraser University](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649d035503460f949d639b/html5/thumbnails/17.jpg)
Quaternion
Real, complex, quaternion (q = a + bi + cj + dk)
Non-commutative (pq ≠ qp)
Convolution, Correlation, Fourier, Wavelet, etc
SVD, EVD, PCA
Light
Surface
Feature
![Page 18: Light, Surface and Feature in Color Images Lilong Shi Postdoc at Caltech Computational Vision Lab, Simon Fraser University](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649d035503460f949d639b/html5/thumbnails/18.jpg)
Texture Feature Extraction
Light
Surface
Feature
QPCA
Image-specific quaternion texture
basisSampled sub-windows
![Page 19: Light, Surface and Feature in Color Images Lilong Shi Postdoc at Caltech Computational Vision Lab, Simon Fraser University](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649d035503460f949d639b/html5/thumbnails/19.jpg)
Texture Feature Extraction
Light
Surface
Feature
Single quaternion
A texture patch
1st QPCA Basisma
gn
itu
de
re
al la
ye
rre
d la
ye
rg
re
en
la
ye
rb
lu
e la
ye
r
T
![Page 20: Light, Surface and Feature in Color Images Lilong Shi Postdoc at Caltech Computational Vision Lab, Simon Fraser University](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649d035503460f949d639b/html5/thumbnails/20.jpg)
Texture Feature 1st Feature
Light
Surface
Feature
magnitude
real layer i layer j layer k layer
![Page 21: Light, Surface and Feature in Color Images Lilong Shi Postdoc at Caltech Computational Vision Lab, Simon Fraser University](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649d035503460f949d639b/html5/thumbnails/21.jpg)
Segmentation
Quaternion Hoang(05)
Light
Surface
Feature
![Page 22: Light, Surface and Feature in Color Images Lilong Shi Postdoc at Caltech Computational Vision Lab, Simon Fraser University](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649d035503460f949d639b/html5/thumbnails/22.jpg)
Segmentation
Light
Surface
Feature
![Page 23: Light, Surface and Feature in Color Images Lilong Shi Postdoc at Caltech Computational Vision Lab, Simon Fraser University](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649d035503460f949d639b/html5/thumbnails/23.jpg)
Color Curvature
Light
Surface
Feature
![Page 24: Light, Surface and Feature in Color Images Lilong Shi Postdoc at Caltech Computational Vision Lab, Simon Fraser University](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649d035503460f949d639b/html5/thumbnails/24.jpg)
Iso-luminance
Color -> Gray
Cancellation in combining +/- derivatives
Light
Surface
Feature
![Page 25: Light, Surface and Feature in Color Images Lilong Shi Postdoc at Caltech Computational Vision Lab, Simon Fraser University](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649d035503460f949d639b/html5/thumbnails/25.jpg)
Hessian Descriptor
2nd order local shape
Principle Curvature
Light
Surface
Feature
2
22
2
2
2
),(
y
I
xy
Iyx
I
x
I
yxH eigenvectors: (e1, e2 )
eigenvalues: |1|<|2|
e1
e2
1
λ2
e1
e2 λ2
1
![Page 26: Light, Surface and Feature in Color Images Lilong Shi Postdoc at Caltech Computational Vision Lab, Simon Fraser University](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649d035503460f949d639b/html5/thumbnails/26.jpg)
Curvature
Tubular, vessel-like structures [Frangi98]
With eigen-values blobness: backgroundness: vesselness: R and S
Gray image, 2 λ’s; RGB image, 6λ’s
Light
Surface
Feature
![Page 27: Light, Surface and Feature in Color Images Lilong Shi Postdoc at Caltech Computational Vision Lab, Simon Fraser University](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649d035503460f949d639b/html5/thumbnails/27.jpg)
Color Curvature
Quaternion-valued Hessian
QSVD on H 2 real singular values
Light
Surface
Feature
k
y
B
xy
Byx
B
x
B
j
y
G
xy
Gyx
G
x
G
i
y
R
xy
Ryx
R
x
R
HQ
2
22
2
2
2
2
22
2
2
2
2
22
2
2
2
QT
Q UVHQ
![Page 28: Light, Surface and Feature in Color Images Lilong Shi Postdoc at Caltech Computational Vision Lab, Simon Fraser University](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649d035503460f949d639b/html5/thumbnails/28.jpg)
Curvature Detection
Light
Surface
Feature
Frangi Quaternion
![Page 29: Light, Surface and Feature in Color Images Lilong Shi Postdoc at Caltech Computational Vision Lab, Simon Fraser University](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649d035503460f949d639b/html5/thumbnails/29.jpg)
Future Works
Content-based color constancy
Color blob/points detection
Possibilities …
Light
Surface
Feature