1

Practical Scene Practical Scene Illuminant Estimation via Illuminant Estimation via

Flash/No-Flash PairsFlash/No-Flash Pairs

Cheng Lu and Mark S. Drew

Simon Fraser University{clu, mark}@cs.sfu.ca

2

Flash/No-flash Imagery – a Brief History

diCarlo, Xiao, & Wandell, CIC 2001Combine flash/no-flash images to produce a pure-flash image.Use dim=3 FDM + knowledge of flash SPD and sensor curves to estimate surface reflectance most likely ambient illuminant

Raskar et al., Non-Realistic Rendering 2004

Filling in night-time imagery with daytime image info.

Copy edges from cloned image region into edge-map of target background; re-integrate.

Blake et al., Poisson Image Editing, Siggraph 2004

Szeliski et al., Siggraph 2004

Transfer lower-noise information from flash image to higher-noise ambient-light image.

Find shadow-mask, copy edges inside shadow from flash image into ambient image, re-integrate.

Drew,Lu,Finlayson, Removing Shadows using Flash/Noflash Image Edges , ICME 2006

3

This paper:

Estimate Ambient Illuminant, usingFlash/No-flash Pairs

Like diCarlo&Wandell approach, but replace knowledge of camera sensor curves with a camera RGB-based calibration using difference of with-flash and no-flash images. How?

- Spectral sharpening- Subtract “both” – “no-flash” pure-flash image- Log’s- Project difference of flash minus ambient into geometric-mean chromaticity color space

Calibrate such to get illuminant chromaticity.

4

What’s the point?:

Can estimate scene (ambient) illuminant without knowing:

- Flash SPD- Camera sensors- Surface reflectance

5

Why estimate the illuminant?

White balance, plus many computer vision applications == intrinsic imageswithout illumination.

- Simple- Fast

What’s good about this method?

6

The set-up:

2 images , one under ambient lighting, & another under flash.

Under Ambient: Image “A”. Under Both: Image “B”.

+

7

The Key: Pure-Flash Image The ambient light from “A” is also in “B”. Therefore if we subtract the two, we have “F”: the pure-flash image.

Under Flash: Image “F”:

+ - =)(

8

Incidentally, note that there are now extra shadows, from the flash(since it’s offset from the lens).

Image “F”: the scene as imaged under Flash light only.

9

1. Lambertian surface:

RGB =

Shading = normal effective light-direction

Illum

inant

Surfa

ceSen

sors

Simple Image Formation Model

will guide us.

Assumptions: 1., 2., 3.

10

)(kQ

2. Narrow-band sensors:

so then

is exactly a single-spike sensor:

11

3. Planckian light:

But, can violate 1., 2., 3. and still succeed.

(in Wien’s approximation)

Gives

12

-Now take Log’s, to pull apart multiplications:

Camera-dep’t vector

Camera-dep’t vector

Intensity and shading

Surface

Color-temperature of light

where

13

Camera-dep’t vector

Surface

Color-temperature of light

So form geometric-mean chromaticity:

-We’d like to remove intensity/shading term:

In logs:

where

14

-The point:

As temp (light color) changes, move along straight line.

-But, we have “A” and “F” images: Subract them, and use same chromaticity trick Only illumination is left!

15

Log-difference Geometric-Mean Chromaticity

So log-log delivers inverse-temperature difference:

-Calibrate for 1/TA-1/TF, then in new scene obtain TA!

{

16

What does this

look like?

Moved to 2D; color-matching functions in geo-mean chromaticity. (9 Planckians, Macbeth ColorChecker, spike sensors, xenon flash SPD)

17

Sony DXC930 sensors, Daylights+F2, actual xenon flash SPD:

“Reference locus”

How to proceed: -Sharpen- Find closest cluster

18

Effect of sharpening:

Poor clusters Better clusters#’ing

Kodak DCS420:

19

Test: can we determine the illuminant?

102 illuminants, Sony camera, Munsell patches

102 illuminants, Sony camera, Macbeth patches

Estimate illum.from Munsell to Macbeth

Nearly 100% correctly identified.

20

Application: White Balance

4 calibration illuminants, HP camera, Macbeth chart(each cluster has 24 dots)

No flash

With flash

- Sharpen- Sample image at 24 locationsevenly over image-Same (“daylight”) color balancefor training and for testing

Image under CWF; CWF+Xenon

21

Overlaps best with CWF, so usewhite patch of Macbeth under CWF for white balance:

“Auto” balance – Wrong.

“Fluor” balance – Correct.

Our color-balance– Much closer.

22

Thanks!To Natural Sciences and

Engineering Research Council of Canada