The Color Of Water
John Breneman IV, Henryk Blasinski, Joyce Farrell
CASIS Workshop, LLNL
5/21/2014
U S I N G U N D E R WAT E R P H O T O G R AP H Y T O E S T I M AT E WAT E R Q U AL I T Y
Underwater color is a challenge
Surface color Manual color
Standard color Model-based color
This talk
1. Motivation
2. Extending the image formation model
a. Spectral basis functions
b. Medium scatter factors
3. Model evaluation
a. Prediction of real world dive data
b. Water content charts
4. Effect on color correction techniques
a. 3x3 vs 3x4 color corrections
Why a physical model for underwater lighting?
1. Better understanding of oceanic composition
A. What’s the “viz” at a popular dive site? Murky or Clear?
B. Is there a plankton bloom?
2. Insights for correcting underwater color images
A. Why are underwater dive filters so popular?
B. What form should our color correction take?
C. How can we identify the best color transform for a picture?
Roesler et al, 1989
Standard Linear Image Formation Model
m j = g jDl s j,lrlIll=1
N
å
Sunlight
Camera
Scene
Lossy Medium
m j = g jDl s j,lrlI0,le-Dpal
l=1
N
å
Camera
Scene
Medium
pD
eI
,0
Beer-Lambert
Law
Depth = Dp meters
Sunlight
Lossy Medium With Absorption Basis
m j = g jDl s j,lrlI0,le-Dpal
l=1
N
å
,,,, NAPCDOMw
,,,, NAPNAPCDOMCDOMw ccc
Lossy Medium With Absorption Basis
Camera
Scene
Medium
m j = g jDl s j,lrlI0,le-Dpal
l=1
N
å
Sunlight
Multilayer Lossy Medium
m j = g jDl s j,lrlI0,le- al ,lDdll=1
p
å
l=1
N
å
Medium 1
Medium 2
…
Medium p Camera
Scene
Sunlight
Multilayer Lossy Medium and Scatter
m j = g jDl s j,lrlI0,le- al ,lDdll=1
p
å
l=1
N
å +bs j
Medium 1
Medium 2
…
Medium p Camera
Scene
Scatter
Direct
Sunlight 6 unknowns: c{NAP,CDOM,Φ},bsj
Model Estimation
First term quasi-convex, for γ≥1
• m is smooth, convex
Use iterative first-order Taylor expansions for m → affine
Solve via cvx convex optimization toolbox
)(min ,
,,
,,,,
^
,,
lk
qtj
qtjqtjbsc
cRmmjlk
0, lkkcB
0jb
lk ,
j
qj
N d
tjjqtj bseIrsgm
p
l
ll
,
1
,0,,,,1
,
1,,
1
,)(
1)(
lklk
k l ll
lk ccdd
cR
For:
Test The Model With An Underwater Color Rig
! !(! ! ! )
= argmin ! ! ,! − !! ! ,! ! !! − ∇! ! ,! ! !
! ! !! ! ! − ! !
! − ! ! ,!! ,!
subject!to ! ! ! ! ,!! ! !
≥ 0 ! ! " #! ! " #,!! ! !
≥ 0 ! ! " #$ ! ! " #$ ,!! ! !
≥ 0
(11)
where
! ! ,! ! !!
= ! ! ! ! ! ! ,! ! ! ,!
!
! ! !
!! ,! !! ! ! !
(! )! ! !
!! !
(12)
∇! ! ,! ! !!
= ! ! ! ! ! ! ,! ! ! ,!
!
! ! !
!! ,! ! ! !! ! ! !
(! )! ! !
!! !
(13)
· Repeat the second step until all ! depth levels have been analyzed.
Note that ! ! ,! represent the measured sensor values of the k-th target in the j-th channel at the p-th depth level. The
subscript ! is ignored for convenience. The inequality constraint introduced in the optimization problem follows from
the non-negativity of the absorption coefficient values.
The convergence of this algorithm is guaranteed by the properties of convex functions. To confirm this, a numerical
stability was empirically verified using data simulated according to this model and solving for the unknown parameters.
Estimation error levels were comparable to the numerical precision.
4. DATA COLLECTION
To test our model for underwater illumination we constructed a color rig using a commercial point-and-shoot camera, a
Canon SX260HS, in an underwater housing and an XRITE color target. The XRITE color target was chosen primarily
because it is widely used and easily obtainable. Although the XRITE color patches themselves are not linearly
independent in the spectral space, they easily cover the spectral extent of the camera’s color filter sensitivities and
provide a sufficient basis for estimating the relative weights of the four absorption basis functions. The color target was
enclosed within a polycarbonate case to protect against the harsh underwater environment.
Figure 2. Underwater rig for color photography. The
color calibration target was an XRITE ColorChecker
(formally known at the Macbeth ColorChecker). The
calibration target was enclosed in a waterproof
polycarbonate casing. A depth gauge was visible in
all camera images in order to record the depth at
which each image was captured. The rig was
designed to minimize specular highlights and
shadows.
4.1 Camera
An alternative firmware for the camera, CHDK[11] was loaded to allow access to the raw image sensor data and to
program the camera to take a series of photos at a predefined interval. The camera was configured to operate in aperture-
priority mode while the exposure duration and ISO-sensitivity were selected automatically using the camera’s light
meter.
Test The Model With An Underwater Color Rig
! !(! ! ! )
= argmin ! ! ,! − !! ! ,! ! !! − ∇! ! ,! ! !
! ! !! ! ! − ! !
! − ! ! ,!! ,!
subject!to ! ! ! ! ,!! ! !
≥ 0 ! ! " #! ! " #,!! ! !
≥ 0 ! ! " #$ ! ! " #$ ,!! ! !
≥ 0
(11)
where
! ! ,! ! !!
= ! ! ! ! ! ! ,! ! ! ,!
!
! ! !
!! ,! !! ! ! !
(! )! ! !
!! !
(12)
∇! ! ,! ! !!
= ! ! ! ! ! ! ,! ! ! ,!
!
! ! !
!! ,! ! ! !! ! ! !
(! )! ! !
!! !
(13)
· Repeat the second step until all ! depth levels have been analyzed.
Note that ! ! ,! represent the measured sensor values of the k-th target in the j-th channel at the p-th depth level. The
subscript ! is ignored for convenience. The inequality constraint introduced in the optimization problem follows from
the non-negativity of the absorption coefficient values.
The convergence of this algorithm is guaranteed by the properties of convex functions. To confirm this, a numerical
stability was empirically verified using data simulated according to this model and solving for the unknown parameters.
Estimation error levels were comparable to the numerical precision.
4. DATA COLLECTION
To test our model for underwater illumination we constructed a color rig using a commercial point-and-shoot camera, a
Canon SX260HS, in an underwater housing and an XRITE color target. The XRITE color target was chosen primarily
because it is widely used and easily obtainable. Although the XRITE color patches themselves are not linearly
independent in the spectral space, they easily cover the spectral extent of the camera’s color filter sensitivities and
provide a sufficient basis for estimating the relative weights of the four absorption basis functions. The color target was
enclosed within a polycarbonate case to protect against the harsh underwater environment.
Figure 2. Underwater rig for color photography. The
color calibration target was an XRITE ColorChecker
(formally known at the Macbeth ColorChecker). The
calibration target was enclosed in a waterproof
polycarbonate casing. A depth gauge was visible in
all camera images in order to record the depth at
which each image was captured. The rig was
designed to minimize specular highlights and
shadows.
4.1 Camera
An alternative firmware for the camera, CHDK[11] was loaded to allow access to the raw image sensor data and to
program the camera to take a series of photos at a predefined interval. The camera was configured to operate in aperture-
priority mode while the exposure duration and ISO-sensitivity were selected automatically using the camera’s light
meter.
Image Comparison at 10m: Clear and Murky Water
Measured Predicted
Murk
y
Cle
ar
Water Content: Clear and Murky Water Murky Water Clear Water
10m Effective Absorption
Color Correction Underwater
Traditional color transform using 3x3 matrix:
Offset color transform using 3x4 matrix:
j
N d
jjj bseIrsgm
p
l
ll
1
,0,1
,
c
c
c
d
d
d
B
G
R
ccc
ccc
ccc
B
G
R
333231
232221
131211
134333231
24232221
14131211
c
c
c
d
d
d
B
G
R
cccc
cccc
cccc
B
G
R
15% - 50%
Color Correction Underwater – Murky Dive Example
Traditional color transform using 3x3 matrix:
Offset color transform using 3x4 matrix:
j
N d
jjj bseIrsgm
p
l
ll
1
,0,1
,
Reference 5 Meters 20 Meters
Reference 5 Meters 20 Meters
Summary
Changing depths changes natural illumination
• We can describe this with a physics-based model
Physics model is quasi-convex
• Solve regularized optimization iteratively
Particulate scatter is not handled well by 3x3 color correction schemes
• Build affine color correction transforms
Questions?
Acknowledgements:
Colin Roesler, Annick Bricaud ,Tetsuichi Fujiki, Sumit Chawla
Clear and Murky Water 20m Example
Corrected Measured
Murky
Clear
Corrected Measured
Why does clear water produce poor color?
Illuminant
Illuminant
Water Quality-Independent Color Solutions
1. Bring artificial light
• Expensive hardware
2. Apply an optical filter
• Tuned to depth, water content
3. Exposure Bracketing
• Long exposure Red
• Short exposure Green, Blue
• Apply 3x4 color transform
Backup – Bi-convex model fitting