intelligent color temperature estimation using fuzzy neural network with application to automatic...

Expert Systems with Applications 38 (2011) 7718–7728

Contents lists available at ScienceDirect

Expert Systems with Applications

journal homepage: www.elsevier .com/locate /eswa

Intelligent color temperature estimation using fuzzy neural network withapplication to automatic white balance

Cheng-Lun Chen ⇑, Shao-Hua LinDepartment of Electrical Engineering, National Chung Hsing University, Taichung 40227, Taiwan

a r t i c l e i n f o a b s t r a c t

Keywords:Color temperature estimationFuzzy neural networkWhite balance

0957-4174/$ - see front matter � 2010 Elsevier Ltd. Adoi:10.1016/j.eswa.2010.12.137

⇑ Corresponding author. Tel.: +886 4 22851549x70E-mail address: [email protected] (C.-L.

In this paper, a novel white balance method is proposed, which consists of two fundamental steps, i.e.,color temperature estimation of the illuminant and adjustment of the components of the color separates.A fuzzy logic system is constructed to infer the color temperature of an illuminant which a digitallyacquired image subjected to. The fuzzy logic system is further optimized by representing the systemas a fuzzy neural network and a training scheme of the FNN parameters is proposed. The estimated colortemperature is then employed to determine the required amount of adjustment for each color separate ofthe image. Experimental procedures are outlined and performed, which validates the feasibility andperformance of the proposed method. A comparative study is also made based on two quantitative indi-ces. The study shows that the proposed approach is preferable to most methods in the literatures in termsof performance and robustness to varieties of illuminants and scenes.

� 2010 Elsevier Ltd. All rights reserved.

1. Introduction

Digital image capturing devices have evolved into a class ofconsumer electronics indispensable for modern family. For mostimage capturing or recording devices, image signals from sensors(CCD or CMOS) must be converted into physical quantities compat-ible with storage devices or monitors. The conversion should takeinto account the fact that dynamic response of the sensor is differentfrom that of the human visual system. That is, the images viewed bysensors are different from that by the human eyes. In addition, theimages captured are actually lights reflected from the surface ofthe objects. Hence, the conversion should also consider the effectof the light source. Color balance is a class of image processing algo-rithms developed to compensate for the effects of light sources oncaptured images. Images which are not color balanced will appearto have been shifted towards one color or another, i.e., have colorcast. A color balance algorithm adjusts the image through red, green,and blue values of the three basic color layers so that the hues of avariety of colors are returned to normal. Since human eyes areparticularly sensitive to the neutral (gray or white) colors withinan image, compensating for such colors has become the main objec-tive of most color balance algorithms. Color balance algorithms aim-ing at compensating the white color within captured images are alsoknown as white balance algorithms. Most image related applica-tions require taking measurement of the colors from surroundingobjects or scenes, but often neglect the fact that the colors of an

ll rights reserved.

4; fax: +886 4 22851410.Chen).

object or scene may subject to change with factors such as ambientlight and illuminants. When invariance of color measurement tosuch factors is important for an application, white balance algorithmmay be incorporated into the corresponding image processing pipe-line to ensure equitable measurement of colors.

A generic white balance algorithm usually consists of twoessential steps: (1) estimating the color temperature of the lightsource for a captured image; (2) adjusting the image contents toremove the effect of the light source. Many high-end digital cam-eras and camcorders have built-in sensors which can measurelighting conditions in real-time and make corresponding correc-tion to the captured images. On the other hand, the market of con-sumer electronics relies largely on the sale of low-end cameras,which must estimate the lighting conditions based on the contentsof the captured images using software algorithm. Existing whitebalance algorithms fall into the following four categories: grayworld method (Buchsbaum, 1980; Gershon, Jepson, & Tsotsos,1988), Retinex method (Blake, 1985; Brainard & Wandell, 1986;Hurlbert, 1986; Land, 1986; McCann, McKee, & Taylor, 1976),gamut mapping method (Finlayson, 1996; Finlayson & Hordley,2000; Forsyth, 1990), and specular reflection method (D’Zmura &Lennie, 1986; Lee, 1986; Tominaga & Wandell, 1989; Tominaga &Wandell, 1990). Gray world method is based on gray worldassumption: Suppose that an image has abundant colors understandard light source (e.g., D65), the average value of its red (R),green (G), and blue (B) separates would be gray, i.e., R = G = B.Therefore, adjustment may be applied to each color separate to en-sure that the modified image satisfies the assumption. Retinexmethod assumes that a normal scene contains various objects

http://dx.doi.org/10.1016/j.eswa.2010.12.137

mailto:[email protected]

http://dx.doi.org/10.1016/j.eswa.2010.12.137

http://www.sciencedirect.com/science/journal/09574174

http://www.elsevier.com/locate/eswa

B

object A

object B

object C

light source #1

C.-L. Chen, S.-H. Lin / Expert Systems with Applications 38 (2011) 7718–7728 7719

which reflect full intensity of the red, green, and blue componentsof the light source. Hence, the maximum red, green, and blue com-ponent values from a captured image may be utilized to estimatethe color of the light source. Gamut mapping method establishesa database of the R/G/B components of the light reflected from allpossible object surfaces under known standard light source, whichforms a convex hull Hs. As next step, various R/G/B component pairsfrom the captured image under unknown light source are mappedinto Hs and form convex hulls Hu, u = 1,2 , . . .. Then the color of theunknown light source is expected to lie in the intersected region ofHu’s. Specular reflection method is based on the fact that if the sur-face of an object is shiny enough, it reflects the actual color of thelight source, independently of the color of the surface. Therefore,the R/G/B components of the brightest or most reflective spot ina captured image can be used to scale the R/G/B values of the otherimage area. Based on a literature review, several main issues arestill not addressed or resolved conclusively. In particular, mostmethods are not robust in the sense that their performance de-pends largely on the context of the scene or the light source. Forexample, gray world method is not useful for scenes having largearea of uniform color. Retinex and specular reflection methodsare not suitable for scenes full of abrasive surfaces or subject to asaturated light source. Furthermore, most methods require makingreasoning and judgment according to the contents of a scene, butfew employs fuzzy systems or neural networks to formulate andsolve the problem. In a word, a desirable white balance algorithmshould incorporate intelligence such that it can adapt to genericimages and is robust to varieties of light sources and scenes.

In this paper, a novel white balance method which comprisesthe aforementioned two essential steps is proposed. A fuzzy infer-ence/logic system is formulated which is capable of correlating adigitally captured image to the color temperature of the illuminantwhich the image is subject to. The capability of the fuzzy inferencesystem is further optimized by representing the system as a fuzzyneural network (FNN) and the parameters of the network areadjusted using a back-propagation algorithm. Specifically, a proce-dure is specified to choose a set of training samples and initial con-ditions for the FNN, which considers both static and dynamicstatistical variation among image pixels of a color sample. To per-form white balance, the estimated color temperature is then uti-lized to determine the required amount of adjustment for eachcolor separate of the image. In particular, the proposed methodcan tackle the issue of an image having zero intensity in part ofthe red or blue color separate. Experimental procedures are sug-gested and conducted to verify the feasibility and performance ofthe proposed method. Comparison with other existing methodsis also made, for which two quantitative indices are implemented.With regard to performance, both quantitative and qualitative re-sults indicate that the proposed approach is superior to mostmethods in the literatures. The proposed method also demon-strates better potentiality of being robust to varieties of illumi-nants and scenes.

Next section begins with formulation of a fuzzy logic systemfor estimation of the color temperature of an unknown illuminant,which is followed by construction and training of a FNN to improvethe performance of the system, and by utilization of the temperatureestimation to perform white balance. Section 3 covers description ofexperimental procedures, results, and discussions. A comparativestudy is also made. Conclusion is given in Section 4.

R Gobject D

light source #2

Fig. 1. The colors reflected from four monochromatic objects (labeled as A, B, C, andD) follow different trajectories in RGB color space when subject to differentilluminants.

2. Automatic white balance with fuzzy neural network

In this section, the framework of a fuzzy inference system isfirst proposed, which is capable of deducing the color temperatureof an unknown illuminant. To optimize the ability, a FNN is

constructed to represent the formulated fuzzy system, which willbe trained using typical back propagation algorithm. The estimatedcolor temperature can then be utilized to white balance a testimage.

2.1. Estimation of the color temperature of an unknown illuminantformulated as a fuzzy inference system

As depicted in Fig. 1, the colors reflected from four monochro-matic objects (labeled as A, B, C, and D) follow different trajectoriesin RGB color space when subject to different illuminants. Whenilluminating by light source #1, four color pairs of (reflected)R/G/B components may be found corresponding to those fourmonochromatic objects. When illuminating by light source #2,another four pairs of R/G/B components may be found. Althoughdifferent monochromatic objects might appear the same in colorunder some illuminants (known as metamerism), one-to-one map-ping between each color pair and a specific illuminant will formgradually as the number of monochromatic objects is increased.Suppose that there are M monochromatic objects and N illumi-nants. Both have known color or spectral properties and thusmay serve as references. Let (Rij,Gij,Bij) represent the color pair ofthe ith reference object under the jth reference illuminant. The fol-lowing IF-THEN rules may be established:

IFðR11;G11;B11Þ AND ðR21;G21;B21Þ AND . . . ANDðRM1;GM1;BM1ÞTHEN Light source#1IFðR12;G12;B12Þ AND ðR22;G22;B22Þ AND . . . ANDðRM2;GM2;BM2ÞTHEN Light source#2

..

.

IFðR1N ;G1N ;B1NÞ AND ðR2N;G2N;B2NÞ AND . . . ANDðRMN;GMN;BMNÞTHEN Light source #N:

In practical circumstance, it is not possible to know precisely thecolor or spectral properties of an object or illuminant. First of all,a monochromatic object with a surface of uniform spectral proper-ties (e.g., transmittance and reflectance) hardly exits in real world.Therefore, slight variation is expected when taking measurementof the colors from different surface area of a monochromatic object.Other factors will also contribute to perturbation of Rij, Gij, and Bij

values, such as fluctuation in spectral power distribution of an illu-minant, measurement inaccuracy of the equipment, and inferencefrom the environment. Alternatively, Rij, Gij, and Bij may be specifiedas fuzzy sets which are defined over intervals in accordance withtheir variation ranges. The color temperature of the illuminantmay also be specified as a fuzzy set defined over an interval. Letx1, x2, and x3 represent one measurement of R, G, and B components,respectively, from an image subject to an illuminant with color

7720 C.-L. Chen, S.-H. Lin / Expert Systems with Applications 38 (2011) 7718–7728

temperature y. The following NM fuzzy inference rules may beconstructed:

Rule 1 � Rule M: If (x1,x2,x3) is Ai11 � Ai

21 � Ai31, then y is B1,

i = 1, . . . ,M.Rule M + 1 � Rule 2M: If (x1,x2,x3) is Ai

12 � Ai22 � Ai

32, then y is B2,i = 1, . . . ,M....

Rule (N � 1)M + 1 � Rule NM: If (x1,x2,x3) is Ai1N � Ai

2N � Ai3N ,

then y is BN, i = 1, . . . ,M.

where � denotes taking ‘AND’ operation of the fuzzy sets, e.g.,[A1 � A2 � A3](x1,x2,x3) = A1(x1)A2(x2 )A3(x3). Besides, Ai

1j;Ai2j, and

Ai3j represent the fuzzy sets of Rij, Gij, and Bij, respectively, corre-

sponding to the ith reference object subject to the jth referenceilluminant. The outputs of the NM rules may be further combinedby a defuzzification algorithm. Here the centroid method or centeraverage defuzzifier is utilized, i.e.,

y ¼PN

j¼1

PMi¼1 Ai

1j � Ai2j � Ai

3j

h iðx1; x2; x3Þ

� �ycjPN

j¼1

PMi¼1 Ai

1j � Ai2j � Ai

3j

h iðx1; x2; x3Þ

� � ; ð1Þ

where ycj denotes the centroid of the fuzzy set Bj. The formulatedfuzzy logic system is such that it accepts the values of R, G, and Bcomponents from a captured image under an unknown illuminantand is capable of making deduction of the color temperature ofthe illuminant.

2.2. Configuration and training of the fuzzy neural network

To enhance the performance and improve the robustness of thefuzzy logic system developed previously. The fuzzy logic systemmay be topologically depicted as a neural network as shown inFig. 2. Let u(k) and a(k) be the input and the output of the kth layerof the network. The functions of the four layers and the corre-sponding nodes and links may be described as follows:

Layer 1: No computation is done in this layer. Here x1, x2, and x3

are the inputs to layer 1 of the neural network, which has 3nodes and transmits the input values directly to the next layer,i.e.,

að1Þ ¼ uð1Þi ¼ xi: ð2Þ

Layer 2: This layer has 3NM nodes and performs the fuzzifica-tion operation. Each node in this layer consists of the fuzzyset, Ai

1j; Ai2j, or Ai

3j, and each fuzzy set is specified as a Gaussianmembership function, i.e.,

1x 2x 3x

� � � � � �

��

Fig. 2. Configuration of the fuzzy neural network.

að2Þ ¼ exp � uð2Þi �mi

ri

!28<:

9=;; ð3Þ

where mi and ri are, respectively, the center (or mean) and thewidth (or variance) of the Gaussian membership function of theith input variable ui. The link weights between layer 1 and 2 areset to unity.

Layer 3: This layer has NM nodes with each node performingthe ‘AND’ operation, i.e.,

að3Þ ¼Y3

i¼1

uð3Þi : ð4Þ

The output of this layer represents the firing strength of the corre-sponding fuzzy rule. The link weights between layer 2 and 3 are alsoset to unity.

Layer 4: This layer consists of a single node which acts as adefuzzifier by integrating all the signals from layer 3 usingthe centroid method, i.e.,

að4Þ ¼PNM

i¼1uð4Þi wiPNMi¼1uð4Þi

ð5Þ

where wi is the centroid of the output membership function and canbe regarding as the link weights between layer 3 and 4.

Training of the FNN involves adjusting the center/width, mi andri, of each Gaussian membership function in layer 2 and the linkweights, wi, between layer 3 and 4 such that the output generatedby the network for a given input pair is as close to the reference ordesired color temperature as possible. Mathematically, the objec-tive is to minimize the error function

EðtÞ ¼ 12½yðtÞ � ydðtÞ�2 ¼ 1

2

PNMi¼1uð4Þi wiPNM

i¼1uð4Þi

� ydðtÞ" #2

; ð6Þ

where yd(t) is the desired output and y(t) is the actual output. Foreach training sample, starting at the input nodes, a forward passis used to compute the activity levels of all the nodes in the networkto obtain the current output y(t) according to (2)–(5).

A back-propagation type algorithm (Huo, Chang, Wang, & Wei,2006) is employed to adjust or update the three aforementionedset of parameters. With the error function defined in (6), the up-date rules for the free parameters in the FNN are summarized asfollows:

(1) Update rule of wk (the link weight between layer 3 and 4):The update rule of wk is

wkðt þ 1Þ ¼ wkðtÞ � g@EðtÞ@wk

; ð7Þ

where

@EðtÞ@wk

¼ @EðtÞ@yðtÞ

@yðtÞ@wk

¼ ðyðtÞ � ydðtÞÞ uð4ÞkPNMi¼1uð4Þi

: ð8Þ

(2) Update rule of mk (the center of the membership function inlayer 2): The update rule of mk is

mkðt þ 1Þ ¼ mkðtÞ � g@EðtÞ@mk

; ð9Þ

where g is the learning rate and


@EðtÞ@mk

¼ ðyðtÞ � ydðtÞÞXNM

i¼1

@að4Þ

@uð4Þi

@uð4Þi

@mk


i¼1

@að4Þ

@uð4Þi

X3

j¼1

@að3Þ

@uð3Þj

@uð3Þj

@mk


i¼1

@að4Þ

@uð4Þi

X3

j¼1

@að3Þ

@uð3Þj

� exp � uð2Þk �mk

rk

!28<:

9=;

2 uð2Þk �mk

� �r2

k

0@

1A: ð10Þ

(3) Update rule of rk (the width of the membership function inlayer 2): The update rule of rk is

rkðt þ 1Þ ¼ rkðtÞ � g@EðtÞ@rk

; ð11Þ

where

@EðtÞ@mk


i¼1

@að4Þ

@uð4Þi

@uð4Þi

@rk


i¼1

@að4Þ

@uð4Þi

X3

j¼1

@að3Þ

@uð3Þj

@uð3Þj

@rk


i¼1

@að4Þ

@uð4Þi

X3

j¼1

@að3Þ

@uð3Þj

� exp � uð2Þk �mk

rk

!28<:

9=;

2 uð2Þk �mk

� �2

r3k

0B@

1CA: ð12Þ

Fig. 4. A color checker subject to different illuminants (a) D65–6500 K (b) CWF-4150 K (c) U30–3000 K (d) A-2856 K.

Once the neural network finishes the training phase, a test im-age captured under an unknown illuminant may be used to evalu-ate the capability of the network for prediction of the colortemperature. The FNN may be fed by all or certain fraction of theimage pixels from the test image. Since the network will deduceor estimate a color temperature for each provided image pixel,additional maneuver is required to obtain a representative colortemperature based on all the estimated temperatures. First, a his-togram for all the estimated temperatures is created. There are afew sensible options for the second step. Let Ti denote each individ-ual estimated temperature and freq(Ti) the corresponding fre-quency in the histogram. The color temperature for the testimage may be set to the one with the maximum frequency in thehistogram, i.e.,

Fig. 3. Experimental set

E:T: ¼ argTi

max freqðTiÞ: ð13Þ

Another method is to discard temperature data with low frequen-cies due to high possibility of the corresponding image pixel beingcontaminated by noise or beyond the interpolating/extrapolatingdomain of the FNN. The color temperature for the test image isset to the average of the temperatures whose frequencies greaterthan or equal to a specified threshold fth, i.e.,

E:T: ¼ 1ct

Xi

Ti; ð14Þ

where

i ¼ the indices k0s such that freqðTkÞP fth

ct ¼ number of i0s

2.3. White balance adjustment

The R, G, and B components of a reference white object subjectto various illuminants of known color temperature are collected.The R, G, and B components of the white object subject to a desiredilluminant (i.e., D65 with color temperature of 6500 K) can then be

up and procedure.

Fig. 5. Histograms for the R/G/B components of the candidate pixels from two of the color patches subject to D65 illuminant.

0 100 200 300 400 500 600 700 800 900 10000

0.02

0.04

0.06

0.08

0.1

0.12

0.14

0.16

0.18

0.2

Training cycle

RM

SE

X: 1000Y: 0.0002486

Fig. 6. Root mean square error versus training cycle.

Table 1The R, G, B components of the reference white object.

2856 K 3000 K 4150 K 6500 K

Red 255 244 237 212Green 179 182 196 208Blue 88 106 138 207

2500 3000 3500 4000 4500 5000 5500 6000 650080

100

120

140

160

180

200

220

240

260

Color temperature (K)

Col

or c

ompo

nent

val

ues

RedGreenBlue

Fig. 7. Curve fitting the data of the R/G/B components of the reference white objectunder various illuminants. (For interpretation of the references to colour in thisfigure legend, the reader is referred to the web version of this article.)


estimated via data interpolation or extrapolation. The R, G, and Bcomponents of the white object subject to the unknown illuminant(i.e., the one under which the image is captured) may also be esti-mated according to the estimated color temperature from the pre-vious section. To finalize white balance, pixel-wise operation in thered, green, and blue color separates of the test image is performedaccording to

Fig. 8. Before being white balanced: (a) reference image-6500 K, (b) test image-4150 K, (c) test image-3000 K, (d) test image-2856 K. After being white balanced: (e) testimage-4150 K, (f) test image-3000 K, (g) test image-2856 K.

Table 3The MSE between test and reference images (in YCbCr color space).

Illuminants for the test images Color temperature

4150 K 3000 K 2856 KBefore being white balanced 283.9 599.94 943.68

After being whitebalanced

SRM 203.25 503.21 705.31GWM 170.1 360 612.66Lam’s method 137.4 260.98 403.55Chikane’smethod

97.21 136.39 422.15

Huo’s method 159.46 528.45 825.72Chen’s method 98.49 173.01 274.62Wang’s method 75.78 459.8 797.9The proposedmethod

50.42 84.62 154.29


Rawb ¼ R� RD65=Rc;

Gawb ¼ G� GD65=Gc;

Bawb ¼ B� BD65=Bc;

ð15Þ

where Rawb, Gawb, and Bawb denote the red, green and blue compo-nents for the white balanced image. In addition, R, G, and B arethe red, green and blue components from the test image. RD65,GD65, and BD65 are the components of the white object under D65illuminant, whereas Rc, Gc, and Bc are the components of the whiteobject under the unknown illuminant.

It is observed that the red or blue color separate of an imagesubject to high or low color temperature tends to have lots of im-age pixels with zero intensity. Those pixels will not be affected by(15) and will therefore degrade the white balanced image. To solvethis issue, (15) may be modified as follows:

Rawb ¼ ðRþmaxð1; ðE:T:� 6500Þ=100ÞÞ � RD65=Rc

�maxð1; ðE:T:� 6500Þ=100ÞGawb ¼ G� GD65=Gc

Bawb ¼ ðBþmaxð1; ð6500� E:T:Þ=100ÞÞ � BD65=Bc

�maxð1; ð6500� E:T:Þ=100Þ;

ð16Þ

where E.T. is the estimated color temperature of the unknown illu-minant. The idea behind the proposed modification is to shift theintensity of the red or blue color separate (add a positive constant)when subject to illuminants of high or low color temperature, ad-just the pixel values, and then shift the intensity of the color sepa-rate back (add a negative constant).

Table 2The MSE between test and reference images (in RGB color space).


4150 K 3000 K 2856 KBefore being white balanced 855.79 1849.67 2836.34


SRM 449.08 1249.71 2076.18GWM 463.75 995.06 1686.37Lam’s method 390.37 806.87 1252.19Chikane’smethod

319.6 496.46 1513.39

Huo’s method 537.4 2023.04 2690.84Chen’s method 242.44 509.47 835.64Wang’smethod

214.83 1453.42 2446.12

The proposedmethod

134.48 236.55 375.05

3. Experimental procedures and results

Fig. 3 illustrates the experimental setup and procedure. A colorchecker containing patches/segments may serve as reference ob-jects with different surface colors. A high-end digital camera witha native resolution of 3888 � 2592 pixels is utilized to acquire rawimage data. Note that the raw image is not affected by the camera’sbuilt-in color balance functions. The raw data is then converted toTIF format which preserves the original lighting condition of theimage. A light booth provides various illuminants with known col-or temperature (A: incandescent of 2856 K, U30: warm white fluo-rescent of 3000 K, CWF: cool white fluorescent of 4150 K, and D65:daylight of 6500 K) inside which the color checker can be placed

Table 4The average chromaticity values of the test images.


4150 K 3000 K 2856 K Indoorscene

Before being white balanced 32.81 43.9 52.38 25.77


SRM 20.91 42.23 48.1 23.46GWM 0.03 0.06 0.4 0.02Lam’s method 5.25 8.23 9.34 4.35Chikane’smethod

9.34 16.05 17.3 7.2

Huo’s method 22.23 34.5 45.37 2.17Chen’s method 2.15 4.79 5.35 0.9Wang’smethod

14.89 38.61 47.68 8.44

The proposedmethod

11.26 10.41 18.09 6.48


with desired orientation. Among them, D65 is the desired or refer-ence illuminant, and the image captured under D65 can be re-garded as the reference image (with desired white balance).Other images captured under A, U30 and CWF are treated as testimages (to be white balanced). The reference image is shown inFig. 4(a), and the test images subject to other illuminants are givenin Fig. 4(b), 4(c), and 4(d).

We propose the following method, which takes into accountstatistical variation (both static and dynamic) among image pixels,to select a set of reference objects which provide samples and ini-tial conditions for training the FNN. First, the color checker is posi-tioned properly inside the light booth. For each known illuminant,raw image of the color checker is collected six times using thehigh-resolution digital camera, with each taken 20 s apart. Pixelsof a specified area (370 � 370 pixels) from each color patch onthe raw image become candidates of training samples. Hence,there are a set of 370 � 370 � 6 candidate pixels correspondingto each color patch, which account for time and spatial variationof that specific color. For each color patch, histograms are gener-ated for the R/G/B components of the candidate pixels. Fig. 5 dem-onstrates the histograms for two of the color patches subject toD65 illuminant. Twenty candidate pixels lying within the intervalof one standard deviation from the mean are then randomly se-lected as the training samples with respect to an illuminant of

Fig. 9. (a) Reference image. (b) Test image-4150 K. After being white balanced using (c) Smethod 3, (i) Wang’s method, and (j) the proposed method.

known color temperature. The means and standard deviations ofthe R/G/B histograms also provide sensible initial values of the cen-ter and width of the membership functions in layer 2 of the FNN.The centroid of the output membership function is set to 0. The de-sired output is set to the normalized color temperatures of theknown illuminants, i.e., [2856,3000,4150,6500]/6500. For eachtraining cycle of the FNN, a training sample and a desired normal-ized color temperature are presented to the network. At the end ofeach training cycle, the square root of (6) or the root mean squareerror (RMSE) is recorded. Note that the learning rate g is set to0.005, and the number of training cycles is set to 1000. The RMSEversus training cycle is shown in Fig. 6. The RMSE is 0.151036 atthe end of the first training cycle and is reduced to 0.005041 after1000 training cycles.

After training of the FNN is complete, test images (see Fig. 4(b),4(c), and 4(d)) are then utilized to demonstrate the proposed whitebalance approach. Using the method described previously with athreshold of 0.5, the estimated color temperatures are 4151 K,2993 K, and 2850 K, which are very close to the actual color tem-peratures of the illuminants. Using the white color patch as the ref-erence white object, of which the R, G, and B components undervarious illuminants are listed in Table 1, the data from Table 1can be curve fitted using polynomials of degree two as shown inFig. 7. As a final step, (16) is applied and the white balanced images

RM, (d) GWM, (e) Lam’s method, (f) Chikane’s method, (g) Huo’s method, (h) Chen’s


are provided in Fig. 8. All the white balanced images, i.e., Fig. 8(e),(f) and (g), can be concluded to be visually closer to the referenceimage, Fig. 8(a), than the original images, i.e., Fig. 8(b), (c) and (d).

A study is made which compares the proposed white balancemethod with other methods in existing literature, i.e., gray worldmethod (GWM) (Buchsbaum, 1980; Gershon et al., 1988), specularreflection method (SRM) (D’Zmura & Lennie, 1986; Lee, 1986;Tominaga & Wandell, 1989; Tominaga & Wandell, 1990), Lam’smethod (Lam, Au, & Wong, 2004), Chikane’s method (Chikane &Fuh, 2006), Huo’s method (Huo et al., 2006), Wang’s method(Wang & Fuh, 2007), and Chen’s method (Chen & Lin, 2011). Thestudy employs the same set of reference image and test imagesas given in Fig. 4. Each test image is processed by each individualalgorithm and produces a white balanced image. To compare theresults quantitatively, two performance indices are introduced.The average chromaticity (AC), commonly used in the literatures(e.g., (Chikane & Fuh, 2006; Huo et al., 2006; Wang & Fuh, 2007),for a white balanced image is given by

AC ¼ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiðCbÞ2 þ ðCrÞ2

q; ð17Þ

where Cb and Cr are mean values of the corresponding Cb and Crcolor separates. Since calculation of the AC value does not involve

Fig. 10. (a) Reference image. (b) Test image-3000 K. After being white balanced using (c)method 3, (i) Wang’s method, and (j) the proposed method.

a reference image (with desired balanced color), it is arguable tostate that a method producing a white balanced image with low va-lue of AC performs better than the others. Therefore, another perfor-mance index for comparative study will be utilized. When areference/desired image is available, it is common practice to usemean square error (MSE) for quantitative evaluation of how wellan image processing algorithm performs (Wang & Bovik, 2006). Inthis study, we have a reference image, which is the image digitallycaptured under desired illuminant (D65). In addition, white balanceis inherently an image processing algorithm which intends to mod-ify the captured image so that the processed image is close to whenit is captured under desired illuminant. Hence, it is plausible toincorporate MSE as a mathematical tool for evaluation of white bal-ance algorithm. The MSE between the reference image and a testimage (before and after being white balanced) is calculated by

MSE ¼ 13MN

XM

i

XN

j

½ðDXði; jÞÞ2 þ ðDYði; jÞÞ2 þ ðDZði; jÞÞ2�; ð18Þ

where M and N are the resolutions of the image, D X(i, j), DY(i, j), andDZ(i, j) are the differences of the color components (in RGB or YCbCrcolor space) at (i, j) pixel between the images. The MSE between thereference image and the white balanced test image is calculated for

SRM, (d) GWM, (e) Lam’s method, (f) Chikane’s method, (g) Huo’s method, (h) Chen’s

Fig. 11. (a) Reference image. (b) Test image-2856 K. After being white balanced using (c) SRM, (d) GWM, (e) Lam’s method, (f) Chikane’s method, (g) Huo’s method, (h) Chen’smethod 3, (i) Wang’s method, and (j) the proposed method.


each algorithm. The average chromaticityffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiC2

b þ C2r

qis also

calculated. The quantitative and qualitative results are summarizedin Tables 2–4 and Figs. 9–11, respectively. The algorithms are alsoapplied to a test image acquired from an indoor scene with un-known ambient light. The results are shown in Table 4 andFig. 12. It can be seen that all algorithms produce satisfactory whitebalanced images when the color temperature of the unknown illu-minant is near that of the reference illuminant (see Fig. 9). As theunknown illuminant deviates more from the reference illuminant,most algorithms exhibit appreciable performance degradation,e.g., SRM, Huo’s method, and Wang’s method (see Figs. 10 and11). Nevertheless, the proposed method produces decent white bal-anced images with minimum MSE for all three cases. Another issueworth notice is that the average chromaticity indicates that GWMshould produce the best white balanced images for all four cases.This, however, does not agree with the visual data given in Figs.9–11. A qualitative ranking of the algorithms, based on the quanti-tative data obtained previously, is given in Table 5, of which execu-tion time, algorithmic complexity, and performance are compared.It can be seen that SRM and GWM demand short execution timeand low algorithmic complexity but result in poor performance.Lam’s and Huo’s methods consume more execution time, have med-ian complexity, and give mediocre performance. Note that a methodwith long execution time does not necessarily offer good perfor-

mance, e.g., Wang’s method. Moreover, a method executing fastcould also possess decent performance, e.g., Chikane’s method.Due to the incorporation of a FNN, the method proposed in thispaper demands more execution time and have relatively highcomplexity. However, the proposed method performs better thanthe others in terms of MSE as the performance index.

4. Conclusion

This paper proposes a novel method with experimental proce-dures for white balancing an image subject to unknown illuminant.A fuzzy logic system is established and plays the role of makingdeduction of the color temperature of the illuminant which a dig-itally captured image is subject to. The parameters of the fuzzy sys-tem are further optimized using the framework of a fuzzy neuralnetwork with specifically chosen training samples and a backpropagation algorithm. A formula of proportional adjustment withzero intensity compensation for red and blue color separate is thensuggested to utilize the estimated color temperature to finalizewhite balance of an image. The feasibility and effectiveness ofthe proposed method are justified by experimental results. A com-parative study is also made, which demonstrates the superiority ofthe proposed method to other methods in the literatures. The

Fig. 12. (a) An indoor scene subject to unknown illuminant. After being white balanced using (b) SRM, (c) GWM, (d) Lam’s method, (e) Chikane’s method, (f) Huo’s method,(g) Chen’s method, (h) Wang’s method, and (i) the proposed method.

Table 5Ranking of comparison of all methods.

Method Time (sec.) Complexity Performance

MSE Chromaticity

SRM 6.8156 (2) Low 8 8GWM 3.3031 (1) Low 5 1Lam’s method 71.1281 (5) Middle 4 3Chikane’s method 19.0438 (4) High 3 5Huo’s method 106.7594 (6) Middle 7 7Chen’s method 14 (3) High 2 2Wang’s method 774.1407 (8) Middle 6 6The proposed method 268.1813 (7) High 1 4


study also implies that MSE is a better index than AC to evaluatethe performance of a white balance algorithm when a referenceimage (with desired illuminant) is available. Although the pro-posed algorithm is relatively complex than existing methods, thefollowing strategy may be considered when implementing thealgorithm in actual imaging product (digital camera or camcorder):The most complex part of the algorithm, which involves configura-tion and training of the FNN, may be performed during the manu-facturing and assembly phases of the product. Once the productleaves the factory, only the second part of the algorithm, i.e.,(16), which is simple and demands little execution time, is acti-vated regularly. Note that the interpolation/extrapolation data of

the reference white may be stored in advance in the memory ofan imaging device. Therefore, the values of Rc,Gc, and Bc may be re-trieved instantly using a lookup table. When further parametricadjustment of the algorithm is requested, e.g., training of theFNN, the user may switch the device into special operation modeto carry out the corresponding calibration, which will demandlonger execution time.

References

Blake, A. (1985). Boundary conditions for lightness computation in Mondrian world.Computer Vision, Graphics, and Image Processing, 32, 314–327.

Brainard, D. A., & Wandell, B. A. (1986). Analysis of the Retinex theory of colorvision. Journal of the Optical Society of America A, 3, 1651–1661.

Buchsbaum, G. (1980). A spatial processor model for object color perception. Journalof the Franklin Institute, 310, 1–26.

Chen, C.-L., & Lin, S.-H. (2011). Formulating and solving a class of optimizationproblems for high-performance gray world automatic white balance. AppliedSoft Computing, 11(1), 523–533.

Chikane & Fuh, C.-S. (2006). Automatic white balance for digital still cameras.Journal of Information Science and Engineering, 22(3), 497–509.

D’Zmura, M., & Lennie, P. (1986). Mechanisms of color constancy. Journal of theOptical Society of America A, 3, 1662–1672.

Finlayson, G. D. (1996). Color in perspective. IEEE Transactions on Pattern Analysisand Machine Intelligence, 18, 1034–1038.

Finlayson, G. D., & Hordley, S. (2000). Improving Gamut mapping color constancy.IEEE Transactions on Image Processing, 9(10), 1774–1783.

Forsyth, D. (1990). A novel algorithm for color constancy. International Journal ofComputer Vision, 5, 5–36.

Gershon, R., Jepson, A. D., & Tsotsos, J. K. (1988). From [R,G,B] to surface reflectance:Computing color constant descriptors in images. Perception, 755–758.


Huo, J.-Y., Chang, Y.-L., Wang, J., & Wei, X.-X. (2006). Robust automatic whitebalance algorithm using gray color points in images. IEEE Transactions onConsumer Electronics, 52(2), 541–546.

Hurlbert, A. (1986). Formal connections between lightness algorithms. Journal of theOptical Society of America A, 3, 1684–1692.

Lam, H. -K., Au, O. C. & Wong, C. -W. (2004). Automatic white balancing usingluminance component and standard deviation of RGB components. In IEEEInternational conference on acoustics, speech, and signal processing, Montreal,Canada (pp. 493–496).

Land, E. H. (1986). Recent advances in Retinex theory. Vision Research, 26, 7–21.Lee, H.-C. (1986). Method for computing the scene-illuminant chromaticity from

specular highlights. Journal of the Optical Society of America A, 3, 1699–1964.

McCann, John J., McKee, Suzanne P., & Taylor, Thomas H. (1976). Quantitativestudies in Retinex theory. Vision Research, 16, 445–458.

Tominaga, S., & Wandell, B. A. (1989). Standard surface-reflectance model andilluminant estimation. Journal of the Optical Society of America A, 6, 576–584.

Tominaga, S., & Wandell, B. A. (1990). Component estimation of surface spectralreflectance. Journal of the Optical Society of America A, 7, 312–317.

Wang, Z., & Bovik, A. C. (2006). Modern Image Quality Assessment. Morgan & ClaypoolPublishers.

Wang, P.-M., & Fuh, C.-S. (2007). Automatic white balance with color temperatureestimation. IEEE International Conference on Consumer Electronics, 1–2.

intelligent color temperature estimation using fuzzy neural network with application to automatic...

Documents