Vol. 11, No. 5/May 1994/J. Opt. Soc. Am. A 1553

Spectral sharpening: sensor transformationsfor improved color constancy

Graham D. Finlayson, Mark S. Drew, and Brian V. Funt

School of Computing Science, Simon Fraser University, Vancouver, B.C., Canada V5A 1S6

Received March 8, 1993; revised manuscript accepted October 28, 1993; accepted October 28, 1993

We develop sensor transformations, collectively called spectral sharpening, that convert a given set of sensorsensitivity functions into a new set that will improve the performance of any color-constancy algorithmthat is based on an independent adjustment of the sensor response channels. Independent adjustment ofmultiplicative coefficients corresponds to the application of a diagonal-matrix transform (DMT) to the sensorresponse vector and is a common feature of many theories of color constancy, Land's retinex and von Kriesadaptation in particular. We set forth three techniques for spectral sharpening. Sensor-based sharpeningfocuses on the production of new sensors as linear combinations of the given ones such that each new sensorhas its spectral sensitivity concentrated as much as possible within a narrow band of wavelengths. Data-based sharpening, on the other hand, extracts new sensors by optimizing the ability of a DMT to accountfor a given illumination change by examining the sensor response vectors obtained from a set of surfacesunder two different illuminants. Finally in perfect sharpening we demonstrate that, if illumination andsurface reflectance are described by two- and three-parameter finite-dimensional models, there exists a uniqueoptimal sharpening transform. All three sharpening methods yield similar results. When sharpened conesensitivities are used as sensors, a DMT models illumination change extremely well. We present simulationresults suggesting that in general nondiagonal transforms can do only marginally better. Our sharpeningresults correlate well with the psychophysical evidence of spectral sharpening in the human visual system.

Key words: spectral sharpening, color constancy, color balancing, lightness, von Kries adaptation.

1. INTRODUCTION

The performance of any color-constancy algorithm,whether implemented biologically or mechanically, willbe strongly affected by the spectral sensitivities of thesensors providing its input. Although in humans thecone sensitivities obviously cannot be changed, we neednot assume that they form the only possible input to thecolor-constancy process. New sensor sensitivities can beconstructed as linear combinations of the original sen-sitivities, and in this paper we explore what the mostadvantageous such linear transformations might be.

We call the sensor response vector for a surface viewedunder an arbitrary test illuminant an observation. Theresponse vector for a surface viewed under a fixed canoni-cal light is called a descriptor. We take as the goal ofcolor constancy the mapping of observations to descrip-tors. Since a descriptor is independent of illumination,it encapsulates surface reflectance properties.'

In the discussion of a color-constancy algorithm thereare two separate issues: the type of mechanism or ve-hicle supporting the transformation from observations todescriptors in general and the method used to calculatethe specific transformation that is applicable under a par-ticular illumination. In this paper we address only theformer and therefore are not proposing a completely newtheory of color constancy.

A diagonal-matrix transformation (DMT) has beenthe transformation vehicle for many color-constancy al-gorithms, in particular von Kries adaptation,2 all theretinex/lightness algorithms,3 -5 and, more recently,Forsyth's gamut-mapping approach.6 All these algo-rithms respond to changing illumination by adjusting the

response of each sensor channel independently, althoughthe strategies that they use to decide on the actual ad-justments differ.

DMT support of color constancy is expressed mathe-matically in relation (1). Here pe denotes an observa-tion (a 3-vector of sensor responses), where i and e areindex surface reflectances and illumination, respectively.The vector pic represents a descriptor and depends onthe single canonical illuminant. The diagonal transformDe best maps observations onto descriptors. Through-out, the superscript e denotes dependence on a variableilluminant and the superscript c denotes dependence onthe fixed canonical illuminant. Boldface indicates vectorquantities.

In general, there may be significant error in this ap-proximation. Indeed West and Brill2 and D'Zmura andLennie7 have shown that a visual system equipped withsensors having the same spectral sensitivity as the hu-man cones can achieve only approximate color constancythrough a DMT.

A DMT will work better with some sensor sensitivitiesthan with others, as one can see by considering how theillumination, surface reflectance, and sensor sensitivitiescombine in the formation of an observation. An observa-tion corresponds to

p = f E(A)S(A)R(A)dA, (2)where E(A), S(A), and R(A) denote illumination, surfacereflectance, and sensor sensitivities, respectively, and the

0740-3232/94/051553-11$06.00 © 1994 Optical Society of America

Finlayson et al.

Pi C Depie. (1)

1554 J. Opt. Soc. Am. A/Vol. 11, No. 5/May 1994

integral is taken over the visible spectrum . For a DMTto suffice in modeling illumination change,2 it must bethe case, given a reference reflectance Sr, a secondaryarbitrary reflectance S, and illuminants Ei and Ei, that,

f Ei(A)S(A)R(A)dA f EJ(A)S(A)R(A)dA@ = @ * ~~~~~~~~~~~(3)

f Ei(A)Sr(A)R(A)dA fEi(A)S(A)R(A)dAAs others have observed, one way to ensure that this

condition holds is to use extremely narrow-band sensors,which in the limit leads to sensors that are sensitive to asingle wavelength (Dirac delta functions).6 Our intuitionwhen we began this work was that if we could find a lin-ear combination of sensor sensitivities such that the newsensors would be sharper (more narrow band), then theperformance of DMT color-constancy algorithms shouldimprove and the error of relation (1) would be reduced.It should be noted, however, that Eq. (3) can be satisfiedin other ways, such as by the placement of constraints onthe space of illuminants or reflectances.2 With the addi-tion of sharpening, relation (1) becomes

Tpi - Tpi, (4)

where T denotes the sharpening transform of the originalsensor sensitivities. It is important to note that applyinga linear transformation to response vectors has the sameeffect as application of the transformation to the sensorsensitivity functions.

The sharpening transform effectively generalizesdiagonal-matrix theories of color constancy. Otherauthors2 36 also have discussed this concept of an inter-mediate (or sharpening) transform. However, our workappears to be the first to consider the precise form of thistransform.

The sharpening transform is a mechanism throughwhich the inherent simplicity of many color-constancyalgorithms can be maintained. For example, Land'sretinex algorithm requires color ratios to be illuminationindependent (and hence implicitly assumes a diagonal-matrix model of color constancy), which, as can be seenfrom Eq. (3), they generally will not be. It seems dif-ficult to improve the accuracy of retinex ratioing di-rectly without making the overall algorithm much morecomplicated8 ; however, if a simple, fixed sharpeningtransformation of the sensors as a preprocessing stageis applied, the rest of the retinex process can remainuntouched. Similar arguments apply to Forsyth's stan-dards for coefficient rule (CRULE)6 and Brill's9 volumet-ric theory.

We initially present two methods for calculating T:sensor-based and data-based sharpening. Sensor-basedsharpening is a general technique for determining thelinear combination of a given sensor set that is maxi-mally sensitive to subintervals of the visible spectrum.This method is founded on the intuition that narrow-band sensors will improve the performance of DMTtheories of color constancy. We apply sensor-basedsharpening over three different ranges in order to gen-erate three new sharpened sensors that are maximallysensitive in the long-wave, medium-wave, and shortwavebands. Figure 1 below contrasts the cone fundamentals

derived by Vos and Walraven10 (VW) before and aftersharpening. Although the new sensitivity functions aresharper, they are far from meeting the intuitive goal ofbeing very narrow band (i.e., with strictly zero response inall but a small spectral region); nonetheless, we performsimulations that show that they in fact work much betterthan the unsharpened cone fundamentals. In Section 2we present the details of sensor-based sharpening.

Sensor-based sharpening does not take into account thecharacteristics of the possible illuminants and reflect-ances but considers only the sharpness of the resultingsensor. Our second sharpening technique, data-basedsharpening, is a tool for validating the sensor-basedsharpening method. Given observations of real surfacereflectances viewed under a test illuminant and theircorresponding descriptors, data-based sharpening findsthe best, subject to a least-squares criterion, sharpen-ing transform T'f. Interestingly, data-based sharpeningyields stable results for all the test illuminations thatwe tried, and in all cases the data-based-derived sensorsare similar to the fixed sensor-based sharpened sensors.Data-based sharpening is presented in Section 3.

In Section 4 we present simulations evaluatingdiagonal-matrix color constancy for sharpened andunsharpened sensor sets. Over a wide range of illu-minations, sensor-based sharpened sensors provide asignificant increase in color-constancy performance.

Data-based sharpening is related to Brill's9 volumetrictheory of color constancy. This relationship is exploredin Section 5. Through spectral sharpening, the volumet-ric theory is shown to be informationally equivalent toLand'sl white-patch retinex.

The data-based sharpening technique finds the opti-mal sharpening transform for a single test illuminant.In Section 6 we investigate the problem of finding a goodsharpening transform relative to multiple illuminants.We show that if surface reflectances are three dimen-sional and illuminants two dimensional there exists asharpening transform with respect to which a diagonalmatrix supports perfect color constancy. This analysisconstitutes a third technique for deriving the sharpeningtransform.

In Section 7 we relate our work specifically to theoriesof human color vision. Sharpened spectral sensitivitieshave been measured in humans.12- 6 We advance thehypothesis that sharpened sensor sensitivities arise as anatural consequence of optimization of the visual system'scolor-constancy abilities through an initial linear trans-formation of the cone outputs.

2. SENSOR-BASED SHARPENING

Sensor-based sharpening is a method of determining thesharpest sensor, given an s-dimensional (s is usually 3)sensor basis R(A) and wavelength interval [Al, A2]. Thesensor Rt(A)c, where c is a coefficient vector, is maximallysensitive in [Al, A2] if the percentage of its norm lying inthis interval is maximal relative to all other sensors. Wecan solve for c by minimizing:

I = L [R(A)tc]2dA + p, [R(A)tc]2dA - ,

Finlayson et al.

I

(5)

Vol. 11, No. 5/May 1994/J. Opt. Soc. Am. A 1555

where (o is the visible spectrum, 0 denotes wavelengthsoutside the sharpening interval, and jz is a Lagrange, mul-tiplier. The Lagrange multiplier guarantees a nontrivialsolution for Eq. (5): the norm of the sharpened sensor isequal to 1. Moreover, this constraint ensures that thesame sharpened sensor is recovered independent of theinitial norms of the basis set R(A).

By differentiating with respect to c and equating to thezero vector, we find the stationary values of Eq. (5):

2 aI = f R(A)R(A)tcdA + pff R(A)R(A)tcdA =0.(6)

Differentiating with respect to /,t simply yields the con-straint equation f, [R(A)tc]2 dA = 1. The solution ofEq. (6) can thus be carried out, assuming that the con-straint holds.

Define the s x s matrix A(a) = It R(A)R(A)t dA so thatEq. (6) becomes

A()c = -A(w)c. (7)

Assuming a nontrivial solution c 0, , 0 0 -and rearrang-ing Eq. (7), we see that solving for c (and consequently forthe sharpened sensor) is an eigenvector problem:

[A(co)]-'A(5)c = -nc. (8)

There are s solutions of Eq. (8), each solution corre-sponding to a stationary value, so we choose the eigenvec-tor that minimizes f¢ [R(A)tc]2 dA. It is important thatc be a real-valued vector, as it implies that our sharpenedsensor is a real-valued function. That c is real valued fol-lows from the facts that the matrices [A())]-' and A(+)are positive definite and that eigenvalues of the product oftwo positive-definite matrices are real and nonnegative.' 7

Solving for c for each of three wavelength intervals yieldsa matrix '2' for use in relation (4). The matrix C isnot dependent on any illuminant and denotes the sensor-based sharpening transform.

We sharpened two sets of sensor sensitivities: thecone absorptance functions measured by Bowmaker andDartnell' 5 (BOW) and the cone fundamentals derived byVos and Walraven' 0 (VW), which take into account thespectral absorptions of the eye's lens and macular pig-ment. The BOW functions were sharpened with respectto the wavelength intervals (in nanometers) [400,480],[510,550], and [580,650] and the VW fundamentalswith respect to the intervals [400,480], [520,560], and[580,650]. (All spectra used in this paper are in therange 400-650 nm measured at 10-nm intervals.) Theseintervals were chosen to ensure that the whole visiblespectrum would be sampled and that the peak sensitivi-ties of the resulting sensors would roughly correlate withthose of the cones.

The results for the VW sensors are presented in Fig. 1(those for the BOW sensors are similar), where it can beseen that the sharpened curves contain negative sensitivi-ties. These need not cause concern in that they do notrepresent negative physical sensitivities but merely repre-sent negative coefficients in a computational mechanism.Clearly, the sharpening intervals are somewhat arbitrary.

They were chosen simply because the resulting sharpenedsensors appeared, to the human eye, significantly sharperthan the unsharpened sensors. Nevertheless, the inter-vals used are sensible, and their suitability is verified bythe fact that they provide sharpened sensors that are inclose agreement with those derived by data-based sharp-ening as described in Section 3. The actual values of thec's in Eq. (8) are given in Section 6 below.

Figure 1 contrasts the sharpened VW sensor set withthe corresponding unsharpened set; the degree of sharp-ening is quite significant. The peak sensitivities ofthe new sensors are shifted with respect to the initialsensitivities; this shift is due to both the choice of sharp-ening intervals and the shape of the VW sensitivities.The sharpened long-wavelength mechanisfn is pushedfurther toward the long-wavelength end of the spectrum;in contrast, the medium-wavelength mechanism isshifted toward the shorter wavelengths; and the short-wavelength mechanism remains essentially the same.Intriguingly, field sensitivities of the human eye mea-sured under white-light conditioning with long testflashes13 are sharpened in an analogous manner.

Table 1 contrasts the percentage-squared norm lying inthe sharpening intervals for the original versus the sharp-ened curves. For both the VW and the BOW sensors thedegree of sharpening is significant. Furthermore, from

0.4

' > 0.3

. - 0.2E Ct (o 0.1

0.0

-0.1

. .

EN :U)E

' a)Z to

z.N>

0.4

0.3

0.2

0.1

0.0

0.4

0.3

0.2

0.1

0.0I

/ / \

400 450 500 550 600 650Wavelength

400 450 500 550 600 650Wavelength

. , ;_,--,----- ,- ;-/

400 450 500 550 600 650Wavelength

Fig. 1. VW fundamentals (solid curves) are contrasted withthe sharpened sensitivities derived through sensor-based(dotted curves) and data-based (dashed curves) sharpening.Top, long-wavelength mechanism; middle, medium-wavelengthmechanism; bottom, short-wavelength mechanism.

Finlayson et~ al.

1556 J. Opt. Soc. Am. A/Vol. 11, No. 5/May 1994 Fnasne l

Table 1. Percentage of Total Squared Normin the Sharpening Intervals

% Squared NormSensors [400,480] [510,550] [580,650]

BOW, original 98.9 51.4 30.1BOW, sharpened 99.4 66.3 76.2

[520,560]VW, original 97.5 67.6 40.3VW, sharpened 97.8 74.7 89.1

Ire~ is unique also, always exists, and equals [e]-l, fordiagonal [De].

It is interesting to compare Eq. (12) with the relationfor the problem of finding the best general transform G that maps observations obtained under a test illuminantto their corresponding descriptors:

Wc,: G e~e. (13)

Fig. it is clear that the sharpening effect is not limitedto the sharpened interval.

3. DATA-BASED SHARPENINGIt could be the case that the best sensors for DMT al-gorithms might vary substantially with the type of -lumination change being modeled. If so, sensor-basedsharpening, which does not take into account any of thestatistical properties of collections of surfaces and illumi-nants, might perform well in some cases and poorly inothers. To test whether radically different sharpeningtransformations are required in different circumstances,we explore a data-based approach to deriving sharpenedsensitivities, in which we optimize the sensors for DMT al-gorithms by examining the relationship between observa-tions, obtained under different test illuminants, and theircorresponding descriptors.

Let WI be a 3 n matrix of descriptors generated froma set n of surfaces observed under a canonical illuminantEc. Similarly, let We be the matrix of observations of nsurfaces imaged under another test illuminant El. Tothe extent that DMT-based algorithms suffice for colorconstancy, WC and We should be approximately equiva-lent under a DMT:

WC =::DeWe. (9)

The diagonal transform is assumed to be an approxi-mate mapping and will have a certaln degree of error.The idea of spectral sharpening is that this approximationerror can be reduced if We and We are first transformedby a matrix rfe;

q'eWc =:De'eWe. (10)

D e will in fact be optimal in the least-squares sense if it

is defined by the Moore-Penrose inverse:

De= rf~ewc[9-ewe]+,(1

where + denotes the Moore-Penrose inverse. (TheMoore-Penrose inverse of the matrix A is defined as

A+- At[AAt]-1.) Now [Te] must be chosen to ensurethat [De] is diagonal. To see how to do this, carry outsome matrix manipulation to obtain

[T'e]-lveje = Wc[We]+. (12)

Since the eigenvector decomposition of the matrix onthe right-hand side of Eq. (12), Wc[We]+ = UleDe[(Je]-l,is unique, its similarity to the left-hand side implies that

Such optimal fitting effectively bounds the possible per-formance within a linear model of color constancy. Whenthe approximation of relation (13) is to be optimized in theleast-squares sense, G I is simply

(14)

Equation (12) can be interpreted, therefore, as simplythe eigenvector decomposition of the optimal generaltransform G .Of course, it is obvious that the opti-mal transform could be diagonalized; what it is impor-tant is that if one knows a sharpening transform Te~],then the best least-squares solution relating ffeWc and

IWe is precisely the diagonal transformation De; thatis, TeWc[cyeWe]+ = De. In other words, when one isusing the sharpened sensors, the optimal transform isguaranteed to be diagonal, so finding the best diagonaltransform after sharpening is equivalent to finding theoptimal general transform. Therefore sharpening allowsus to replace the problem of determining the nine pa-rameters of GeI with that of determining only the threeparameters of De.

Data-based sharpening raises two main questions: (1)Will the resulting sensors be similar to those obtained bysensor-based sharpening, and (2) will the derived sensorsvary substantially with the illuminant? To answer thesequestions requires the application of data-based sharpen-ing to response vectors obtained under a single canonicalilluminant and several test illuminants. For illuminantswe used five Judd daylight spectra' 9 and CIE standard il-luminant A,10 and as reflectances we used the set of 462Munsell spectra 20 We arbitrarily chose Judd's D55 (55stands for 5500 K) as the canonical illuminant; descrip-tors are the response vectors for surfaces viewed underD55. For each of the other illumninants El, we derivedthe data-based sharpening transform Tje in accordancewith Eq. (12).

Figure 2 shows for the VW sensors the range of the fivesets of data-based sharpened sensors obtained for map-ping between each of the five illuminants and D55. Forthese five illuminants, the results are remarkably stableand hence relatively independent of the particular illumi-nant, so the mean of these sharpened sensors character-izes the set of them quite well. Referring once again toFig. 1, we can see that these mean sensors are very simi-lar to those derived through sensor-based sharpening.

The stability of the results for the five cases and thesimilarity of these results to the sensor-based resultsis reassuring; nonetheless, it would be nice to find thesharpening transformation that optimizes over all theilluminants simultaneously. This issue is addressed inSection 6, where we show that, given illuminant and re-flectance spectra that are two and three dimensional,

Finlayson et al.

Ge = Wc[Wel+.

Vol. 11, No. 5/May 1994/J. Opt. Soc. Am. A 1557

0.4

~0 0.3,N .> 0.2a _E n; ' 0.1

0.0

-0.1

0.4

0.3

0.2

0.1

0.0

0.4

0.3

0.2

0.1

0.0

400 450 500 550 600 650Wavelength

400 450 500 550 600 650Wavelength

40Q 450 500 550 600 650Wavelength

Fig. 2. Data-based sharpening generates different sharpenedsensors for each illuminant. The range of sharpened curvesover all the test illuminants (CIE A, D48, D65, D75, andD100) mapped to D55 is shown for the VW cone mechanisms.Top, long-wavelength mechanism; middle, medium-wavelengthmechanism; bottom, short-wavelength mechanism.

respectively, there exists a unique optimal sharpeningtransform.

4. EVALUATING SHARPENED SENSORS

Since the sensor-based and data-based sharpened sensorsare similar, we restrict our further attention to the evalu-ation of sensor-based sharpened sensors alone. The re-sults for VW and BOW sensors are similar as well, so weinclude figures only for the VW case.

For each illuminant we generated sensor responsesfor our 462 test surface reflectances, using both thesharpened and the unsharpened VW sensors so that wecould determine how much sharpening improves DMTperformance.

To measure the performance of a DMT in mapping testobservations to canonical descriptors, we compare fit-ted observations (observations mapped to the canonicalilluminant by means of a diagonal matrix) with corre-sponding (canonical) descriptors. The Euclidean dis-tance between a fitted observation qe and its descriptorpC, normalized with respect to the descriptor's length,provides a good error metric, given the definition of colorconstancy that we are using. This percent normalized

fitted distance (NFD) metric is defined as

NFD = 100 * IIp - qej1IOpCI (15)

Let WI be a 3 X 462 matrix of descriptors correspond-ing to the 462 surfaces viewed under the canonical illu-minant. Similarly, let WI denote the 3 X 462 matrix ofobservations for the 462 surfaces viewed under a test illu-minant. Relation (9) can then be solved to yield the bestdiagonal transformation in the least-squares sense; thisprocedure will be called simple diagonal fitting. SinceD e is a diagonal matrix, each row of We is fitted indepen-dently. The components of De are derived as follows:

D,, = Wi [We]+ = Wvc[We,]t/Wie[We]t , (16)

where the single subscript i denotes the ith matrix rowand the double subscript ii denotes the matrix element atrow i column i.

Given a fixed set of sensor functions, Eq. (16) yieldsthe best diagonal transformation that takes observationsunder the test illuminant onto their corresponding de-scriptors. Simple diagonal fitting, therefore, does not in-clude sharpening but rather for a fixed set of sensors findsthe best diagonal matrix DI for that set of sensors.

For performance comparisons we are also interested inthe NFD that results under transformed diagonal fitting.Transformed diagonal fitting proceeds in two stages:

1. 'tWc D"ec'We, where ' is the fixed sharpeningtransform and D I is calculated by means of Eq. (16);

2. q1`19jWc r1-lDeTwe.

Application of T`1 transforms the fitted observationsback to the original (unsharpened) sensor set so that anappropriate comparison can be made between the fittingerrors: the performance of sharpened diagonal-matrixconstancy is measured relative to the original unsharp-ened sensors.

Figure 3 shows NFD cumulative histograms for diago-nal fitting of VW observations (solid curves) and diago-nal fitting of sharpened VW observations (dotted curves).For each illuminant the sharpened sensors show bet-ter performance than the unsharpened ones, as indi-cated by the fact that in the cumulative NFD histogramsthe sharpened sensor values are always above those forthe unsharpened ones. In general the performance dif-ference increases as the illuminant color becomes moreextreme: from D55 to D100 the illuminants become pro-gressively bluer and toward CIE A, redder.

Figure 4 shows the cumulative NFD histograms for di-agonal fitting of VW observations (solid curves) and trans-formed diagonal fitting of sharpened VW observations(dashed curves). Once again it is clear that the sharp-ened sensors perform better; however, the performancedifference is greater, which shows that the question ofsensor performance is linked to the axes in which colorspace is described.

Finally, Fig. 5 contrasts the cumulative NFD histo-grams for optimal fitting of VW observations [the unre-stricted least-squares fit of relation (13) (solid curves)]and transformed diagonal fitting of sensor-based sharp-

C) .N>X ,_

z n

x >a) N >

E Ci a)Z Un

Finlayson et al.

1558 J. Opt. Soc. Am. A/Vol. 11, No. 5/May 1994

100

80

_ 60

840

20

100

80

60

C 40

20

CIE A

0 2 4 6 8 10%error

D75

D48100

80

C 60

40

20

0

100

80

n 600QC 40

20

0

D65

100

80

"I

60

. 40

20

0 2 4 6 8 10%error

D1 00100

80

_ 60

C 40

20

0

0 2 4 6 8 10%error

Average

0.2 4 6 810 0 2 4 6 810 0 2 4 6 8 10%error %error %error

Fig. 3. Cumulative NFD histogram obtained with each testilluminant (CIE A, D48, D65, D75, and D100) for diagonalfitting of VW observations (solid curves) and diagonal fittingof sensor-based sharpened VW observations (dotted curves).The sixth cumulative NFD histogram shows the average fittingperformance.

Finlayson et al.

ened VW observations (dashed curves). For these cases,with the exception of CIE A, a DMT achieves almostthe same level of performance as the best nondiagonaltransform.

5. DATA-BASED SHARPENINGAND VOLUMETRIC THEORY

Data-based sharpening is a useful tool for validatingour choice of sensor-based sharpened sensors. However,more than this, data-based sharpening can also be viewedas a generalization of Brill's9 volumetric theory of colorconstancy. In that theory Brill develops a method forgenerating illuminant-invariant descriptors that is basedon two key assumptions:

1. Surface reflectances are well modeled by a three-dimensional basis set and are thus defined by a surfaceweight vector oa. For example, S(A) = 1 Si(A)o-1.

2. Each image contains three known reference re-flectances. In the discussion that follows, Qe denotes the3 x 3 matrix of observations for the reference patches seenunder Ee(A).

Given the first assumption, observations for surfacesviewed under Ee(A) are generated by application of alighting matrix (a term first used by Maloney2 1) to surfaceweight vectors:

pe = A eo, (17)

where the ijth entry of Ae is equal to f Rj(A)Ee(A)Sj(A)dA.

100

80

U 60

2C 40

20

0

CIE A

0 2 4 6 8 10%error

D75

100

80

= 60al

C 40

20

0

D48

0 2 4 6 8 10%error

D1 00

D65

100

80

CC 60

c 40

20

0

0 2 4 6 8 10%error

Average

CIE A

100

80

608

C 40

20

0 2 4 6 8 10%error

D75

100

80

- 60

C 40

20

D48

0 2 4 6 8 10%error

D1 00

100

80

_60

C 40

20

D65

0 2 4 6 8 10%error

Average100

80

- 60

C 4021

20

0

100

80

-60

c, 40

20

0

100

80

-C 60

C 40

20

0

100

80

C 40

20

0

0 2 4 6 8 10 0 2 4 6 810 0 2 4 6 810%error %error %error

Fig. 4. Cumulative NFD histogram obtained with each testilluminant (CIE A, D48, D65, D75, and D100) for diagonal fittingof VW observations (solid curves) and transformed diagonal fit-ting of sensor-based sharpened VW observations (dotted curves).The sixth cumulative NFD histogram shows the average fittingperformance.

100

80

-60S

C 40

20

0

100

80

60

C 4021

20

0

0 2 4 6 810 0 2 4 6 810 0 2 4 6 810%error %error %error

Fig. 5. Cumulative NFD histogram obtained with each testilluminant (CIE A, D48, D65, D75, and D100) for optimal fittingof VW observations (solid curves) and transformed diagonal fit-ting of sensor-based sharpened V' observations (dotted curves).The sixth cumulative NFD histogram shows the average fittingperformance.

Finlayson et al.

It follows immediately that Qe is a fixed linear transformof the lighting matrix:

Qe Ae_4, (18)

where the columns of .A correspond to the surface weightvectors of the reference reflectances and are independentof the illuminant. Now, given any arbitrary responsevector pe, one can easily generate an illuminant-invariantdescriptor by premultiplying with [Qe]-l:

[Qe]l-pe = -q--[Aep]lAeu = -A' 0. (19)

The color-constancy performance of Brill's volumetrictheory is linked directly to the dimensionality of surfacereflectances. Real reflectance spectra are generally notthree dimensional (Maloney2 ' suggests that a basis set ofbetween three and six functions is required), leading toinaccuracies in the calculated descriptors.

Data-based sharpening, like volumetric theory, aimsto generate illuminant-invariant descriptors by applyinga linear transform. If we impose the extremely strongconstraint that all surface reflectances appear in eachimage, then data-based sharpening can be viewed as analgorithm for color constancy. As a color-constancy al-gorithm, data-based sharpening has a distinct advantageover volumetric theory in that it is optimal with respectto the least-squares criterion and consequently is guar-anteed to outperform the volumetric method. Unfortu-nately, this performance increase is gained at the expenseof the extremely strong requirement that all surface re-flectances appear in each image.

In practice we can weaken this constraint and assumethat there are k known reference patches per image,where k is small. Novak and Shafer2 2 develop a simi-lar theory called supervised color constancy, which isbased on the assumption that there are 24 known refer-ence patches in each image; however, unlike data-basedsharpening, their constancy transform is derived by ex-amination of the relationship between measured re-sponses and finite-dimensional models of reflectance andillumination. Certainly for the reference patches them-selves the data-based sharpening method will outper-form Novak's supervised color constancy since, for thesepatches, data-based sharpening finds the optimal least-squares transform. However, further study is requiredfor comparing overall color-constancy performance. Inthe limiting case, where k = 3, data-based sharpeningreduces to the volumetric theory.

Volumetric theory requires three reference patchesin order to recover the nine parameters of [e]-1 andthereby achieve color constancy. As shown by the per-formance tests of the preceding section, however, aftera fixed sharpening transformation of DMT models illu-mination change almost as well as does a nondiagonalmatrix. Since only three parameters instead of nineneed to be determined for specification of the diagonalmatrix when sharpened sensors are being used, only onereference patch is required instead of three for achievingcolor constancy. This follows because a single responsevector seen under a test illuminant Ee(A) can be mapped

Vol. 11, No. 5/May 1994/J. Opt Soc. Am. A 1559

to its canonical appearance by a single diagonal matrix:

pC = Depe,

Pic

(20)

(21)

If we choose our reference patch to be a white re-flectance, then, through sharpening, volumetric theory re-duces to Land's white-patch retinex.1 Similarly, Westand Brill2 consider white-patch normalization to be con-sistent with von Kries adaptation.

We performed a simulation, called transformed white-patch normalization, to evaluate the quality of color con-stancy obtainable with use of a single reference patch.For each illumination (CIE A, D48, D55, D65, D75, andD100) we

1. generated a matrix We of observations of Munsellpatches for VW sensors;

2. transformed observations to the (sensor-based)sharpened sensors, T'we;

3. calculated DeT'We, where Diie = l/pW (the re-ciprocal of the ith sharpened sensor's response to thewhite patch; the Munsell reflectance closest to the uni-form white, in the least-squares sense, was chosen as thewhite reference patch);

4. transformed white-patch normalized observations-back to VW sensors, 9`1DegWe.

Again D55 was the canonical light. Thus we evalu-ated constancy by calculating the NFD between thewhite-patch-corrected observations seen under D55 (thedescriptors) and the white-patch-corrected observationsunder each other illuminant. In Fig. 6 we contrast thecumulative NFD histograms for white-patch normaliza-tion (dotted curves) with those for the optimal fittingperformance (solid curves). White-patch normalizationyields very good constancy results that are generallycomparable with the optimal fitting performance.

6. SHARPENING RELATIVE TOMULTIPLE ILLUMINANTS

Data-based sharpening was introduced primarily to vali-date the idea of sensor-based sharpening and to ensurethat our particular choice of sharpening parameters ledto reasonable results. Figure 2 shows that the optimalsensors, as determined by means of data-based sharpen-ing for each illuminant, closely resemble one another andfurthermore that they resemble the unique set of sensor-based sharpened sensors as well. Although all the sen-sors are similar, the question remains regarding whetherthere might be an optimal sharpening transform for theentire illuminant set.

In Ref. 23 we derive conditions for perfect DMT colorconstancy, using sharpened sensors. Because the sharp-ening transform applies to a whole space of illuminants,it is in essence a type of global data-based sharpening.

The theoretical result is based on finite-dimensionalapproximations of surface reflectance and illumination,and what is shown is -that, if surface reflectances- arewell modeled by three basis functions and illuminantsby two basis functions, then there exists a set of new

1560 J. Opt. Soc. Am. A/Vol. 11, No. 5/May 1994

tions of Ac and N AC, as a result of Eq. (24). Conse-quently an observation vector obtained for any surfaceunder an illuminant EL(A) = aEC(A) + ,BE2(A) can be ex-pressed as a linear transform of its descriptor vector:

pe = [al + I3.M]ACo = [al + /3IpC, (25)

where I is the identity matrix. Calculating the eigen-vector decomposition of N,

(26)a M = aix ie f

and expressing the identity matrix in terms of C,

100

80

_60

40

20

100

80

; 602 0

40

20

100

80o

-60

40

0. it I 0 o -J . I o i

0 2 4 6 810 0 2 4 6 a 10 0 2 4 6 810%error %error %error

Fig. 6. Cumulative NFD histogram obtained with each testilluminant (CIE A, D48, D65, D75, and D100) for optimal fittingof VW observations (solid curves) and transformed white-patchnormalization of VW observations (dotted curves). The sixthcumulative NFD histogram shows average color constancyperformance.

sensors for which a DMT can yield perfect color con-stancy. These restrictions are quite strong; nonetheless,statistical studies have shown that a three-dimensionalbasis set provides a fair approximation to real surfacereflectance2 ' and that a two-dimensional basis set de-scribes daylight illumination 9 reasonably well. More-over, Marimont and Wandell2 4 have developed a methodfor deriving basis functions that is dependent on how re-flectance, illuminant, and sensors interact to form sen-sor responses [i.e., Eq. (2) above is at the heart of theirmethod]. A three-dimensional model of reflectance anda two-dimensional model of illumination is shown to pro-vide a good model of actual response vectors.

Given these dimensionality restrictions on reflectanceand illumination, cone response vectors of surfaces viewedunder a canonical illuminant, that is descriptors, can bewritten as

pC = A r, (22)

(27)

enables us to rewrite the relationship between an obser-vation and a descriptor, Eq. (25), as a diagonal transform:

T'pe = [a I + D ]Tpc. (28)

Writing pC in terms of pe leads directly to

tpc = [al +D] *Tpe. (29)

The import of Eq. (29) is that when the appropriateinitial sharpening transformation T is applied, a diago-nal transform supports perfect color constancy, subject ofcourse to the restrictions imposed on illumination andreflectance.

These restrictions compare favorably with thoseemployed by D'Zmura,2 5 who showed that, given three-dimensional reflectances and three-dimensional illumi-nants, one can obtain perfect color constancy, given twoimages of three color patches under two different illumi-nants and using a nondiagonal transform.

From the Munsell reflectance spectra and our six testilluminants, we used principal-component analysis to de-rive the basis vectors for reflectance and illumination.Using these vectors, we constructed lighting matrices andthen calculated the sharpening transform by means ofEq. (26). The formulas for the perfect sharpened sensorsare given in Eqs. (30)-(32), where they are contrastedwith the corresponding formulas obtained with respect tosensor-based and data-based sharpening. The symbolsR, G, and B denote the VW red, green, and blue conemechanisms, respectively, scaled to have unit norms, andthe superscripts p, s, and d denote perfect, sensor-based,and data-based sharpening, respectively:

RP = 2.44R - 1.93G + 0.110B,

where the superscript c denotes the canonical illuminant.Since illumination is two dimensional, there is necessarilya second illuminant E2 (A) linearly independent with EC(A)(together they form the span). Associated with this sec-ond illuminant is a second linearly independent lightingmatrix A2. It follows immediately that A2 is some lineartransform NM away from AC:

A2 = MAC,

NM =A2[AC]-

(23)

(24)

Since every illuminant is a linear combination of EC(A)and E2(A), lighting matrices in turn' are linear combina-

Rs = 2.46R - 1.97G + 0.075B,

Rd = 2.46R - 1.98G + 0.100B,

GP = 1.55G - 0.63R - 0.16B,

Gs = 1.58G - 0.66R - 0.12B,

Gd = 1.52G - 0.58R - 0.14B,

BP 1.OB - 0.13G + 0.081,

Bs = 1.OB - 0.14G + 0.09R,

Bd = 1.OB - 0.13G + 0.07R. (32)

CIE A D48

100

80

.- 60

N40

20

100

80

fl 60

aC 40

20

100

80

60

a 40

20

D65

0 2 4 6 8 10%error

Average

0 2 4 6 8 10%error

D75

0 2 4 6 8 10%error

D100

(30)

(31)

Finlayson et al.

= TgriTe

Vol. 11, No. 5/May 1994/J. Opt. Soc. Am. A 1561

It is reassuring that the perfect sharpened sensors arealmost identical to those derived through sensor-basedand data-based sharpening. Therefore, even though nei-ther the sensor-based sharpened sensors nor the data-based sharpened ones are optimized relative to a wholeset of illuminants, sharpening in all cases generates sen-sors that are similar to those that work perfectly for alarge, although restricted, class of illuminants. This the-oretical result provides strong support for the hypothe-sis, already confirmed in part by the consistency of thedata-based sharpening results, that a single sharpeningtransformation will work well for a reasonable range ofilluminants.

7. SPECTRAL SHARPENING AN])THE HUMAN VISUAL SYSTEM

If the human visual system employs a DMT for colorconstancy, our results show that we should expect it touse sharpened sensors, since doing so would optimize itsperformance. In this section we briefly examine some ofthe psychophysical evidence for sharpened sensitivities.

Sharpened sensitivities have been detected in both fieldand test spectral sensitivity experiments (for a reviewof these terms see Foster2 6). Sperling and Harwerth' 3

measured the test spectral sensitivities of human subjectsconditioned to a large white background and found, con-sistent with our findings for sharpened sensors, sharp-ened peaks at 530 and 610 nm, with no sharpening of theblue mechanism.

These authors propose that a linear combination ofthe cone responses accounts for the sharpening. Theyfound that the sharpened red sensor can be modeled asthe red cone minus a fraction of the green and that thesharpened green sensor can be modeled as the green coneminus a fraction of the red. This finding correspondswell with our theoretical results in that our sharpeningtransformations, Eqs. (30) and (31), basically involve redminus green and green minus red, with only a slightcontribution from the blue.

More recently, Foster' 2 observed that field and testspectral sensitivities show sharpened peaks when theyare derived in the presence of a small monochromaticauxiliary field coincident with the test field. Foster andSnelgar27 extended this work by performing a hybrid ex-periment with a white, spatially coincident auxiliary field,and sharpened sensitivities again were found. In bothcases these experimentally determined, sharpened sen-sitivities agree with our theoretical results. Foster andSnelgar28 verified, as did Sperling and Harwerth,13 thatthe sharpened sensitivities were a linear combination ofthe cone sensitivities.

Krastel and Braun2 9 measured spectral field sensitivi-ties under changing illumination when, as in the experi-ments by Sperling and Harwerth,' 3 a white conditioningfield is employed. Illumination color was changed byplacement of colored filters in front of the eye. Thesame test spectral-sensitivity curve results under botha reddish and a bluish illuminant, suggesting that theeye's sharpened mechanisms are unaffected by illumi-nation. More recently Kalloniatis and Harwerth14 mea-sured cone spectral sensitivities under white adaptingfields of different intensity and found the sharpened sensi-

tivities to be independent of the intensity of the adaptingfield.

Poirson and Wandell'6 developed techniques for mea-suring the spectral sensitivity of the eye with respect tothe task of color discrimination. For color discriminationamong briefly presented targets, the spectral sensitivitycurve has relatively sharp peaks at 530 and 610 nm.

Although the general correspondence between oursharpened sensors and the above psychophysical resultsdoes not imply that sharpening in humans exists for thepurpose of color constancy, at least the evidence thata linear combination of the cone responses is employedsomewhere in the visual system lends plausibility to theidea that sharpening might be used in human color-constancy processing. On the other hand, our findingthat sharpening could improve the performance of somecolor-constancy methods suggests an explanation for theexistence of spectral sharpening in humans.

8. CONCLUSION

Spectral sharpening generates sensors that improve theperformance of methods based on color constancy theo-ries (e.g., von Kries adaptation, Land's retinex) that use

0.4

D D 0.3N>0 0.2E c° u 0.1

0.0

-0.1

0.4

. > 0.3N>Ad 0.2E c8 XD 0.1

0.0

-0.1

0.4

, >, 0.3

0.2Ec°e 0.1

0.0

400 450 500 550Wavelength

400 450 500 550Wavelength

600 650

600 650

400 450 500 550 600 650Wavelength

Fig. 7. Solid curves show results for the Kodak Wratten filters#66, #52, and #38; dotted curves show the results of sensor-basedsharpening; dashed curves show the mean of the data-basedsharpened sensors obtained for the five test illuminants (CIEA, D48, D65, D75, and D100). Top, long-wavelength mecha-nism; middle, medium-wavelength mechanism; bottom, short-wavelength mechanism.

Finlayson et al.

1562 J. Opt. Soc.Am. A/Vol. 11, No. 5/May 1994

diagonal-matrix transformations. Data-based sharpen-ing finds sensors that are optimal with respect to a givenset of surface reflectances and illuminants. Sensor-based sharpening finds the most-narrow-band sensorsthat can be created as a linear combination of a givenset of sensors. Finally, for restricted classes of illumi-nants and reflectance (they are constrained to be two andthree dimensional, respectively) we have shown that thereexists a sharpening transform with respect to which a di-agonal matrix will support perfect color constancy. Thesharpening transform derived by means of this analysis isin close agreement with the sensor-based sharpening anddata-based sharpening transforms. In all three casessharpened sensors substantially improve the accuracywith which a DMT can model changes in illumination.The sensor-based and the data-based sharpening tech-niques are quite general and can be applied to visualsystems that have more than three sensors.

As with the cone sensitivities, sharpening of a colorcamera's sensitivities can also have a significant effect.Use of overlapping, broadband filters such as Wratten#66, #52, and #38 (Ref. 30) could be advantageous, sincefrom an exposure standpoint they filter out less light, buttheir use could be disadvantageous from a color-constancystandpoint. Sharpening such filters, as shown in Fig. 7,can provide a good compromise among the competingrequirements.

Spectral sharpening is not, in itself, a theory of colorconstancy, in that it makes no statement about how tochoose the coefficients of the diagonal matrix. Instead,we propose sharpening as a mechanism for improving thetheoretical performance of DMT algorithms of color con-stancy regardless of how any particular algorithm mightcalculate the diagonal-matrix coefficients to be used inadjustment for an illumination change.

Since the performance of DMT algorithms improvessignificantly when sharpened sensitivities are employed,and, furthermore, since it then compares favorably withthe performance of the best possible nondiagonal trans-form, our results suggest that if a linear transform is acentral mechanism of human color constancy, then afteran appropriate, fixed sharpening transformation of thesensors there is little to be gained through the use of any-thing more general than a DMT.

In general, our results lend support to DMT-basedtheories of color constancy. In addition, since spectralsharpening aids color constancy, we have advanced thehypothesis that sharpening might provide an explanationfor the psychophysical finding of sharpening in the hu-man visual system.

ACKNOWLEDGMENTS

This research has been supported by the Simon FraserUniversity Center for Systems Science and the NaturalSciences and Engineering Research Council of Canadaunder grant 4322.

REFERENCES

1. J. Beck, Surface Color Perception (Cornell U. Press, Ithaca,N.Y., 1972).

2. G. West and M. H. Brill, "Necessary and sufficient conditionsfor von Kries chromatic adaption to give colour constancy,"J. Math. Biol. 15, 249-258 (1982).

3. E. H. Land and J. J. McCann, "Lightness and retinex the-ory," J. Opt. Soc. Am. 61, 1-11 (1971).

4. B. K. P. Horn, "Determining lightness from an image," Coin-put. Vision Graphics Image Process. 3, 277-299 (1974).

5. A. Blake, "Boundary conditions for lightness computation inMondrian world," Comput. Vision Graphics Image Process.32, 314-327 (1985).

6. D. Forsyth, "A novel algorithm for color constancy," Int. J.Comput. Vision 5, 5-36 (1990).

7. M. D'Zmura and P. Lennie, "Mechanisms of color constancy,"J. Opt. Soc. Am. A 3, 1662-1672 (1986).

8. B. V. Funt and M. S. Drew, "Color constancy computationin near-Mondrian scenes using a finite dimensional linearmodel," in Computer Vision and Pattern Recognition Pro-ceedings (Institute of Electrical and Electronics Engineers,New York, 1988), pp. 544-549.

9. M. H. Brill, "Computer-simulated object-color recognizer,"Tech. Rep. 122 (MIT Research Laboratory of Electronics,Cambridge, Mass., 1980).

10. G. Wyszecki and W. S. Stiles, Color Science: Concepts andMethods, Quantitative Data and Formulas, 2nd ed. (Wiley,New York, 1982).

11. E. H. Land, "The retinex theory of color vision," Sci. Am.237, 108-129 (1977).

12. D. H. Foster, "Changes in field spectral sensitivities ofred-, green- and blue-sensitive colour mechanisms obtainedon small background fields," Vision Res. 21, 1433-1455(1981).

13. H. G. Sperling and R. S. Harwerth, "Red-green cone in-teractions in the increment-threshold spectral sensitivity ofprimates," Science 172, 180-184 (1971).

14. M. Kalloniatis and R. S. Harwerth, "Spectral sensitivityand adaptation characteristics of cone mechanisms underwhite-light adaptation," J. Opt. Soc. Am. A 7, 1912-1928(1990).

15. D. C. Hood and M. A. Finkelstein, "A case for the revision oftextbook models of colour vision," in Colour Vision: Physi-ology and Psychophysics, J. D. Mollon and L. T. Sharpe, eds.(Academic, New York, 1983), pp. 385-398.

16. A. B. Poirson and B. A. Wandell, "Task-dependent colordiscrimination," J. Opt. Soc. Am. A 7, 776-782 (1990).

17. J. H. Wilkinson, Algebraic Eigenvalue Problem, Monographson Numerical Analysis (Oxford U. Press, Oxford, 1965).

18. J. K. Bowmaker and H. J. A. Dartnell, "Visual pigmentsof rods and cones in the human retina," J. Physiol. 298,501-511 (1980).

19. D. B. Judd, D. L. MacAdam, and G. Wyszecki, "Spectraldistribution of typical daylight as a function of correlatedcolor temperature," J. Opt. Soc. Am. 54, 1031-1040 (1964).

20. J. Cohen, "Dependency of the spectral reflectance curves ofthe Munsell color chips," Psychon. Sci. 1, 369-370 (1964).

21. L. T. Maloney, "Evaluation of linear models of surface spec-tral reflectance with small numbers of parameters," J. Opt.Soc. Am. A 3, 1673-1683 (1986).

22. C. L. Novak and S. A. Shafer, "Supervised color con-stancy using a color chart," Tech. Rep. CMU-CS-90-140(Carnegie-Mellon University School of Computer Science,Pittsburgh, Pa., 1990).

23. G. D. Finlayson, M. S. Drew, and B. V. Funt, "Enhancingvon Kries adaptation via sensor transformations," in Hu-man Vision, Visual Processing, and Digital Display IV, J. P.Allebach and Bernice E. Roqowitz, eds., Proc. Soc. Photo-Opt. Instrum. Eng. 1913, 473-484 (1993).

24. D. H. Marimont and B. A. Wandell, "Linear models of surfaceand illuminant spectra," J. Opt. Soc. Am. A 9, 1905-1913(1992).

25. M. D'Zmura, "Color constancy: surface color from changingillumination," J. Opt. Soc. Am. A 9, 490-493 (1992).

26. D. H. Foster, "Colour vision," Contemp. Phys. 25, 477-497(1984).

27. D. H. Foster and R. S. Snelgar, "Test and field spectralsensitivities of colour mechanisms obtained on small whitebackgrounds: action of unitary opponent-colour processes?"Vision Res. 23, 787-797 (1983).

Finlayson et al.

Finlayson et al. Vol. 11, No. 5/May 1994/J. Opt. Soc. Am. A 1563

28. D. H. Foster and R. S. Snelgar, "Initial analysis of opponent-colour interactions revealed in sharpened field sensitivi-ties," in Colour Vision: Physiology and Psychophysics, J. D.Mollon and L. T. Sharpe, eds. (Academic, New York, 1983),pp. 303-312.

29. W. Jaeger, H. Krastel, and S. Braun, "An increment-

threshold evaluation of mechanisms underlying colour con-stancy," in Colour Vision: Physiology and Psychophysics,J. D. Mollon and L. T. Sharpe, eds. (Academic, New York,1983), pp. 545-552.

30. Kodak Filters: for Scientific and Technical Uses, 2nd ed.(Eastman Kodak, Rochester, N.Y., 1981).