Color Calibrated High Dynamic RangeImaging with ICC Profiles

Michael Goesele�, Wolfgang Heidrich

�, Hans-PeterSeidel

��Max-Planck-Institut für Inf ormatik

�The Universityof British Columbia

Saarbrücken,Germany Vancouver, Canada

Abstract

High dynamicrange(HDR) imaginghasbecomea pow-erful tool in computergraphics,and is being appliedtoscenarioslike simulationof differentfilm responses,mo-tion blur, andimage-basedillumination. TheHDR imagesfor theseapplicationsaretypicallygeneratedbycombiningtheinformationfrom multiple photographstakenat differ-entexposuresettings.

Unfortunately, thecolorcalibrationof theseimageshassofarbeenlimited to verysimplisticapproachessuchasasimplewhitebalancealgorithm.Moresophisticatedmeth-odsusedfor device-independentcolor representationsarenoteasilyapplicablebecausethey inherentlyassumealim-ited dynamicrange. In this paper, we introducea novelapproachfor constructingHDR imagesdirectly from lowdynamicrangeimagesthat werecalibratedusingan ICCinputprofile.

1. Intr oduction

Normalphotographshavenon-linearfilm responseandarethereforenot directly usefulas input datafor mostcom-putergraphicsalgorithms,as theserequirelinearizedra-diancevalues. For example, in order to illuminate syn-theticobjectsusingphotographsof arealenvironment(seee.g.[3]), wehave to integrateover theincidentradianceatevery point on the syntheticobject. If this informationisto betakenfrom animage,thenthelimited dynamicrangeand the non-linearfilm responseof the imagehasto betaken into account,or effects suchas saturationof verybright imagepartswill show upasartifactsin thefinal ren-dering.Thesameis truefor effectslikemotionblur [4], orfor image-basedmeasurementsof reflectionproperties[6].

High dynamicrange(HDR) imagingis apowerful toolfor avoiding theseproblems.Using methodssummarizedin moredetailin thenext section,differentlyexposedpho-tographsof the samescenearefirst usedto estimatethenon-linearfilm responsecurve, which can then be usedto generateimageswith a high dynamicrangeanda lin-earizedfilm responsethatdonotsaturatein thebrightparts.

Unfortunately, the stateof the art of color calibrationin HDR imagingusesaverysimplisticapproach,basedona simplewhite balancebetweenthe red, green,andbluechannelsof theimage.Theuseof moresophisticatedtech-niquessuchasICC profileswasso far not possiblesincetheseprofiles are inherentlytailored towardsthe caseofimageswith limited dynamicrange.

In this paperwe proposea new methodfor construct-ing HDR imagesdirectly from low dynamicrangeimagescalibratedwith an ICC input profile. This will not onlyallow usto havemeaningfulcolorsin image-basedillumi-nationandmeasurement,but alsoto simulatetheeffectsofdifferentcamera/lens/filmcombinationswith realisticcol-ors.

Theremainderof thispaperis organizedasfollows: inSection2 we briefly summarizethework relatedto HDRimaging. In Section3 we analyzethe theoreticalfoun-dationsof HDR imaging in XYZ color space. We thendescribehow to useICC profilesfor HDR imaging(Sec-tion 4), andvalidateour assumptionsaboutthe resultingcolor spacein Section5. In Section6 we summarizeouralgorithmbeforeweconcludeby presentingresultsfor theproposedmethod(Section7).

2. PreviousWork

Highdynamicrangeimagingwasfirst introducedbyWyck-off [13] in the 1960swho inventedan analogfilm withseveralemulsionsof differentsensitivity levels. His falsecolorfilm hadanextendeddynamicrangeof about

� � �.

More recently, severalauthorsproposedmethodto ex-tend the dynamicrangeof digital imagesby combiningmultiple imagesof thesamescenethatdiffer only in expo-suretime. Madden[7] assumeslinearresponseof a CCDimagerandselectsfor eachpixel an intensityvaluefromthe brightestnon-saturatedimage. Mann and Picard[8]constructthe responsecurve of the cameraby observingthe intensityvaluesof singlepixels in anexposureseries.A more robust methodto determinethe responsecurve,which selectsa small numberof pixels from the images

and� performsanoptimizationwith asmoothnessconstraint,wasintroducedby DebevecandMalik [4]. They alsoin-troduceda white balancestep in order to determinetherelationshipbetweenthe channelsof a RGB color image.Robertsonet al. [11] improvedon this by optimizingoverall pixels in all imagesand using a different weightingfunction.

In addition,severalmethodshavebeenproposedto al-ter thedesignof a digital camerain orderto extendits dy-namic range[9, 10]. Notably, Moriwaki [9] focusesonthe correctcaptureof the ratios of the RGB color chan-nelsin thepresenceof highcontrastfor colorsegmentationalgorithms. Furthermore,modernCMOS imagershave ahigherdynamicrangethanthemorecommonlyusedCCDimagers.

Theabovemethodsfor constructingaHDRimagefroman exposureseriesare eitherassuminga linear responseof the imagingsystemor determinetheresponsecurve ina separatestep,treatingmultiple color channelsindepen-dently. If the relationshipbetweenthe color channelsisconsideredat all, it is only donefor thefinal HDR image.

In contrastto that we assumethat colors are recon-structedcorrectly in eachinput imageif the pixel underconsiderationis not saturatedor underexposed. We usean ICC input profile [5, 12] describingthe propertiesofour imagingsystemto captureaccuratecolorsandto con-vert theinput imagesinto theCIEXYZ profile connectionspace(PCS).We assumeandconfirmwith ameasurementthatthetransformationinto thePCSleadsto a linearXYZspace1, allowing us to calculateHDR valuesby a simpleaveragingoperation.

3. High Dynamic RangeImaging in XYZColor Space

TheCIEXYZ colorspaceis definedrelativeto thespectralpowerdistributions ���� � , �� � � , �� � � of theCIE 1931stan-dardcolorimetricobserver [2]. The � , � , � tristimulusvaluesaredefinedfor secondarylight sources(reflectingor transmittingobjects)as

���������� � � ���� � � ���������� � � �� � � � ���������� � � ���� � � ���! " "#$&% '

� ( )* '� ( + �

where� � � is thespectraldistributionof thelight and, � �is the relative spectralpower distribution of the illumi-

1WeuseCIEXYZ to denotetheCIE XYZ colorspaceasdefinedin [2]and XYZ for any other, not necessarilywell definedcolor spacewithsimilarproperties.

nant. Note that the tristimulusvaluesare independentofthepowerof theilluminantanddependlinearlyon � � � .

If we areableto acquirecolor correctedphotographsin CIEXYZ space(e.g.usingICC profiles,seeSection4),then this linearity relationshipmeansthat every pixel -in the color correctedimagewill get the following colorvalue:

�/.0��1 243 �� ��. � � ���� � � 65��.7��1 243 �����. � � ���� � � 65�8.9��1 243 ���� . � � ���� � � 65

where2 is theexposuretime and 1 is anunknown scalingfactor. That is, theCIEXYZ color valuesareproportionalto the exposuretime andthe sceneluminancedueto thelinearityof theCIEXYZ colorspace.

3.1. ExposureSeries

This propertycanbe usedto scaleandaveragethe pixelvaluesfor a set of images : ; < = of the samescenewithdifferentexposuretimes2�< in orderto createaHDR imageof thesceneaccordingto thefollowing formulas:

��.��42 >? < ��. @ < 2BA <DC � �/. @ < 5 ��. @ < 5 �8. @ < ? < C � �/. @ < 5 ��. @ < 5 �8. @ <

��.��42 >? < ��. @ < 2EA <DC � ��. @ < 5 ��. @ < 5 �8. @ < ? < C � �/. @ < 5 ��. @ < 5 �8. @ <

�8.6�42 >? < ��. @ < 2BA <DC � �/. @ < 5 ��. @ < 5 �8. @ < ? < C � �/. @ < 5 ��. @ < 5 �8. @ <

(1)

26> is a scalingfactorto determinethe exposurelevelof the HDR image. As the images : ; < = have a limiteddynamic rangethe weighting function C is usedto ig-norepixel valueswhicharesaturatedor underexposedandthereforeinvalid. In additionit canbeusedto emphasizethecontribution of pixel valuesin themediumpartof thedynamicrange,which areassumedto bemoreaccurate.

The resultingHDR pixel valuesarenow proportionalto the luminanceof the correspondingpart of the scene.An absolutecalibrationcanbeachievedby capturingatar-getwith known luminance.Alternatively theexposurefor-mula given in [1] can be usedto obtain an approximatecalibration.

4. Using ICC Profiles

In ordertocalibratethecolorof ourindividualphotographs,we use ICC profiles that describethe color characteris-tics of devicessuchasdigital cameras,monitors,or print-ers [5, 12]. A profile can be usedto convert input data

fromF the color spaceof the input device into a commonprofile connectionspace(PCS)which is either CIELABor CIEXYZ with a definedwhite point (illuminant D50).CIELAB valuescanbe convertedinto CIEXYZ andviceversa. An outputor monitor profile converts imagedatafrom the PCSto the color spaceof the outputdevice be-fore it is printedor displayed.Theseoperationsareusuallyperformedby a colormanagementsystem.

A suitableinput profile can be usedto convert inputdatafrom a digital camerainto the linear CIEXYZ colorspaceof thePCSandcorrespondsthereforeto theresponsecurvethatis traditionallyusedin HDR imaging.However,in contrastto the responsecurve it not only linearizestheinput databut converts it into the well definedCIEXYZcolorspace.

Many algorithmswhich useHDR imagesshouldbeableto directly usea XYZ HDR imageinsteadof a RGBHDR imageasthetwo colorspacesarerelated,othershaveto beadaptedto thenew color space.If all operationsthatareappliedto an XYZ HDR imagepreserve the linearityof theCIEXYZ spaceit is possibleto applyanICC outputprofileto theHDR imagein orderto haveacompletecolormanagementchain.

This approachdependson the assumptionthat the in-put profile converts the input imagesinto a linear XYZspace. Dependingon the propertiesof the input devicethis might be difficult anda tradeoff betweengoodcolorrepresentationandlinearity hasto bemadeduringthecre-ationof the input profile. In addition,aninput profile canbegeneratedfor a preferredreproduction,e.g. to empha-sizeshadow regions,sothattheresultingprofilemightper-form a desiredbut not necessarilyfaithful conversionintoCIEXYZ spacewhich changesthe actualcolor. Due tothesereasonit is necessaryto checkhow lineartheresult-ing XYZ spaceis (seethe next section). The impact oftheremainingnonlinearitieson theresultingimagesis an-alyzedin Section7.

5. Linearity Measurements

To test our approachwe acquiredan exposureseriesofan IT8 target with a KodakDCS 560 digital cameraandHMI metalhalidelighting. The seriesconsistsof 13 im-ageswith exposuretimes between G " H and I " " H in G f-stop increments.A customICC input profile wasgener-atedfor a correctlyexposedimagefrom this seriesusingthe Linotype CPSScanOpenICC DCam software. Theimagesof theexposureserieswerethenconvertedinto 16bit perchannelXYZ color spacewith our own implemen-tation of a color managementsystemusingthe generatedprofile. Pixelsfor whichat leastonechannelwassaturatedor for whichall channelswereunderexposed,weremarkedinvalid andexcludedfrom furtherprocessing.

We implementedtheapproachof Robertsonet al. [11]for 16 bit imagesandcalculatedthe responsecurvesforall color channels.Figure1 shows a plot of thecomputedresponsecurvesanda linear responsecurve for compari-son. Note that no smoothingoperationis appliedduringthecomputation. Therecoveredresponsecurvesconfirm

Figure 1: Logarithmicplot of the responsecurve of the X, Y,andZ channelsfor theIT8 targetexposureseries.Thehorizontalaxisshows 16 bit digital units. For comparisona plot of a linearresponsecurve is included.

Figure 2: Plot of thelinearity of theresponsecurve for theX, Y,andZ channels(samedataasin Figure1). The horizontalaxisshows 16 bit digital units,theverticalaxisshows therelative de-viation from the linear responsecurve. A smoothingoperationwasappliedto thecurve in orderto emphasizetheglobalbehav-ior. For comparisona plot of a linearresponsecurve is included.

that the resultingcolor spaceis approximatelylinear (seeFigure2. But they alsoshow severalproblems:J Theresponsecurvesarenotsmoothandshow abias

in the brighter regions (especiallyin the X and Ychannel).J In thevery darkregions,thesignalis stronglynon-linear and noisy. This can be due to several rea-sons,especiallythe limited signal-to-noiseratio of

thecamera,quantizationartifactsin theintermediateimagerepresentations,andproblemswith thegener-atedprofile.J In the very bright regionsthe responsecurve hasahighvariance.

Theweightingfunction C shouldtake theseproblemsintoaccountandsuppressvaluesat both endsof the dynamicrange.Furtherexperimentshaveshown, thatthelocalvari-ationsthroughoutthedynamicrangedependon thescenecontent. Including them into the weighting function isthereforenot appropriate.

The largestproblemis the biasin the X andY chan-nel. This couldbedealtwith by manuallylinearizingtheresponsecurve which directly compromisesthe integrityof the profile connectionspace.Alternatively, combiningseveralimagescanalsoreducetheerrorwithoutexplicitlychangingthePCS.This approachhastheadvantagethatitis unbiased,i.e. it leadsto a correctresultif theexposureseriesconsistsof a singleimage.

Taking theseconsiderationsinto account,we decidedto leavetheresponsecurveunmodifiedandto useasimpleweightingfunction C with a linear rampat bothendsanda constantvalueinbetween.Dueto thedifferentbehaviorof the threechannels,the widths of the rampsarediffer-ent for eachchannel. As shown in Section7 this choiceworkedwell for our setup.However, dependingon thein-dividual setupused,adifferentweightingfunctionmaybeappropriate.

6. Algorithm

Given a suitableinput profile, the generationof a HDRimagefrom an exposureseriesis not difficult. First theimageshave to beconvertedinto CIEXYZ colorspaceus-ing theprofile. Pixelsfor which at leastonecolor channelis saturatedor for which all color channelsareunderex-posedin theoriginal imageshouldbemarkedinvalid andexcludedfrom theHDR calculation.

Thecontributionsof theimagesto thefinal HDRimagearesummedup accordingto Equation1 usingtheweight-ing function C . TheresultingHDR imagecanthenbear-bitrarily processedaslongasthelinearity is preserved.Fi-nally, the imagehasto be convertedbackto the PCSbycorrectingall pixelsfor which thevalueof � is K � � � . Asthesepixelsare“overexposed”they cane.g. besetto thetristimulus valuesof the light source. The resultingim-agecanthenbe usedin the color managementworkflowwithout limitations.

7. Resultsand Conclusion

Wedescribedin thispaperhow theICC profilemechanismcanbeusedto acquireHDR imagesin CIEXYZ spacewith

calibratedcolors.As long asfurtherprocessingsteps(e.g.image-basedrenderingalgorithms)preserve the linearityof the CIEXYZ color space,it is possibleto returnto thePCSandapplysuitableoutputprofilesto theimageslead-ing to anintegrationof HDR imagingwith traditionalcolormanagementsystems.

Figure3 shows a normal image,which wasobtainedwith our methodby generatinga HDR imagefrom theex-posureseriesdescribedin Section5. The imagewasnor-malized(i.e., the intensity wasscaledso that the largestpixel value equals100) in CIEXYZ space,convertedtoCIELAB andfinally to sRGB.For comparison,anoriginalinput imagewas also normalizedin CIEXYZ spaceandconvertedto LAB. TheEuclideandistanceL�M�NO P betweenthe two imageshasbeencalculatedfor all pixels and isshown togetherwith a histogramof thecolor patchpartofthe target in Figures4 and5. Thematchbetweenthe im-agesis verygood.Theresultsbecomeonly slightly worse,if lessimages(e.g.,a seriesseparatedby a full f-stop)areused.

Figure 3: Imagegeneratedfrom theHDR imageof theIT8 target.

Figure 4: Differencebetweenan appropriatelyscaledoriginalinputimageandFigure3calculatedusingthe QEREST U metric.Mostcolorsarereproducedverywell, thelargesterroris in thepatchesJ3andJ4with QEREST U6VXW Y .

Oncethelinearityof theXYZ spaceis established,theproposedmethodis easyto implementandrequiresonly

Figure 5: Histogramof QERZST U in thecolor patchpartof the IT8target(seeFigure4). Thehorizontalaxisshows QER ST U , theverti-cal axisshows thecorrespondingfractionof pixels.

a small amountof memoryas the imagescan processedsequentially. Thequality of theresultsdependsmainly onthe input profile andthe precisionof the input imagesisnot limited to a certainnumberof bits perchannel.

8. Acknowledgments

The authorswould like to thank the reviewers for theirvaluablecommentsandsuggestions,ThomasNeumannforimplementingpartsof thecolor managementsystem,andHartmutSchirmacherfor proofreadingthefinal paper.

References[1] AnselAdams. The Negative. Little, Brown andCompany,

6thpaperbackprintingedition,1999.

[2] Colorimetry. PublicationCIE No. 15.2,1986.

[3] Paul Debevec. RenderingSynthetic Objects Into RealScenes:Bridging Traditional and Image-BasedGraphicsWith Global Illumination andHigh DynamicRangePho-tography. In Computer Graphics (SIGGRAPH ’98 Proceed-ings), pages189–198,July 1998.

[4] Paul Debevec and JitendraMalik. Recovering High Dy-namic RangeRadianceMaps from Photographs.In Pro-ceedings of SIGGRAPH 97, pages369–378,August1997.

[5] SpecificationICC.1:1998-09,File Format for Color Pro-files. availablefrom http://www.color.org, 1998.

[6] Hendrik P.A. Lensch,JanKautz, Michael Goesele,Wolf-gangHeidrich, andHans-PeterSeidel. Image-BasedRe-constructionof Spatially Varying Materials. In StevenGortler and Karol Myszkowski, editors,Rendering Tech-niques 2001, Proceedings of the 12th Eurographics Work-shop on Rendering, London,GreatBritain, 2001.Springer.

[7] Brian C. Madden. ExtendedIntensity RangeImaging.Technicalreport,Universityof Pennsylvania,GRASPLab-oratory, 1993.

[8] Steve MannandRosalindW. Picard. On being‘undigital’with digital cameras:ExtendingDynamicRangeby Com-biningDifferentlyExposedPictures.In IS&T’s 48th AnnualConference, pages422–428,1995.

[9] Kousuke Moriwaki. Adaptive Exposure Image InputStsyemfor ObtainingHigh-QualityColorInformation.Sys-tems and Computers in Japan, 25(8):51–60,1994.

[10] ShreeK. Nayar and Tomoo Mitsunaga. High DynamicRangeImaging: Spatially Varying Pixel Exposures. InIEEE Conference on Computer Vision and Pattern Recog-nition (CVPR-00), pages472–479,June2000.

[11] Mark A. Robertson,SeanBorman,andRobertL. Steven-son. DynamicRangeImprovementThroughMultiple Ex-posures. In Proceedings of the 1999 International Con-ference on Image Processing (ICIP-99), pages159–163.IEEE,oct.1999.

[12] Dawn Wallner. Building ICC Profiles – theMechanics and Engineering. available athttp://www.color.org/iccprofiles.html,2000.

[13] CharlesW. Wyckoff andStanA. Feigenbaum.An Exper-imentalExtendedExposureResponseFilm. SPIE, 1:117–125,1963.

9. Biographies

MichaelGoeseleis aPhDstudentin thecomputergraphicsgroupof the Max-Planck-Institutfür Informatik in Saar-brücken, Germany. He studiedcomputerscienceat theUniversityof Ulm, Germany, andattheUniversityof NorthCarolinaat ChapelHill, USA, andreceivedhis Masterincomputersciencefrom theUniversityof Ulm in 1999.Hisinterestsincludecolor science,acquisitiontechniquesforcomputergraphics,andimage-basedrendering.

WolfgangHeidrichisanAssistantProfessorattheUni-versityof British Columbia,Canada.Previously hewasaResearchAssociateat thecomputergraphicsgroupof theMax-Planck-Institutfür Informatik in Saarbrücken, Ger-many, wherehechairedtheactivities on image-basedandhardware-acceleratedrendering. He received a PhD incomputersciencefrom the University of Erlangen,Ger-many, in 1999,a Masterin mathematicsfrom theUniver-sity of Waterloo,Canada,in 1996,anda Diplomain com-putersciencefromtheUniversityof Erlangen,Germany, in1995.His researchinterestsincludehardware-acceleratedandimage-basedrendering,globalillumination,andinter-activecomputergraphics.

Hans-PeterSeidel is the scientific director and chairof thecomputergraphicsgroupat theMax-Planck-Institutfür Informatik anda professorof computerscienceat theUniversity of Saarbrücken, Germany. He is an adjunctprofessorof computerscienceat the University of Erlan-gen,Germany, andat theUniversityof Waterloo,Canada.Seidel’s researchinterestsincludecomputergraphics,ge-ometricmodeling,high-quality rendering,and3D imageanalysisandsynthesis.He haspublishedsome150 tech-nical papersin the field andhaslecturedwidely on thesetopics.