# Digital canvas removal in paintings

Post on 12-Sep-2016

213 views

TRANSCRIPT

Fast communication

s$

lis a

l (VUB

enlaan

Digital painting analysis

Canvas removal

Denoising

cture

nd d

mod

noise

ima

Henc

the

extension with low-complexity to obtain a generic digital canvas removal lter.

& 2011 Elsevier B.V. All rights reserved.

nd thelablellectioart obled toysis sp

ostlyed in

canvas in digital images of the painting.

Contents lists available at SciVerse ScienceDirect

journal homepage: www.els

Signal Pro

Project G021311N and the Postdoctoral fellowship of Peter Schelkens).n Corresponding author at: Department of Electronics and Informatics

Signal Processing 92 (2012) 116611710165-1684/$ - see front matter & 2011 Elsevier B.V. All rights reserved.

doi:10.1016/j.sigpro.2011.11.012The problem led us to propose a new generic way ofperiodic noise ltering for removing canvas contaminationin paintings to aid objective inspection of the state ofa painting. Several solutions for periodic (or quasi periodic)

(ETRO), Vrije Universiteit Brussel (VUB), Pleinlaan 2, B-1050 Brussels,

Belgium. Tel.: 32 26291671.E-mail address: bcorneli@etro.vub.ac.be (B. Cornelis).# Member of EURASIP.the creases of the weave and was xed by varnish appliedduring conservation treatment and not by Gauguin him-self. This all adds to the prominent appearance of the

$ This research was supported by the Fund for Scientic Research

Flanders (FWO) (PhD fellowship of Bruno Cornelis, Project G.0206.08,process. Therefore, the digital removal of canvas can aid Moreover, due to the rough texture of the support (mapparent in raking light images [5]), dirt accumulatematicians and art historians, as can be witnessed by ourclose collaboration with the Royal Museums of Fine Arts ofBelgium in Brussels [1]. In this paper, we focus on thevisual enhancement of high resolution images of paintingson canvas. This type of support becomes visually promi-nent in recordings, disturbing the digital art analysis

where the emerging grid-like structure caused by theunderlying support is quite disturbing. The poor state ofthe painting can be explained by the circumstances inwhich Gauguin painted and by his preference for coarsefabric (typically 48 threads per cm, sackcloth in case ofthe Bambridge painting) during his Tahiti period [4].Wiener lter

1. Introduction

Advances in image acquisition aimaging modalities currently avaimuseums to start digitizing their coarchiving but also for analyzing thedigital image counterpart. This hasary interaction between image analwide range ofhave triggeredns, not only forject through itscross-disciplin-ecialists, math-

art conservators to better judge the state of a painting orto more accurately determine its history. Furthermore, itcan enhance digital inspection of brushstrokes which isused to determine an artists style, date the painting oreven help in its authentication [2,3].

The question was raised in particular for the paintingPortrait of Suzanne Bambridge (1891) by Paul Gauguin(18481903), which is in urgent need of restoration andPeriodic noiseDigital canvas removal in painting

Bruno Cornelis a,b,n, Ann Dooms a,b, Jan Cornea Department of Electronics and Informatics (ETRO), Vrije Universiteit Brusseb Interdisciplinary Institute for Broadband Technology (IBBT), Gaston Cromm

a r t i c l e i n f o

Article history:

Received 20 July 2011

Received in revised form

11 November 2011

Accepted 12 November 2011Available online 19 November 2011

Keywords:

a b s t r a c t

The periodic stru

quite prominent a

construct a new

considered as a

resolution of the

denoising lters.

lter is applied to, Peter Schelkens a,b,#

), Pleinlaan 2, B-1050 Brussels, Belgium

8 (box 102), B-9050 Ghent, Belgium

of the underlying support of paintings on canvas can become

isturbing in high resolution digital recordings. In this paper, we

el and method for the digital removal of canvas which is

component superimposed on the painting artwork. The high

ges prohibits the efcient application of existing adaptive

e, a two-step approach is proposed. First a (smoothing) Wiener

complete image. The second step consists of a spatially adaptive

evier.com/locate/sigpro

cessing

process (or rst order Markov random eld):

xi,j r1xi1,jr2xi,j1r1;2xi1,j1wi,j,

where wi,j is a two-dimensional i.i.d. Gaussian sequenceand r1 and r2 are one step correlations. Whenr1;2 r1r2, the model is separable and has an exponen-tially decreasing non-isotropic autocorrelation function:

Rxxt1,t2 s2xr9t191 r

9t292 , 0rr1,r2o1,

where r1 and r2 are typically around 0.95 for naturalimages [9] and where s2x is the variance of pixel values xi,j.

In the second case, a two-dimensional non-causalmodelis used to represent the pixel values:

xi,j axi1,jxi,j1xi1,jxi,j1wi,j, 9a9o1=4:Within this model, the autocorrelation behaves almost

isotropically at small displacements [9] and can beapproximated by an isotropic (non-separable) function:

Rxxt1,t2 s2xrt21 t22

pwhere r e

a

p:

Based upon the minimal error obtained after leastsquares tting (LSF) on Ryy , the rst or second model isinitially selected to represent the autocorrelation Rxx ofthe uncompromised painting (patch) x.

2.1.2. Noise model

Obviously, a canvas contaminated patch contains astrong horizontal and vertical periodic component which

Fig. 1. Flowchart of the proposed digital canvas removal method: blockA represents the modeling part, block B represents the two-step

processing applied on the contaminated image y, yielding x.

B. Cornelis et al. / Signal Processing 92 (2012) 11661171 1167noise removal have been proposed in the literature suchas frequency domain median lters [6,7], notch lters, aGaussian notch reject lter (GNF) and its improvement,the Windowed GNF [8]. However, none of them arereadily applicable due to the highly specic nature ofthe canvas contamination.

We start by constructing our lter to specically targetthe removal of (additive) independent periodic noisecaused by the canvas without affecting the ner struc-tures of the painting (e.g. brushstrokes, craquelure, pre-vious restorations or details). To cover the often irregularnature of canvas in paintings, we extend the lter in a fastsecond step by introducing spatial adaptivity withoutdramatic increase in computational complexity. Ourexperimental results conrm the all-round applicabilityof the resulting lter on two datasets of high-resolutionmultispectral images of paintings on canvas.

2. Constructing the lter

We will tackle the removal of canvas from the digitalimage of a painting as a denoising problem. The uncompro-mised image x will be retrieved from the contaminated imagey xn, where the underlying support is represented by anadditive (periodic) noise component n, which is locallyindependent of the painting signal x. We employ a (smooth-ing) Wiener lter in an unconventional way, namely toremove the additive periodic contaminating component n,superimposed on the desired image x. The Wiener lter isoptimal in the sense of minimum mean-square error(MMSE), and dened in the frequency domain as

HW u,v Sxxu,v

Sxxu,vSnnu,v, 1

where Sxxu,v is the power spectral density (PSD) of theuncontaminated image and Snnu,v is the PSD of the noise.

Note that the PSD of a signal exists if and only if it is awide-sense stationary process (WSS). Although in general,wide-sense stationarity is not guaranteed throughout thewhole painting, it is evident that the WSS assumptionholds within small enough patches. Therefore, we rstconstruct a quad-tree decomposition of the entire image,splitting each block when the standard deviation of itspixel values exceeds a given threshold ths. Then, we selectthe most contaminated patch within one quad-tree block(see Section 2.2) to construct the Wiener lter (seeFig. 1block A).

Using the WienerKhinchin theorem, we can obtain thePSD of a signal s through the Fourier transform of itsautocorrelation Rss. Starting from Ryy RxxRnn, werecursively derive the models for Sxx and Snn. Examplesof the autocorrelation Ryy are shown in Figs. 2 and 4 (left).

2.1. WSS or local approach

2.1.1. Image model

The actual image x xi,j of a painting (patch) y can berepresented by either a causal (separable) or a non-causalmodel. In the rst case, we model the pixel values xi,j asthe outcome of a two-dimensional GaussMarkov causalQuad-tree partitioning (homogeneity criterion:

standard deviation)

Extract patches from quad-tree

blocks

Select most contaminated patch

Construct filter HW

Determine from patches

y

Filter

Correction per quad-tree block

x =x + (1-)y

Step 1 Step 2

uad-tree partitioninho eneit criterion: standard deviation

xtractfrom

elect moontaminated patch

Construct filter H

atc es ad-tree

fratches

ilt orrection e

-Step 1 te 2

A

B x

ches (

B. Cornelis et al. / Signal Processing 92 (2012) 116611711168Fig. 2. Autocorrelation Ryy of painting patis reected in the noise autocorrelation Rnn (see Fig. 2,where the difference between the right and the leftimages is Rnn). To model Snn, we subtract the initialmodeled Rxx from Ryy and take the Fourier transformto extract the peaks that correspond to the horizontaland the vertical thread periodicities of the canvas.Retaining these peak frequency components, theirimmediate neighbors and their rst harmonics, and per-forming the inverse FFT results in a fairly goodapproximation of the difference between the observedRyy and the modeled Rxx (see Fig. 3). We assume that thedegree of canvas contamination is evenly present in thehorizontal and the vertical directions. Therefore, weequate the amplitudes of the main peaks in Snn, retainingthe maximum component value. This assumption ismotivated by the experimental results on all our cases,where we observed that horizontal and vertical canvaspatterns are equally well removed. Moreover, as theperiodicity and the orientation of natural canvas canvary locally, we convolve the obtained power spectrumwith a 2D Gaussian function whose variance is chosenexperimentally.

Fig. 3. Left: PSD canvas model. Right: modleft: without canvas; right: with canvas).2.1.3. Model renements

The model of Rxx can now be improved upon byretting the image autocorrelation model on RyyRnn.As an illustration, a total derived model for Ryy RxxRnnis depicted in Fig. 4 (right).

2.2. Non-WSS or global approach

Step 1 of the canvas removal (see Fig. 1block B)consists of applying the Wiener lter on the entire image.In general, the canvas is not uniformly visible throughoutdifferent regions of the painting, violating the stationarityassumption of our lter construction. As a result, theaforementioned approach can lead to artifacts in high-contrast zones (for example dark regions that containbright spots such as cracks exposing the underlying whitepriming layer or a bright brushstroke) or interfere withsharp edges that contain the frequency of the canvasthreads (Fig. 5 shows an example in the iris of SuzanneBambridge after such a ltering). On the other hand,building an adaptive lter through estimation of canvascontamination in an area centered around each pixel

eled autocorrelation Rnn of canvas.

ght: m

B. Cornelis et al. / Signal Processing 92 (2012) 11661171 1169Fig. 4. Left: real autocorrelation Ryy . Riwould be computationally very expensive if not infeasi-ble. To tackle this problem, we propose a global spatiallyadaptive method with low-complexity to correct for thevariation in needed lter strength. The proposed methodis based upon the observation that there seems to be arelation between the average intensity value of a patchand the prominence or visibility of the canvas inducednoise. After the quad-tree decomposition of the entireimage, we can assume that the painting is decomposedinto WSS blocks (of varying sizes). We then extract a largenumber of randomly positioned patches within thisdecomposition of size minimally 5 times the distancebetween successive canvas threads (to decently capturethe noise periodicity) and determine the degree of canvascontamination a, 0rar1, as the ratio between theamplitude of the peaks caused by Snn and the rest of Syy.From these samples, the relation between the meanintensity value of a block and the degree of canvascontamination a is modeled. After applying the Wienerlter on the entire image (Step 1) using the lter fromSection 2.1, the lter strength is corrected per quad-treeblock (Step 2):

x ax^1ay with 0rar1, 2

where x^ is the ltered block and y is the original(contaminated) block.

Fig. 5. Left: original. Right: ltered (odeled autocorrelation Ryy RxxRnn .3. Experiments and results

The datasets used for these experiments are highresolution multispectral images of two paintings consist-ing of 13 16-bit spectral acquisitions (approximately12 0008600 pixels per band), each taken at a specicwavelength within the spectrum. Ten of the thirteenphysical lters of the multispectral camera are centeredat 400 nm, 440 nm,y,760 nm and cover the entire visiblespectrum in narrow bands of 40 nm. The three remaininglters lie in the IR spectrum and are centered at 800 nm,900 nm and 1000 nm respectively and have a bandwidthof 100 nm.1

Since the analysis of each individual spectral band is ofrelevance for the study of the painting in question, theWiener ltering and the correction for non-WSS are doneon each spectral band separately.

In case of the Bambridge painting it can be observedthat the canvas is more present in brighter areas whiledarker areas seem to be less affected (or even not at all)which leads us to assume a linear relationship betweenthe average intensity value of the block and a. Thedimensions of the blocks of the quad-tree decomposition

details are contrast enhanced).

1 More information about the camera specications can be found in [10].

midd

B. Cornelis et al. / Signal Processing 92 (2012) 116611711170Fig. 6. Filtering results on Gauguin painting (left: original;range from 20482048 to 1616 and ths is heuristicallyset to ths 1800. The standard deviation of the Gaussianused to construct the model for Snn (see Section 2.1.2) ischosen to be 0.35.

Some of the results obtained on the 560 nm spectralband of the Gauguin painting (with lter mask size190190) are shown in Fig. 6. Results on a 560 nmspectral band patch of the second painting on canvas,Der Trancespieler (ca. 1909) by Kokoschka, are shown inFig. 7 (lter mask size is 7474). Results on the spectralreconstruction of the Bambridge painting are shown inFig. 8. The spectral reectance curves are constructedthrough cubic spline interpolation and are then projectedinto the RGB color space using the daylight D65 illumi-nant (which is a commonly used standard illuminant,corresponding roughly to a mid-day sun) [11].

Fig. 7. Filtering results on Kokoschka painting (left: original; mid

Fig. 8. Filtering results on Gauguin painting (left: origile: ltered; right: difference), for spectral band at 560 nm.4. Conclusion and discussion

In this paper, we demonstrated that carefulmodeling of the autocorrelation of both the canvas-freepainting and the canvas itself can be used to constructan optimized lter for the removal of canvas patternsin paintings. We proposed a xed Wiener lterfollowed by an extension to cover non-stationarity. Theconstruction of an adaptive lter at each pixel locationwould be computationally too expensive for highresolution datasets. We conclude that our genericapproach leads to a low-complexity method for periodicnoise ltering and removes canvas contamination inpaintings, aiding objective inspection of the stateof a painting and enabling brushstroke analysis andrestoration.

dle: ltered; right: difference), for spectral band at 560 nm.

nal; right: ltered) after spectral reconstruction.

References

[1] B. Cornelis, A. Dooms, J.P. Cornelis, F. Leen, P. Schelkens, Digitalpainting analysis, at the cross section of engineering, mathematicsand culture, in: EUSIPCO 2011 (19th European Signal ProcessingConference 2011), Barcelona, Spain.

[2] C.R. Johnson, E. Hendriks, I. Berezhnoy, E. Brevdo, S. Hughes,I. Daubechies, J. Li, E. Postma, J.Z. Wang, Image processing for artistidentication computerized analysis of Vincent van Goghs paint-ing brushstrokes, IEEE Signal Processing Magazine 25 (4) (2008)3748, doi:10.1109/MSP.2008.923513.

[3] J.M. Hughes, D.J. Graham, D.N. Rockmore, Quantication of artisticstyle through sparse coding analysis in the drawings of PieterBruegel the Elder, Proceedings of the National Academy of Sciences107 (2010) 12791283.

[4] C. Christensen, The painting materials and technique of PaulGauguin, in: Conservation Research, Monograph Series II, Studiesin the History of Art, vol. 41, 1993, pp. 63103.

[5] B. Cornelis, A. Dooms, F. Leen, A. Munteanu, P. Schelkens, Multi-spectral imaging for digital painting analysis: a gauguin case study,

in: Proceedings of SPIE, Applications of Digital Image ProcessingXXXIII, vol. 7798, San Diego, USA.

[6] I. Aizenberg, C. Butakoff, Frequency domain median-like lter forperiodic and quasi-periodic noise removal, Proceedings of SPIE(2002) 181191.

[7] G.A. Hudhud, M. Turner, Digital removal of power frequencyartifacts using a Fourier space median lter, IEEE Signal ProcessingLetters 12 (2005) 573576.

[8] I. Aizenberg, C. Butakoff, A windowed Gaussian notch lter forquasi-periodic noise removal, Image and Vision Computing 26(2008) 13471353.

[9] A.K. Jain, Fundamentals of Digital Image Processing, Prentice-Hall,Inc., Upper Saddle River, NJ, USA, 1989.

[10] P. Cotte, M. Dupouy, CRISATEL: a high resolution multispectralsystem, in: Proceedings of PICS03 Conference, Rochester, USA,2003, pp. 161165.

[11] A. Ribes, F. Schmitt, R. Pillay, C. Lahanier, Calibration and spectralreconstruction for CRISATEL: an art painting multispectral acquisi-tion system, Journal of Imaging Science and Technology 49 (2005)563573.

B. Cornelis et al. / Signal Processing 92 (2012) 11661171 1171

Digital canvas removal in paintingsIntroductionConstructing the filterWSS or local approachImage modelNoise modelModel refinements

Non-WSS or global approach

Experiments and resultsConclusion and discussionReferences