digital canvas removal in paintings

6
Fast communication Digital canvas removal in paintings $ Bruno Cornelis a,b,n , Ann Dooms a,b , Jan Cornelis a , Peter Schelkens a,b,# a Department of Electronics and Informatics (ETRO), Vrije Universiteit Brussel (VUB), Pleinlaan 2, B-1050 Brussels, Belgium b Interdisciplinary Institute for Broadband Technology (IBBT), Gaston Crommenlaan 8 (box 102), B-9050 Ghent, Belgium article info Article history: Received 20 July 2011 Received in revised form 11 November 2011 Accepted 12 November 2011 Available online 19 November 2011 Keywords: Digital painting analysis Canvas removal Denoising Periodic noise Wiener filter abstract The periodic structure of the underlying support of paintings on canvas can become quite prominent and disturbing in high resolution digital recordings. In this paper, we construct a new model and method for the digital removal of canvas which is considered as a noise component superimposed on the painting artwork. The high resolution of the images prohibits the efficient application of existing adaptive denoising filters. Hence, a two-step approach is proposed. First a (smoothing) Wiener filter is applied to the complete image. The second step consists of a spatially adaptive extension with low-complexity to obtain a generic digital canvas removal filter. & 2011 Elsevier B.V. All rights reserved. 1. Introduction Advances in image acquisition and the wide range of imaging modalities currently available have triggered museums to start digitizing their collections, not only for archiving but also for analyzing the art object through its digital image counterpart. This has led to cross-disciplin- ary interaction between image analysis specialists, math- ematicians and art historians, as can be witnessed by our close collaboration with the Royal Museums of Fine Arts of Belgium in Brussels [1]. In this paper, we focus on the visual enhancement of high resolution images of paintings on canvas. This type of support becomes visually promi- nent in recordings, disturbing the digital art analysis process. Therefore, the digital removal of canvas can aid art conservators to better judge the state of a painting or to more accurately determine its history. Furthermore, it can enhance digital inspection of brushstrokes which is used to determine an artist’s style, date the painting or even help in its authentication [2,3]. The question was raised in particular for the painting Portrait of Suzanne Bambridge (1891) by Paul Gauguin (1848–1903), which is in urgent need of restoration and where the emerging grid-like structure caused by the underlying support is quite disturbing. The poor state of the painting can be explained by the circumstances in which Gauguin painted and by his preference for coarse fabric (typically 4–8 threads per cm, sackcloth in case of the Bambridge painting) during his Tahiti period [4]. Moreover, due to the rough texture of the support (mostly apparent in raking light images [5]), dirt accumulated in the creases of the weave and was fixed by varnish applied during conservation treatment and not by Gauguin him- self. This all adds to the prominent appearance of the canvas in digital images of the painting. The problem led us to propose a new generic way of periodic noise filtering for removing canvas contamination in paintings to aid objective inspection of the state of a painting. Several solutions for periodic (or quasi periodic) Contents lists available at SciVerse ScienceDirect journal homepage: www.elsevier.com/locate/sigpro Signal Processing 0165-1684/$ - see front matter & 2011 Elsevier B.V. All rights reserved. doi:10.1016/j.sigpro.2011.11.012 $ This research was supported by the Fund for Scientific Research Flanders (FWO) (PhD fellowship of Bruno Cornelis, Project G.0206.08, Project G021311N and the Postdoctoral fellowship of Peter Schelkens). n Corresponding author at: Department of Electronics and Informatics (ETRO), Vrije Universiteit Brussel (VUB), Pleinlaan 2, B-1050 Brussels, Belgium. Tel.: þ32 26291671. E-mail address: [email protected] (B. Cornelis). # Member of EURASIP. Signal Processing 92 (2012) 1166–1171

Upload: bruno-cornelis

Post on 12-Sep-2016

214 views

Category:

Documents


2 download

TRANSCRIPT

Contents lists available at SciVerse ScienceDirect

Signal Processing

Signal Processing 92 (2012) 1166–1171

0165-16

doi:10.1

$ Thi

Flander

Projectn Corr

(ETRO),

Belgium

E-m# M

journal homepage: www.elsevier.com/locate/sigpro

Fast communication

Digital canvas removal in paintings$

Bruno Cornelis a,b,n, Ann Dooms a,b, Jan Cornelis a, Peter Schelkens a,b,#

a Department of Electronics and Informatics (ETRO), Vrije Universiteit Brussel (VUB), Pleinlaan 2, B-1050 Brussels, Belgiumb Interdisciplinary Institute for Broadband Technology (IBBT), Gaston Crommenlaan 8 (box 102), B-9050 Ghent, Belgium

a r t i c l e i n f o

Article history:

Received 20 July 2011

Received in revised form

11 November 2011

Accepted 12 November 2011Available online 19 November 2011

Keywords:

Digital painting analysis

Canvas removal

Denoising

Periodic noise

Wiener filter

84/$ - see front matter & 2011 Elsevier B.V. A

016/j.sigpro.2011.11.012

s research was supported by the Fund for

s (FWO) (PhD fellowship of Bruno Cornelis,

G021311N and the Postdoctoral fellowship o

esponding author at: Department of Electron

Vrije Universiteit Brussel (VUB), Pleinlaan 2

. Tel.: þ32 26291671.

ail address: [email protected] (B. Corne

ember of EURASIP.

a b s t r a c t

The periodic structure of the underlying support of paintings on canvas can become

quite prominent and disturbing in high resolution digital recordings. In this paper, we

construct a new model and method for the digital removal of canvas which is

considered as a noise component superimposed on the painting artwork. The high

resolution of the images prohibits the efficient application of existing adaptive

denoising filters. Hence, a two-step approach is proposed. First a (smoothing) Wiener

filter is applied to the complete image. The second step consists of a spatially adaptive

extension with low-complexity to obtain a generic digital canvas removal filter.

& 2011 Elsevier B.V. All rights reserved.

1. Introduction

Advances in image acquisition and the wide range ofimaging modalities currently available have triggeredmuseums to start digitizing their collections, not only forarchiving but also for analyzing the art object through itsdigital image counterpart. This has led to cross-disciplin-ary interaction between image analysis specialists, math-ematicians and art historians, as can be witnessed by ourclose collaboration with the Royal Museums of Fine Arts ofBelgium in Brussels [1]. In this paper, we focus on thevisual enhancement of high resolution images of paintingson canvas. This type of support becomes visually promi-nent in recordings, disturbing the digital art analysisprocess. Therefore, the digital removal of canvas can aid

ll rights reserved.

Scientific Research

Project G.0206.08,

f Peter Schelkens).

ics and Informatics

, B-1050 Brussels,

lis).

art conservators to better judge the state of a painting orto more accurately determine its history. Furthermore, itcan enhance digital inspection of brushstrokes which isused to determine an artist’s style, date the painting oreven help in its authentication [2,3].

The question was raised in particular for the paintingPortrait of Suzanne Bambridge (1891) by Paul Gauguin(1848–1903), which is in urgent need of restoration andwhere the emerging grid-like structure caused by theunderlying support is quite disturbing. The poor state ofthe painting can be explained by the circumstances inwhich Gauguin painted and by his preference for coarsefabric (typically 4–8 threads per cm, sackcloth in case ofthe Bambridge painting) during his Tahiti period [4].Moreover, due to the rough texture of the support (mostlyapparent in raking light images [5]), dirt accumulated inthe creases of the weave and was fixed by varnish appliedduring conservation treatment and not by Gauguin him-self. This all adds to the prominent appearance of thecanvas in digital images of the painting.

The problem led us to propose a new generic way ofperiodic noise filtering for removing canvas contaminationin paintings to aid objective inspection of the state ofa painting. Several solutions for periodic (or quasi periodic)

Quad-tree partitioning (homogeneity criterion:

standard deviation)

Extract patches from quad-tree

blocks

Select most contaminated patch

Construct filter HW

Determine α from patches

y

Filter

α

Correction per quad-tree block

x = αx + (1-α)y

Step 1 Step 2

uad-tree partitioninho eneit criterion: standard deviation

xtractfrom

elect moontaminated patch

Construct filter H

atc es ad-tree

fratches

ilt orrection e

-Step 1 te 2

A

B x

Fig. 1. Flowchart of the proposed digital canvas removal method: block

A represents the modeling part, block B represents the two-step

processing applied on the contaminated image y, yielding x.

B. Cornelis et al. / Signal Processing 92 (2012) 1166–1171 1167

noise removal have been proposed in the literature suchas frequency domain median filters [6,7], notch filters, aGaussian notch reject filter (GNF) and its improvement,the Windowed GNF [8]. However, none of them arereadily applicable due to the highly specific nature ofthe canvas contamination.

We start by constructing our filter to specifically targetthe removal of (additive) independent periodic noisecaused by the canvas without affecting the finer struc-tures of the painting (e.g. brushstrokes, craquelure, pre-vious restorations or details). To cover the often irregularnature of canvas in paintings, we extend the filter in a fastsecond step by introducing spatial adaptivity withoutdramatic increase in computational complexity. Ourexperimental results confirm the all-round applicabilityof the resulting filter on two datasets of high-resolutionmultispectral images of paintings on canvas.

2. Constructing the filter

We will tackle the removal of canvas from the digitalimage of a painting as a denoising problem. The uncompro-

mised image x will be retrieved from the contaminated imagey¼ xþn, where the underlying support is represented by anadditive (periodic) noise component n, which is locallyindependent of the painting signal x. We employ a (smooth-ing) Wiener filter in an unconventional way, namely toremove the additive periodic contaminating component n,superimposed on the desired image x. The Wiener filter isoptimal in the sense of minimum mean-square error(MMSE), and defined in the frequency domain as

HW ðu,vÞ ¼Sxxðu,vÞ

Sxxðu,vÞþSnnðu,vÞ, ð1Þ

where Sxxðu,vÞ is the power spectral density (PSD) of theuncontaminated image and Snnðu,vÞ is the PSD of the noise.

Note that the PSD of a signal exists if and only if it is awide-sense stationary process (WSS). Although in general,wide-sense stationarity is not guaranteed throughout thewhole painting, it is evident that the WSS assumptionholds within small enough patches. Therefore, we firstconstruct a quad-tree decomposition of the entire image,splitting each block when the standard deviation of itspixel values exceeds a given threshold ths. Then, we selectthe most contaminated patch within one quad-tree block(see Section 2.2) to construct the Wiener filter (seeFig. 1—block A).

Using the Wiener–Khinchin theorem, we can obtain thePSD of a signal s through the Fourier transform of itsautocorrelation Rss. Starting from Ryy ¼ RxxþRnn, werecursively derive the models for Sxx and Snn. Examplesof the autocorrelation Ryy are shown in Figs. 2 and 4 (left).

2.1. WSS or local approach

2.1.1. Image model

The actual image x¼ ðxi,jÞ of a painting (patch) y can berepresented by either a causal (separable) or a non-causal

model. In the first case, we model the pixel values xi,j asthe outcome of a two-dimensional Gauss–Markov causal

process (or first order Markov random field):

xi,j ¼ r1xi�1,jþr2xi,j�1þr1;2xi�1,j�1þwi,j,

where wi,j is a two-dimensional i.i.d. Gaussian sequenceand r1 and r2 are one step correlations. Whenr1;2 ¼�r1r2, the model is separable and has an exponen-tially decreasing non-isotropic autocorrelation function:

Rxxðt1,t2Þ ¼ s2xr

9t191 r9t29

2 , 0rr1,r2o1,

where r1 and r2 are typically around 0.95 for naturalimages [9] and where s2

x is the variance of pixel values xi,j.In the second case, a two-dimensional non-causal model

is used to represent the pixel values:

xi,j ¼ aðxi�1,jþxi,j�1þxiþ1,jþxi,jþ1Þþwi,j, 9a9o1=4:

Within this model, the autocorrelation behaves almostisotropically at small displacements [9] and can beapproximated by an isotropic (non-separable) function:

Rxxðt1,t2Þ ¼ s2xr

ffiffiffiffiffiffiffiffiffiffiffiffiðt2

1þ t22Þ

pwhere r¼ e�

ffiffiap

:

Based upon the minimal error obtained after leastsquares fitting (LSF) on Ryy , the first or second model isinitially selected to represent the autocorrelation Rxx ofthe uncompromised painting (patch) x.

2.1.2. Noise model

Obviously, a canvas contaminated patch contains astrong horizontal and vertical periodic component which

Fig. 2. Autocorrelation Ryy of painting patches (left: without canvas; right: with canvas).

Fig. 3. Left: PSD canvas model. Right: modeled autocorrelation Rnn of canvas.

B. Cornelis et al. / Signal Processing 92 (2012) 1166–11711168

is reflected in the noise autocorrelation Rnn (see Fig. 2,where the difference between the right and the leftimages is Rnn). To model Snn, we subtract the initialmodeled Rxx from Ryy and take the Fourier transformto extract the peaks that correspond to the horizontaland the vertical thread periodicities of the canvas.Retaining these peak frequency components, theirimmediate neighbors and their first harmonics, and per-forming the inverse FFT results in a fairly goodapproximation of the difference between the observedRyy and the modeled Rxx (see Fig. 3). We assume that thedegree of canvas contamination is evenly present in thehorizontal and the vertical directions. Therefore, weequate the amplitudes of the main peaks in Snn, retainingthe maximum component value. This assumption ismotivated by the experimental results on all our cases,where we observed that horizontal and vertical canvaspatterns are equally well removed. Moreover, as theperiodicity and the orientation of natural canvas canvary locally, we convolve the obtained power spectrumwith a 2D Gaussian function whose variance is chosenexperimentally.

2.1.3. Model refinements

The model of Rxx can now be improved upon byrefitting the image autocorrelation model on Ryy�Rnn.As an illustration, a total derived model for Ryy ¼ RxxþRnn

is depicted in Fig. 4 (right).

2.2. Non-WSS or global approach

Step 1 of the canvas removal (see Fig. 1—block B)consists of applying the Wiener filter on the entire image.In general, the canvas is not uniformly visible throughoutdifferent regions of the painting, violating the stationarityassumption of our filter construction. As a result, theaforementioned approach can lead to artifacts in high-contrast zones (for example dark regions that containbright spots such as cracks exposing the underlying whitepriming layer or a bright brushstroke) or interfere withsharp edges that contain the frequency of the canvasthreads (Fig. 5 shows an example in the iris of SuzanneBambridge after such a filtering). On the other hand,building an adaptive filter through estimation of canvascontamination in an area centered around each pixel

Fig. 4. Left: real autocorrelation Ryy . Right: modeled autocorrelation Ryy ¼ RxxþRnn .

Fig. 5. Left: original. Right: filtered (details are contrast enhanced).

1 More information about the camera specifications can be found in [10].

B. Cornelis et al. / Signal Processing 92 (2012) 1166–1171 1169

would be computationally very expensive if not infeasi-ble. To tackle this problem, we propose a global spatiallyadaptive method with low-complexity to correct for thevariation in needed filter strength. The proposed methodis based upon the observation that there seems to be arelation between the average intensity value of a patchand the prominence or visibility of the canvas inducednoise. After the quad-tree decomposition of the entireimage, we can assume that the painting is decomposedinto WSS blocks (of varying sizes). We then extract a largenumber of randomly positioned patches within thisdecomposition of size minimally 5 times the distancebetween successive canvas threads (to decently capturethe noise periodicity) and determine the degree of canvas

contamination a, 0rar1, as the ratio between theamplitude of the peaks caused by Snn and the rest of Syy.From these samples, the relation between the meanintensity value of a block and the degree of canvascontamination a is modeled. After applying the Wienerfilter on the entire image (Step 1) using the filter fromSection 2.1, the filter strength is corrected per quad-treeblock (Step 2):

x¼ axþð1�aÞy with 0rar1, ð2Þ

where x is the filtered block and y is the original(contaminated) block.

3. Experiments and results

The datasets used for these experiments are highresolution multispectral images of two paintings consist-ing of 13 16-bit spectral acquisitions (approximately12 000�8600 pixels per band), each taken at a specificwavelength within the spectrum. Ten of the thirteenphysical filters of the multispectral camera are centeredat 400 nm, 440 nm,y,760 nm and cover the entire visiblespectrum in narrow bands of 40 nm. The three remainingfilters lie in the IR spectrum and are centered at 800 nm,900 nm and 1000 nm respectively and have a bandwidthof 100 nm.1

Since the analysis of each individual spectral band is ofrelevance for the study of the painting in question, theWiener filtering and the correction for non-WSS are doneon each spectral band separately.

In case of the Bambridge painting it can be observedthat the canvas is more present in brighter areas whiledarker areas seem to be less affected (or even not at all)which leads us to assume a linear relationship betweenthe average intensity value of the block and a. Thedimensions of the blocks of the quad-tree decomposition

Fig. 6. Filtering results on Gauguin painting (left: original; middle: filtered; right: difference), for spectral band at 560 nm.

Fig. 7. Filtering results on Kokoschka painting (left: original; middle: filtered; right: difference), for spectral band at 560 nm.

Fig. 8. Filtering results on Gauguin painting (left: original; right: filtered) after spectral reconstruction.

B. Cornelis et al. / Signal Processing 92 (2012) 1166–11711170

range from 2048�2048 to 16�16 and ths is heuristicallyset to ths ¼ 1800. The standard deviation of the Gaussianused to construct the model for Snn (see Section 2.1.2) ischosen to be 0.35.

Some of the results obtained on the 560 nm spectralband of the Gauguin painting (with filter mask size190�190) are shown in Fig. 6. Results on a 560 nmspectral band patch of the second painting on canvas,Der Trancespieler (ca. 1909) by Kokoschka, are shown inFig. 7 (filter mask size is 74�74). Results on the spectralreconstruction of the Bambridge painting are shown inFig. 8. The spectral reflectance curves are constructedthrough cubic spline interpolation and are then projectedinto the RGB color space using the daylight D65 illumi-nant (which is a commonly used standard illuminant,corresponding roughly to a mid-day sun) [11].

4. Conclusion and discussion

In this paper, we demonstrated that carefulmodeling of the autocorrelation of both the canvas-freepainting and the canvas itself can be used to constructan optimized filter for the removal of canvas patternsin paintings. We proposed a fixed Wiener filterfollowed by an extension to cover non-stationarity. Theconstruction of an adaptive filter at each pixel locationwould be computationally too expensive for highresolution datasets. We conclude that our genericapproach leads to a low-complexity method for periodicnoise filtering and removes canvas contamination inpaintings, aiding objective inspection of the stateof a painting and enabling brushstroke analysis andrestoration.

B. Cornelis et al. / Signal Processing 92 (2012) 1166–1171 1171

References

[1] B. Cornelis, A. Dooms, J.P. Cornelis, F. Leen, P. Schelkens, Digitalpainting analysis, at the cross section of engineering, mathematicsand culture, in: EUSIPCO 2011 (19th European Signal ProcessingConference 2011), Barcelona, Spain.

[2] C.R. Johnson, E. Hendriks, I. Berezhnoy, E. Brevdo, S. Hughes,I. Daubechies, J. Li, E. Postma, J.Z. Wang, Image processing for artistidentification — computerized analysis of Vincent van Gogh’s paint-ing brushstrokes, IEEE Signal Processing Magazine 25 (4) (2008)37–48, doi:10.1109/MSP.2008.923513.

[3] J.M. Hughes, D.J. Graham, D.N. Rockmore, Quantification of artisticstyle through sparse coding analysis in the drawings of PieterBruegel the Elder, Proceedings of the National Academy of Sciences107 (2010) 1279–1283.

[4] C. Christensen, The painting materials and technique of PaulGauguin, in: Conservation Research, Monograph Series II, Studiesin the History of Art, vol. 41, 1993, pp. 63–103.

[5] B. Cornelis, A. Dooms, F. Leen, A. Munteanu, P. Schelkens, Multi-spectral imaging for digital painting analysis: a gauguin case study,

in: Proceedings of SPIE, Applications of Digital Image ProcessingXXXIII, vol. 7798, San Diego, USA.

[6] I. Aizenberg, C. Butakoff, Frequency domain median-like filter forperiodic and quasi-periodic noise removal, Proceedings of SPIE(2002) 181–191.

[7] G.A. Hudhud, M. Turner, Digital removal of power frequencyartifacts using a Fourier space median filter, IEEE Signal ProcessingLetters 12 (2005) 573–576.

[8] I. Aizenberg, C. Butakoff, A windowed Gaussian notch filter forquasi-periodic noise removal, Image and Vision Computing 26(2008) 1347–1353.

[9] A.K. Jain, Fundamentals of Digital Image Processing, Prentice-Hall,Inc., Upper Saddle River, NJ, USA, 1989.

[10] P. Cotte, M. Dupouy, CRISATEL: a high resolution multispectralsystem, in: Proceedings of PICS’03 Conference, Rochester, USA,2003, pp. 161–165.

[11] A. Ribes, F. Schmitt, R. Pillay, C. Lahanier, Calibration and spectralreconstruction for CRISATEL: an art painting multispectral acquisi-tion system, Journal of Imaging Science and Technology 49 (2005)563–573.