ieee-tgrs-2011_2

738 IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, VOL. 49, NO. 2, FEBRUARY 2011

A New Pan-Sharpening Method Usinga Compressed Sensing Technique

Shutao Li, Member, IEEE, and Bin Yang

AbstractThis paper addresses the remote sensing image pan-sharpening problem from the perspective of compressed sensing(CS) theory which ensures that with the sparsity regularization,a compressible signal can be correctly recovered from the globallinear sampled data. First, the degradation model from a high-to low-resolution multispectral (MS) image and high-resolutionpanchromatic (PAN) image is constructed as a linear samplingprocess which is formulated as a matrix. Then, the model matrix isconsidered as the measurement matrix in CS, so pan-sharpeningis converted into signal restoration problem with sparsity regular-ization. Finally, the basis pursuit (BP) algorithm is used to resolvethe restoration problem, which can recover the high-resolution MSimage effectively. The QuickBird and IKONOS satellite images areused to test the proposed method. The experimental results showthat the proposed method can well preserve spectral and spatialdetails of the source images. The pan-sharpened high-resolutionMS image by the proposed method is competitive or even superiorto those images fused by other well-known methods.

Index TermsCompressed sensing, image fusion, multispectral(MS) image, panchromatic (PAN) image, remote sensing, sparserepresentation.

I. INTRODUCTION

O PTICAL remote sensors in satellites can provide imagesabout the surface of the Earth which are valuable forenvironmental monitoring, land-cover classification, weatherforecasting, etc. Practically, most optical Earth observationsatellites, such as QuickBird and IKONOS, provide imagedata with spectral and spatial information separately, such aslow-resolution multispectral (MS) images and high-resolutionpanchromatic (PAN) image. In order to benefit from both spec-tral and spatial information, these two kinds of images can befused to get one high-spectral- and high-spatial-resolution re-mote sensing image [1]. The fusing process, classically referredas pan-sharpening technique, has become a key preprocessingstep in many remote sensing applications [2].

Various methods have been proposed for pan-sharpening,which usually consider physics of the remote sensing processand make some assumptions on the original PAN and MS

Manuscript received November 27, 2009; revised April 5, 2010 andJuly 2, 2010; accepted August 5, 2010. Date of publication October 11, 2010;date of current version January 21, 2011. This work was supported in part bythe National Natural Science Foundation of China (No. 60871096), the Ph.D.Programs Foundation of Ministry of Education of China (No.200805320006),the Key Project of Chinese Ministry of Education (2009-120), and the OpenProjects Program of National Laboratory of Pattern Recognition.

The authors are with the College of Electrical and Information Engineering,Hunan University, Changsha 410082, China (e-mail: [email protected];[email protected]).

Color versions of one or more of the figures in this paper are available onlineat http://ieeexplore.ieee.org.

Digital Object Identifier 10.1109/TGRS.2010.2067219

images [3][7]. The most classical methods are projectionsubstitution-based which assume that the PAN image is equiv-alent to the structural component of the MS images whenprojected the MS images into a new space. The most famousprojectionsubstitution methods include the intensity hue satu-ration (IHS) [8], [9], the principal component analysis [10], theGramSchmidt (GS) transform [11] based methods, and so on.

In recent years, the methods based on the ARSIS concepthave been popular which assume that the missing spatial in-formation in the MS images can be obtained from the highfrequencies of the PAN image [12], [13]. In [2], Thomas et al.showed that this category method prevents from introducingspectral distortion into fused products in some degree. Thus,it offers a reliable framework for further developments. Themultiresolution transforms, such as discrete wavelet transform(DWT) [6], trous wavelet transform (ATWT) [14], con-tourlets [15], and support value transform [16], are usuallyused to extract the high frequencies of the PAN image. TheATWT allows an image to be decomposed into nearly disjointedbandpass changes in spatial frequency domain, which makesit particularly suitable for remote image fusion. An ATWT-based method with the additive wavelet luminance proportional(AWLP) model was proposed in [17]. The AWLP model injectshigh-pass details proportionally to low-pass MS components,and improves the spectral quality of fused image.

Nevertheless, the high frequencies extracted from the PANimage are not exactly equivalent to those of the MS images. Soadjusting those high frequencies, called the interband structuremodel (IBSM) [13], is needed before they are injected into theMS images. One of the famous IBSMs is the context-basedinjection model [18]. The undecimated DWT or the generalizedLaplacian pyramid is used to extract the high-frequency detailsof the PAN image. Garzelli and Nencini proposed the ATWT-based method with context-based decision (CBD) injectionmodel in [19]. The experimental results showed that the CBDmodel yields better results than those traditional methods. In ad-dition, Garzelli and Nencini applied the genetic algorithm [20]to optimize the injection model of the ATWT-based method bymaximizing the quality index of the fused product, and the pan-sharpening results are obviously improved.

Also, as a popular idea, the inverse-problem-based methodsare used to restore the original high-resolution MS image fromits degraded versions, i.e., the PAN and MS images [21], [22].Because much information is lost in the degrading process, thisis an ill-posed inverse problem and the solution is not unique.This means that according to the degrading process, many dif-ferent high-resolution MS images can produce the same high-resolution PAN and low-resolution MS images. Thus, various

0196-2892/$26.00 2010 IEEE

LI AND YANG: NEW PAN-SHARPENING METHOD USING A COMPRESSED SENSING TECHNIQUE 739

regularization-based methods have been proposed to resolvethe ill-posed inverse problem [23], [24]. In [23], the pan-sharpening process is modeled as restoring the high-resolutionMS images aided by the PAN image. The fused results canpreserve spectral characteristics of the true MS images aswell as high spatial resolution of the source PAN image. Theregularization and iterative optimal method was proposed in[24], in which the fused result is obtained through optimizingan image quality index. The Bayesian method is an importanttool to resolve ill-posed inverse problems [25][27]. Withinthe Bayesian framework, the fused image is extracted from theBayesian posterior distribution in which the prior knowledgeand artificial constraints on the fusion results are incorporated.Joshi and Jalobeanu modeled the pan-sharpening problem asseparate inhomogeneous Gaussian Markov random fields andapplied a maximum a posteriori estimation to obtain the fusedimage for each MS band [27]. The experiments on both syn-thetic data and the QuickBird satellite images demonstrate itseffectiveness. However, its performance would degrade if thereare not enough training images.

Recently, Donoho and Cands et al. proposed a new sam-pling theory, compressed sensing (CS) theory, for data acquisi-tion [28], [29]. The CS theory can recover an unknown sparsesignal from a small set of linear projections, which has been ap-plied in many image processing applications [30][34]. The keypoint of CS theory is the sparsity regularization which refers tothe nature of the natural signals [28]. It is very suitable to re-solve the inverse problem of compressible signals/images. Withthe perspective of compressed sensing, this paper proposesa novel pan-sharpening method with sparsity regularization.We formulate the remote sensing imaging formation model aslinear transform corresponding to the measurement matrix inthe CS theory and the high-resolution PAN and low-resolutionMS images are referred as measurements. Thus, based on thesparsity regularization, the high-resolution MS images can berecovered precisely.

The rest of this paper is organized into five sections. InSection II, the sparsity representation and the CS theory arebriefly introduced. In Section III, we give the model of thehigh-resolution PAN image and the low-resolution MS imagesfrom the high-resolution MS images. Then, the fusion schemebased on the CS theory is proposed. Numerical experiments anddiscussions are presented in Section IV. Conclusions togetherwith some suggestions about the future work are given inSection V.

II. SPARSE REPRESENTATION AND COMPRESSED SENSING

The development of image processing in the past severaldecades reveals that a reliable image model is very important.In fact, natural images tend to be sparse in some image basesspace [30]. This brings us to the sparse and redundant represen-tation model of image.

Consider a family of signals {xi, i = 1, 2, . . . , g}, xi Rn.Specially, in this paper, each such signal is assumed to be ann image patch, obtained by lexicographically stacking

the pixel values. Sparse representation theory supposes the ex-istence of a matrix D RnT , n T , each column of whichcorresponds to a possible image (lexicographically stacking the

pixel values as a vector). These possible images are referredto as atomic images, and the matrix D as a dictionary of theatomic images. Thus, an image signal x can be representedas x = D. For overcomplete D (n T ), there are manypossible satisfying x = D. Our aim is to find the with the fewest nonzero elements. Thus, the is called thesparse representation of x with dictionary D. Formally, thisproblem can be obtained by solving the following optimizationproblem:

= argmin 0 s.t.D x22 = 0 (1)

where 0 denotes the number of nonzero components in .In practice, because of various restrictions, we cannot get x

directly; instead, only a small set of measurements y of x isobserved. The observation y can be represented as

y = Lx (2)

where L Rkn with k < n. (2) is interpreted as the encodeprocess of the CS theory, where L is a CS measurement matrix.The CS theory ensures that under sparsity regularization [29],the signal x can be correctly recovered from the observa-tion y by

min 0 s.t.y 22 (3)

where = LD, is the reconstruction error which depends onnoise level of source image, and x = D. The effectivenessof sparsity as a prior for regularizing the ill-posed problem hasbeen validated by many literatures [30][34]. In this paper, wepropose one remote sensing image fusion method from the per-spective of compressed sensing. The high-resolution PAN andlow-resolution MS images are referred as the measurementsy. The matrix L is constructed by the model from the high-resolution MS images to the high-resolution PAN and low-resolution MS images. Thus, the sparse representation of thehigh-resolution MS images corresponding to dictionary D canbe recovered from measurements y according to the sparsityregularization of (3), and the high-resolution MS images areconstructed by x = D.

However, because 0 is combinatorial, the above opti-mization is an NP-hard problem [30]. In fact, when the coeffi-cients are sufficiently sparse, this problem can be replaced withminimizing the l1-norm problem

min 1 s.t.y 22 . (4)The proof from (3) to (4) can be found in [35]. The l1-norm

problem (4) can be efficiently resolved by the basis pursuit (BP)method [36]. In the BP method, (4) is converted to the standardlinear program formed as

min cT s.t.A = b, 0 (5)

where = (+;); c = (1;1); A = (,); b = y. Then,the sparse representation in (4) can be obtained by = + . The sharpened high-resolution MS image patch is obtainedby x = D.


Fig. 1. Relationship between a single low-resolution MS band and its corre-sponding high-resolution version ( 2 means the decimation).

III. PROPOSED IMAGE FUSION SCHEME

A. Image Formation Model

Remote sensing physics should be carefully considered whiledesigning the pan-sharpening process. Let Xhighp and Y lowp ,p = 1, . . . , P , represent the pth band of the high-resolutionand low-resolution MS images, respectively, where P denotesthe number of bands of the MS images. The observed low-resolution MS images are modeled as decimated and noisyversions of the corresponding high-resolution MS images, asshown in Fig. 1.

In fact, intensity of the low-resolution image is due to inte-gration of light intensity that falls on the charge-coupled devicesensor array of suitable area compared to the desired the high-resolution images [23], so the low-resolution intensity can beseen as neighborhood pixels average of the high-resolutionintensities corrupted with additive noise. The relationship be-tween Xhighp and Y lowp is written as

Y lowp = MXhighp + vp (6)

where M is the decimation matrix and vp is the noise vector.In fact, the PAN image usually covers a broad range of

the wavelength spectrum; whereas, one MS band covers onlya narrow spectral range. Moreover, the range of wavelengthspectrum of the PAN modality is usually overlapped or partlyoverlapped with those of the MS bands. This overlappingcharacteristic motivates us making the assumption that the PANimage is approximately written as a linear combination of theoriginal MS images

Y PAN =p

wpXhighp + vp (7)

where wp is the weight and vp is the additive zero-meanGaussian noise [29]. However, we should note that the linearrelationship between the PAN and the MS image is only ap-proximated by the linear model because of the complexity ofphysics, atmospheric dispersion, and so on.

We consider a pan-sharpening case with four spectral bands:1) Red (R); 2) green (G); 3) blue (B); 4) and near infrared(NIR), and the decimation factor from high to low spatial reso-lution is four. Fig. 2 shows the remote sensing image formationmodel, representing the relationship between the available low-resolution MS images and its high-resolution versions. Letx = (x1,1, . . . , x1,16, . . . , x4,1, . . . , x4,16)

T represent the lexi-cographically ordered high-spatial-resolution MS image patchand yMS = (y1, y2, y3, y4)T is the vector consisting of thepixels from the low-resolution MS images shown in Fig. 2.Then, we can write

yMS = M1x+ v1 (8)

Fig. 2. Remote sensing image formation model.

where M1 is the decimation matrix of size 4 64 and v1 isthe 4 1 zero-mean Gaussian noise vector. The M1 matrixcan be written as M1 = (1/16) I 1T, where I R44 isan identity matrix, and 1 is a 16 1 vector with all entriesequaling to one.

Let yPAN = (y5, . . . , y20)T as the corresponding lexico-graphically ordered high-resolution PAN image patch, then wecan write

yPAN = M2x+ v2 (9)

where M2 = (w1I, w2I, w3I, w4I), and I R1616 isan identity matrix. The v2 is the additive zero-mean Gaussiannoise vector. We assume that v1 and v2 have the same standarddeviation . Then, combining (8) and (9), we get

y = Mx+ v (10)

where y =(yMSyPAN

)and M =

(M1M2

).

The goal of image fusion is to recover x from y. As shownin Section II, if signal is compressible by a sparsity transform,the CS theory ensures that the original signal can be accuratelyreconstructed from a small set of incomplete measurements.Thus, the signal recovering problem of (10) can be formulatedas a minimization problem with sparsity constraints

= argmin 0 s.t.y 22 (11)

where = MD, D = (d1, d2, . . . , dK) is a dictionary andx = D which explains x as a linear combination of columnsfrom D. The vector is very sparse. Finally, the estimated xcan be obtained by x = D. In this paper, we assume thatthe source images are only contaminated by the additive zero-mean Gaussian noise. Thus, the parameter can be set by = n(C)2, where n is the length of y, C is a constant [31].The noise level is estimated by user according to real satelliteoptical imaging system, and the constant C can be set by thegeneralized Rayleigh law as described in [31]. In the followingexperiments, the constant C is set to 1.15 for 8 8 imagepatch [32]. We can see that (11) is the same as (3) mentioned in


Fig. 3. Proposed fusion scheme.

Section II. Thus, this problem can be easily resolved by the BPmethod [36].

In addition, we should note that (11) is operated on 4 4 4 image patch in the high-resolution MS images correspond-ing to 4 4 image patch in the PAN image and one pixel in eachband of the MS images. However, such small patch containslittle texture and structure information. On the other hand, alarge patch captures local texture well but lacks generalizationcapability, and involves high computational complexity. In thispaper, we set the size of image patch in the high-resolution MSimages as 8 8 4, corresponding to 8 8 image patch inthe PAN image and 2 2 4 patch in the low-resolution MSimages, which appears to be a good tradeoff. Thus, the matrixM1 in (8) is constructed as (1/16) I88 (1T (I22 1T)), where INN is a N N identity matrix and 1 is a 4 1 vector with all entries equal to one. The matrix M2 in (9) isconstructed as (w1I w2I w3I w4I) where I R6464 isan identity matrix.

The proposed pan-sharpening scheme is illustrated in Fig. 3.As shown in Fig. 3, all the patches of the PAN and MS imagesare processed in raster-scan order, from left-top to right-bottomwith step of four pixels in the PAN image and one pixel in theMS images. First, according to Fig. 3, the PAN patch yPANis combined with the MS patch yMS to generate the vectory. Then, the sparsity regularization (11) is resolved using theBP method to get the sparse representation of the fused MSimage patch. Finally, the fused MS image patch is obtained byx = D.

B. Random Raw Patches Dictionary

The dictionary of sparse representation is a collection ofparameterized waveforms called atoms [36]. The dictionaryis overcomplete; in which case, the number of atoms of thedictionary exceeds the length of signal. In recent years, thesparsity-regularization-based image processing methods, suchas denoising [31], [32], face recognition [33], and super-resolution [34], always lead to the state-of-the-art results.

In image processing applications, each column of the over-complete dictionary D corresponds to an image or image patchin Rn. Those images or image patches are referred as atomicimages. A collection of parameterized 2-D waveforms of wave-lets, Gabor dictionaries, and cosine packets is the commonlyused dictionary [36]. Moreover, an application-orientated non-parametric dictionary may produce better result. However, dic-tionary learning is difficult and time-consuming. Simply, thedictionary can also be generated by randomly sampling rawpatches from training images with similar statistical nature. Theeffectiveness of this kind of dictionary is validated in face

recognition and image super-resolution application [33], [34].In this paper, we generated the dictionary by randomly sam-pling raw patches from high-resolution MS satellite images.Because different satellite optical sensors have different char-acteristics in reflecting spectral information of scene, the high-resolution MS images with the same type of sensors should beused to construct the dictionary. When the sparse regularizationis used, the proposed method selects a number of relevantdictionary elements adaptively for each patch. Thus, the recon-structed image can well preserve the spectral information of thescene. Our experiments in the following section also show theeffectiveness of such dictionary.

In addition, based on the CS theory, correctly recoveringsignal x in (11) requires that the measurement matrix M isincoherent/uncorrelated to the dictionary D. Because that thedictionary is generated by randomly sampling, it has the highprobability satisfying this condition. Thus, for a variety ofmeasurement matrix M, any sufficiently sparse linear represen-tation of x in terms of the dictionary D can be recovered almostperfectly from the high-resolution PAN and low-resolution MSimages.

IV. EXPERIMENTAL RESULTS AND COMPARISONS

A. Quality Indices for Assessing Image FusionTo evaluate different fusion methods, we need to check

the synthesis property that any synthetic image should be asidentical as possible to the image that the corresponding sensorwould observe with the highest spatial resolution [37], [38]. So,Walds protocol [37] is adopted, i.e., the fusion is performed ondegraded data sets and the fused high-resolution MS imagesare then compared with the original low-resolution MS imageswhich are seen as the reference images. In this paper, thefollowing five typical evaluation metrics are used.

1) The correlation coefficient (CC) [39] is calculated by

CC =

Mi=1

Nj=1

[F (i, j) F ] [X(i, j) X]

Mi=1

Nj=1

[F (i, j) F ]2 M

i=1

Nj=1

[X(i, j) X]2

(12)where X and F denote the source MS images andthe fused image with size M N , respectively. Thecorrelation coefficient indicates the degree of correlationbetween X and F . When F equals to X , the correlationcoefficient approaches to one.

2) The spectral angle mapper (SAM) [39] reflects the spec-tral distortion by the absolute angles between the twovectors constructed from each pixel of the source imageX and the fused image F . Let uX and uF denote thespectral vector of a pixel of X and F , respectively. TheSAM is calculated by

SAM = arccos( uX ,uF uX2 uF 2

). (13)

The SAM is averaged over the whole image to yield aglobal measurement of spectral distortion. For ideal fusedimage, the SAM value should be zero.


Fig. 4. Twenty satellite images used to construct dictionary.

3) The root mean squared error (RMSE) gives the stan-dard measure of difference between two images. It isdefined as

RMSE = 1MN

MNi,j=1

(X(i, j) F (i, j))2. (14)

4) The erreur relative global adimensionnelle de synthse(ERGAS) gives a global quality measure of the fusedimage F [38], which is defined as

ERGAS = 100hl

1P

Li=1

(RMSE2(i)MEAN(i)

)(15)

where h is spatial resolution of the PAN image; l is spatialresolution of the MS images; P is the number of bands ofthe fused image F ; MEAN(i) is the mean value of theith band of the reference MS images and RMSE(i) is theRMSE between the ith band of the reference MS imagesand the ith band of the fused image. Smaller ERGASindicates better fusion result (ideally zero).

5) The Q4 index is defined as [40]Q4 = E(Q4DD) (16)

where Q4DD={4[E(xy)xy]/(E(x2)x2+E(y2)y2)} {x y/(x2+y2)}. The

quaternion x = X1(m,n) + iX2(m,n) + jX3(m,n) +kX4(m,n), y = F1(m,n) + iF2(m,n) + jF3(m,n) +kF4(m,n), Fi and Xi are the ith band of F and X ,respectively. The y is the complex conjugate of y, E()denotes the quaternion obtained by averaging the pixelquaternion within a D D block with size of 8 8, andx is the modulus of x. The Q4 index takes values in[0, 1] with one being the best value.

B. Fusion Results of QuickBird DataQuickBird is a high-resolution satellite which provides PAN

image at 0.7-m resolution and MS images at 2.8-m resolution.In order to evaluate the fusion results with objective measures,we spatially degrade the PAN and MS images with low-passfilter and decimation operator by four to yield one 2.8-m resolu-tion PAN image and four 11.2- m resolution MS images. Then,the degraded low-resolution PAN and MS images are fused toproduce one high-resolution MS image with 2.8-m resolution.The fused results are compared with the true 2.8-m MS imageusing the objective measures. The proposed method needs adictionary which contains the raw image patches respondingto 2.8-m resolution MS bands. In this paper, we randomlysample 10 000 raw patches from 20 images downloaded fromthe website http://glcf.umiacs.umd.edu/data/quickbird/. Empir-ically, we find that such dictionary can always produce verygood fusion results. Fig. 4 shows 20 training images used


Fig. 5. Source QuickBird images (dock) and the fused results using differentmethods. (a) The resampled low-resolution MS image (RGB, 256 256,11.2 m); (b) High-resolution PAN image (256 256, 2.8 m); (c) Original high-resolution MS image (RGB, 256 256, 2.8 m); (d) Fused RGB bands by SVTmethod; (e) Fused RGB bands by GA method; (f) Fused RGB bands by theproposed method.

to generate the dictionary. Here, only three color composites(bands 321) of the 2.8-m resolution MS data are reported.

One free parameter of the proposed method is , whichrestricts the reconstruction error. When the noise level of thesource images is , we can directly set = n(C)2. In ourexperience, the source images are all clear (standard deviation = 0), thus according to = n(C)2, the parameter shouldbe zero. However, as the value of decreases, the process ofthe BP becomes slower. In fact, in our experiments, the pan-sharpening quality is stable over a large range of , such as [0, 5]. Therefore, in order to ensure pan-sharpening qualityas well as the computation efficiency of the proposed method,the parameter is set to be 1 in our experimental section. Theexperiments show that this gives good results for all the testcases. The weights in (9) are set from the QuickBird normalizedspectral response as: w1 = 0.1139 for the Blue band; w2 =0.2315 for the Green band; w3 = 0.2308 for the Red band; andw4 = 0.4239 for the NIR band [21]. In addition, we normalizecolumns of the dictionary as di/di2, i = 1, 2, . . . , 10000, sothat the atoms in the dictionary have unit length.

The proposed method is compared with six commonly usedpan-sharpening methods, namely, the generalized IHS (GIHS)[9], the GS transform [11], SVT [16], the ATWT-based methodwith the CBD injection model [19], the ATWT-based methodwith the AWLP model [17], and the ATWT-based method withthe genetic algorithm (GA) [20]. Two levels of decompositionsare used for the ATWT transform used for CBD, AWLP, andGA. In the case of SVT-based method, the 2 in the GaussianRBF kernel is set to 1.2, and the parameter of the mapped LS-SVM is set to 1, which give the best results. The GS algorithm isimplemented in the environment for visualizing images (ENVI)software [11], and we set that the low-resolution PAN image issimulated by averaging the low-resolution MS images.

Figs. 5(a) and (b) and 6(a) and (b) show two QuickBridsource images of the urban areas of Sundarbans, India, acquiredon November 21, 2002. The original true 2.8-m RGB bands

Fig. 6. Source QuickBird images (highway) and the fused results usingdifferent methods. (a) The resampled low-resolution MS image (RGB, 256 256, 11.2 m); (b) High-resolution PAN image (256 256, 2.8 m); (c) Originalhigh-resolution MS image (RGB, 256 256, 2.8 m); (d) Fused RGB bands bySVT method; (e) Fused RGB bands by GA method; (f) Fused RGB bands bythe proposed method.

of the MS data are presented in Figs. 5(c) and 6(c) for visualreference. In order to save space, only the results of SVT- andGA-based methods and the proposed method are illustrated inFigs. 5(d)(f) and 6(d)(f), respectively. By visually comparingthe fused images with the original source images, we can seethat all the experimental methods can effectively pan-sharpenthe MS data. However, compared to the original reference MSimages, the pan-sharpened images with SVT- and GA-basedmethods are blurred in some degree. Careful inspections ofFigs. 5(f) and 6(f) indicate that the proposed method not onlyprovides high-quality spatial details but also preserves spectralinformation well. Particularly, the road in Fig. 6(f) is clearerthan that in Fig. 6(d) and (e).

The objective measures for Figs. 5 and 6 are reported inTables I and II, respectively, in which the best results for eachquality measure are labeled in bold. The correlation coefficient(CC) allows us to determine the correlation between the pan-sharpened band and the original band in both spectral andspatial information. Both the tables show that the proposedapproach provides the highest CC values for all the four bands.The root mean squared error (RMSE) measure gives radiomet-ric distortion of the pan-sharpened band from the original MSdata, while ERGAS offers a global depiction of the quality ofradiometric distortion of the fused product. We can see that theproposed method only loses the NIR band for the RMSE. Forthe ERGAS, the proposed method also gives the best results.The SAM yields a global measurement of spectral distortionof the pan-sharpened images. Both the two tables show thatthe AWLP method gives the lowest SAM. We should notethat, for other quality indexes, the proposed method performsobviously better than the AWLP method. Since the ERGASonly considers RMSE, and the SAM only considers spectraldistortion, a more comprehensive measure of quality Q4 hasbeen developed to test both spectral and spatial qualities of thefused images. For the Q4, the proposed method also providesthe best results. This is mainly due to one advantage of the


TABLE ICOMPARISON OF THE PROPOSED ALGORITHM WITH THE EXISTING

METHODS ON QUICKBIRD IMAGES SHOWN IN FIG. 5

TABLE IICOMPARISON OF THE PROPOSED ALGORITHM WITH THE EXISTING

METHODS ON QUICKBIRD IMAGES SHOWN IN FIG. 6

proposed scheme over other comparing methods that it selectsatom patches from the dictionary adaptively for each patch.Those patches are the most relevant ones in the dictionary torepresent the given low-resolution MS images and the PANimage because of the sparsity regularization in (11). Thus,the reconstructed high-resolution MS images from the selectedpatches can preserve well both spatial and spectral informationof the source images.

C. Fusion Results of IKONOS DataIn this section, we describe and analyze the experimental

results on the IKONOS satellite data. The IKONOS satel-lite captures 4-m MS images, i.e., blue, green, red, andNIR images, and 1-m PAN image. As the QuickBrid exper-iment, we spatially degrade the IKONOS images with dec-imation by four to yield 4-m PAN and 16-m MS images.Then, the degraded images are fused to produce 4-m MSimages. The fusion results are compared with the true 4-mMS image to achieve the objective measures. The dictionary,which contains the 10 000 raw image patches, is randomlysampled from 20 training images obtained from the web-site http://glcf.umiacs.umd.edu/data/ikonos/index.shtml. ForIKONOS fusion, the weights in (10) are set the same as thosefor QuickBird data since the spectral response of IKONOS is

Fig. 7. Source IKONOS images (river area) and the fused results usingdifferent methods. (a) The resampled low-resolution MS image (RGB, 256 256, 16 m); (b) High-resolution PAN image (256 256, 4 m); (c) Originalhigh-resolution MS image (RGB, 256 256, 4 m); (d) Fused RGB bands bySVT method; (e) Fused RGB bands by GA method; (f) Fused RGB bands bythe proposed method.

Fig. 8. Source IKONOS images (rural area) and the fused results usingdifferent methods. (a) The resampled low-resolution MS image (RGB, 256 256, 16 m); (b) High-resolution PAN image (256 256, 4 m); (c) Originalhigh-resolution MS image (RGB, 256 256, 4 m); (d) Fused RGB bands bySVT method; (e) Fused RGB bands by GA method; (f) Fused RGB bands bythe proposed method.

very similar to QuickBird, and other experimental settings arealso the same as those for QuickBrid case.

Fig. 7(a) shows a resampled low-resolution IKONOS MSimage of one region in Sichuan, China, acquired on May 15,2008. Fig. 7(b) gives the corresponding degraded IKONOShigh-resolution PAN image. The true high-resolution 4-m MSimage is presented in Fig. 7(c). Comparing Fig. 7(b) withFig. 7(c), it can be found that the details of the PAN imageare not as rich as those of the MS image. For example, theriver beach area in the PAN image is darker than the area ofthe MS image. These features make this figure be difficult forpan-sharpening. Fig. 7(d)(f) illustrate the results of differentfusion methods. As shown in Fig. 7(d), the SVT provides thefused image with well-preserved spatial and spectral details.The fused image with the GA-based method preserves spectral


TABLE IIICOMPARISON OF THE PROPOSED ALGORITHM WITH THE EXISTING

METHODS ON IKONOS IMAGES SHOWN IN FIG. 7

TABLE IVCOMPARISON OF THE PROPOSED ALGORITHM WITH THE EXISTING

METHODS ON IKONOS IMAGES SHOWN IN FIG. 8

details well. However, the spatial details in the river area arelost. The fused image obtained by the proposed CS-based ap-proach preserves both spectral and spatial detailed informationwell. In particular, one can see that the details in the mountainareas and the river area are sharply increased. Fig. 8 showsanother IKONOS image of one region in Sichuan, China. Thesimilar conclusions can be drawn from Fig. 8.

The objective measures of the fused images in Figs. 7 and8 are listed in Tables III and IV, respectively. From Table III,we can see that the proposed method performs the best interm of measures CC, RMSE, ERGAS, and Q4. The AWLPmethod provides the best value of SAM followed by the pro-posed method. However, for other quality indexes, the proposedmethod performs better than the AWLP. Therefore, on thewhole, the proposed method provides the best fused results.For the Table IV, the proposed method only loses the SAMquality index. The proposed method performs the best for othermeasures.

V. CONCLUSION

In this paper, a novel pan-sharpening method based on CStechnique is presented. Based on the PAN and MS imagesgeneration model, we referred the pan-sharpening problemas an ill-posed inverse problem inherently. Then, the sparsityregularization is employed to address the ill-posed inverse prob-

lem, and the high-resolution spectral image can be effectivelyrecovered. The proposed method is tested on QuickBird andIKONOS images and compared with six well-known methods:1) GIHS; 2) GS; 3) SVT; 4) CBD; 5) AWLP; and 6) GA-basedmethods. The spectral and spatial information are comprehen-sively evaluated using several image quality measures. Theexperimental results demonstrate the effectiveness of sparsityas a prior for satellite PAN and MS image fusion. In addition,we notice that the proposed method can easily process imagefusion and restoration when the source images are corrupted bynoise by only adjusting the parameter in (11).

However, the proposed scheme takes more time than tradi-tional methods. In the future, we can implement the proposedfusion scheme paralleled on multicore processors, because thetime-consuming BP can be done independently on each patch.And, we should note that, in much imaging situations, theimages may be contaminated by other types of noise, such asthe Poisson noise in low-light-level images. Thus, the setting ofparameter or even the optimal function (11) need to be furtherstudied. For simplicity, only the Gaussian noise is consideredin this paper. In addition, readers should note that the linearrelationship used in this paper is only an approximate model.It is feasible for QuickBird and IKONOS satellites, but needto be modified when applying to other space missions or othermodalities.

ACKNOWLEDGMENT

The authors would like to thank the anonymous reviewersfor their detailed review, valuable comments, and constructivesuggestions.

REFERENCES[1] L. Wald, Some terms of reference in data fusion, IEEE Trans. Geosci.

Remote Sens., vol. 37, no. 3, pp. 11901193, May 1999.[2] C. Thomas, T. Ranchin, L. Wald, and J. Chanussot, Synthesis of mul-

tispectral images to high spatial resolution: A critical review of fusionmethods based on remote sensing physics, IEEE Trans. Geosci. RemoteSens., vol. 46, no. 5, pp. 13011312, May 2008.

[3] B. Aiazzi, S. Baronti, and M. Selva, Improving component substitutionpan-sharpening through multivariate regression of MS +Pan data, IEEETrans. Geosci. Remote Sens., vol. 45, no. 10, pp. 32303239, Oct. 2007.

[4] J. Lee and C. Lee, Fast and efficient panchromatic sharpening, IEEETrans. Geosci. Remote Sens., vol. 48, no. 1, pp. 155163, Jan. 2010.

[5] M. Gonzlez-Audcana, X. Otazu, O. Fors, and J. Alvarez-Mozos, A lowcomputational-cost method to fuse IKONOS images using the spectralresponse function of its sensors, IEEE Trans. Geosci. Remote Sens.,vol. 44, no. 6, pp. 16831691, Jun. 2006.

[6] P. S. Pradhan, R. L. King, N. H. Younan, and D. W. Holcomb, Estima-tion of the number of decomposition levels for a wavelet-based multi-resolution multisensor image fusion, IEEE Trans. Geosci. Remote Sens.,vol. 44, no. 12, pp. 36743686, Dec. 2006.

[7] V. Buntilov and T. R. Bretschneider, A content separation image fusionapproach: Toward conformity between spectral and spatial information,IEEE Trans. Geosci. Remote Sens., vol. 45, no. 10, pp. 32523263,Oct. 2007.

[8] M. Choi, A new intensity-hue-saturation fusion approach to image fusionwith a tradeoff parameter, IEEE Trans. Geosci. Remote Sens., vol. 44,no. 6, pp. 16721682, Jun. 2006.

[9] T.-M. Tu, P. S. Huang, C.-L. Hung, and C.-P. Chang, A fast intensity-hue-saturation fusion technique with spectral adjustment for IKONOSimagery, IEEE Geosci. Remote Sens. Lett., vol. 1, no. 4, pp. 309312,Oct. 2004.

[10] P. S. Chavez, S. C. Sides, and J. A. Anderson, Comparison of three dif-ferent methods to merge multiresolution and multispectral data: Landsat


TM and SPOT panchromatic, Photogramm. Eng. Remote Sens., vol. 57,no. 3, pp. 295303, Mar. 1991.

[11] C. A. Laben and B. V. Brower, Process for enhancing the spatialresolution of multispectral imagery using pan-sharpening, U.S. Patent6 011 875, Jan. 4, 2000, Tech. Rep., Eastman Kodak Company.

[12] T. Ranchin and L. Wald, Fusion of high spatial and spectral resolutionimages: The ARSIS concept and its implementation, Photogramm. Eng.Remote Sens., vol. 66, no. 1, pp. 4961, Jan. 2000.

[13] T. Ranchin, B. Aiazzi, L. Alparone, S. Baronti, and L. Wald, Image fu-sion. The ARSIS concept and some successful implementation schemes,ISPRS J. Photogramm. Remote Sens., vol. 58, no. 1/2, pp. 418, Jun. 2003.

[14] J. Nez, X. Otazu, O. Fors, A. Prades, V. Pal, and R. Arbiol,Multiresolution-based image fusion with additive wavelet decomposi-tion, IEEE Trans. Geosci. Remote Sens., vol. 37, no. 3, pp. 12041211,May 1999.

[15] V. P. Shah, N. H. Younan, and R. L. King, An efficient pan-sharpeningmethod via a combined adaptive PCA approach and contourlets, IEEETrans. Geosci. Remote Sens., vol. 46, no. 5, pp. 13231335, May 2008.

[16] S. Zheng, W. Z. Shi, J. Liu, and J. W. Tian, Remote sensing image fusionusing multiscale mapped LS-SVM, IEEE Trans. Geosci. Remote Sens.,vol. 46, no. 5, pp. 13131322, May 2008.

[17] X. Otazu, M. Gonzlez-Audcana, O. Fors, and J. Nez, Introductionof sensor spectral response into image fusion methods. Application towavelet-based methods, IEEE Trans. Geosci. Remote Sens., vol. 43,no. 10, pp. 23762385, Oct. 2005.

[18] B. Aiazzi, L. Alparone, S. Baronti, and A. Garzelli, Context-driven fu-sion of high spatial and spectral resolution images based on oversampledmultiresolution analysis, IEEE Trans. Geosci. Remote Sens., vol. 40,no. 10, pp. 23002312, Oct. 2002.

[19] A. Garzelli and F. Nencini, Interband structure modeling for pan-sharpening of very high-resolution multispectral images, Inf. Fusion,vol. 6, no. 3, pp. 213224, Sep. 2005.

[20] A. Garzelli and F. Nencini, PAN-sharpening of very high resolution mul-tispectral images using genetic algorithms, Int. J. Remote Sens., vol. 27,no. 15, pp. 32733292, Aug. 2006.

[21] M. V. Joshi, L. Bruzzone, and S. Chaudhuri, A model-based approach tomultiresolution fusion in remotely sensed images, IEEE Trans. Geosci.Remote Sens., vol. 44, no. 9, pp. 25492562, Sep. 2006.

[22] A. Garzelli, F. Nencini, and L. Capobianco, Optimal MMSE pan sharp-ening of very high resolution multispectral images, IEEE Trans. Geosci.Remote Sens., vol. 46, no. 1, pp. 228236, Jan. 2008.

[23] Z. H. Li and H. Leung, Fusion of multispectral and panchromatic imagesusing a restoration-based method, IEEE Trans. Geosci. Remote Sens.,vol. 47, no. 5, pp. 14821491, May 2009.

[24] K. A. Kalpoma and J. I. Kudoh, Image fusion processing for IKONOS1-m color imagery, IEEE Trans. Geosci. Remote Sens., vol. 45, no. 10,pp. 30753086, Oct. 2007.

[25] D. Fasbender, J. Radoux, and P. Bogaert, Bayesian data fusion for adapt-able image pansharpening, IEEE Trans. Geosci. Remote Sens., vol. 46,no. 6, pp. 18471857, Jun. 2008.

[26] Y. F. Zhang, S. D. Backer, and P. Scheunders, Noise-resistant wavelet-based Bayesian fusion of multispectral and hyperspectral images, IEEETrans. Geosci. Remote Sens., vol. 47, no. 11, pp. 38343843, Nov. 2009.

[27] M. Joshi and A. Jalobeanu, MAP estimation for multiresolution fusionin remotely sensed images using an IGMRF prior model, IEEE Trans.Geosci. Remote Sens., vol. 48, no. 3, pp. 12451255, Mar. 2010.

[28] D. L. Donoho, Compressed sensing, IEEE Trans. Inf. Theory, vol. 52,no. 4, pp. 12891306, Apr. 2006.

[29] E. Cands, J. Romberg, and T. Tao, Stable signal recovery from incom-plete and inaccurate information, Commun. Pure Appl. Math., vol. 59,no. 8, pp. 12071233, Feb. 2005.

[30] M. Elad, M. A. T. Figueiredo, and Y. Ma, On the role of sparse andredundant representations in image processing, Proc. IEEE, vol. 98,no. 6, pp. 972982, Jun. 2010.

[31] J. Mairal, M. Elad, and G. Sapiro, Sparse representation for color im-age restoration, IEEE Trans. Image Process., vol. 17, no. 1, pp. 5369,Jan. 2008.

[32] M. Elad and M. Aharon, Image denoising via sparse and redundantrepresentations over learned dictionaries, IEEE Trans. Image Process.,vol. 15, no. 12, pp. 37363745, Dec. 2006.

[33] J. Wright, A. Y. Yang, A. Ganesh, S. S. Sastry, and Y. Ma, Robust facerecognition via sparse representation, IEEE Trans. Pattern Anal. Mach.Intell., vol. 31, no. 2, pp. 210227, Feb. 2009.

[34] J. C. Yang, J. Wright, Y. Ma, and T. Huang, Image super-resolution assparse representation of raw image patches, in Proc. IEEE Int. Conf.Comput. Vis. Pattern Recog., Anchorage, AK, Jun. 2008, pp. 18.

[35] D. Donoho, For most large underdetermined systems of linear equationsthe minimal l1-norm near solution approximates the sparsest solution,Commun. Pure Appl. Math., vol. 59, no. 10, pp. 907934, Jul. 2006.

[36] S. Chen, D. Donoho, and M. Saunders, Atomic decomposition by basispursuit, SIAM Rev., vol. 43, no. 1, pp. 129159, 2001.

[37] L. Wald, T. Ranchin, and M. Mangolini, Fusion of satellite images ofdifferent spatial resolutions: Assessing the quality of resulting images,Photogramm. Eng. Remote Sens., vol. 63, no. 6, pp. 691699, Jun. 1997.

[38] M. M. Khan, L. Alparone, and J. Chanussot, Pansharpening qualityassessment using the modulation transfer functions of instruments, IEEETrans. Geosci. Remote Sens., vol. 47, no. 11, pp. 38803891, Nov. 2009.

[39] L. Alparone, L. Wald, J. Chanussot, C. Thomas, P. Gamba, and L. M.Bruce, Comparison of pansharpening algorithms: Outcome of the 2006GRS-S data-fusion contest, IEEE Trans. Geosci. Remote Sens., vol. 45,no. 10, pp. 30123021, Oct. 2007.

[40] L. Alparone, S. Baronti, A. Garzelli, and F. Nencini, A global qualitymeasurement of pan-sharpened multispectral imagery, IEEE Geosci. Re-mote Sens. Lett., vol. 1, no. 4, pp. 313317, Oct. 2004.

Shutao Li (M07) received the B.S., M.S., and Ph.D.degrees in electrical engineering from Hunan Uni-versity, Changsha, China, in 1995, 1997, and 2001,respectively.

From May 2001 to October 2001, he was a Re-search Associate with the Department of ComputerScience, Hong Kong University of Science and Tech-nology, Clear Water Bay, NT, Hong Kong. FromNovember 2002 to November 2003, he was a Post-doctoral Fellow with the Royal Holloway College,University of London, Egham, U.K. From April

2005 to June 2005, he was a Visiting Professor with the Department ofComputer Science, Hong Kong University of Science and Technology. Heis currently a Full Professor with the College of Electrical and InformationEngineering, Hunan University. He has authored or coauthored more than100 refereed papers. His professional interests are information fusion, patternrecognition, computational intelligence, and image processing.

Dr. Li has won two Second-Grade National Awards at the Science andTechnology Progress of China in 2004 and 2006

Bin Yang received the B.S. degree from ZhengzhouUniversity of Light Industry, Zhengzhou, China, in2005. He is currently working toward the Ph.D.degree in Hunan University, Changsha, China.

His technical interests include image fusion andpattern recognition.

/ColorImageDict > /JPEG2000ColorACSImageDict > /JPEG2000ColorImageDict > /AntiAliasGrayImages false /CropGrayImages true /GrayImageMinResolution 300 /GrayImageMinResolutionPolicy /OK /DownsampleGrayImages true /GrayImageDownsampleType /Bicubic /GrayImageResolution 300 /GrayImageDepth -1 /GrayImageMinDownsampleDepth 2 /GrayImageDownsampleThreshold 1.50000 /EncodeGrayImages true /GrayImageFilter /DCTEncode /AutoFilterGrayImages false /GrayImageAutoFilterStrategy /JPEG /GrayACSImageDict > /GrayImageDict > /JPEG2000GrayACSImageDict > /JPEG2000GrayImageDict > /AntiAliasMonoImages false /CropMonoImages true /MonoImageMinResolution 1200 /MonoImageMinResolutionPolicy /OK /DownsampleMonoImages true /MonoImageDownsampleType /Bicubic /MonoImageResolution 600 /MonoImageDepth -1 /MonoImageDownsampleThreshold 1.50000 /EncodeMonoImages true /MonoImageFilter /CCITTFaxEncode /MonoImageDict > /AllowPSXObjects false /CheckCompliance [ /None ] /PDFX1aCheck false /PDFX3Check false /PDFXCompliantPDFOnly false /PDFXNoTrimBoxError true /PDFXTrimBoxToMediaBoxOffset [ 0.00000 0.00000 0.00000 0.00000 ] /PDFXSetBleedBoxToMediaBox true /PDFXBleedBoxToTrimBoxOffset [ 0.00000 0.00000 0.00000 0.00000 ] /PDFXOutputIntentProfile (None) /PDFXOutputConditionIdentifier () /PDFXOutputCondition () /PDFXRegistryName () /PDFXTrapped /False

/Description > /Namespace [ (Adobe) (Common) (1.0) ] /OtherNamespaces [ > /FormElements false /GenerateStructure false /IncludeBookmarks false /IncludeHyperlinks false /IncludeInteractive false /IncludeLayers false /IncludeProfiles false /MultimediaHandling /UseObjectSettings /Namespace [ (Adobe) (CreativeSuite) (2.0) ] /PDFXOutputIntentProfileSelector /DocumentCMYK /PreserveEditing true /UntaggedCMYKHandling /LeaveUntagged /UntaggedRGBHandling /UseDocumentProfile /UseDocumentBleed false >> ]>> setdistillerparams> setpagedevice

ieee-tgrs-2011_2

Documents

original pan

highresolutionms image

image fusion

multispectralms image

remote sensing applications

remote sensing processand

proposed method

source images