7/29/2019 10.1.1.67.2754

1/4

Improving Image Retrieval Performance by Using Both Color and Texture Features

Dengsheng ZhangGippsland School of Comp. and Info. Tech. Monash University, Churchill, Victoria 3842, Australia

Email: dengsheng.zhang@infotech.monash.edu.au

Abstract

Most content-based image retrievals (CBIR) use color asimage features. However, image retrieval using color featuresoften gives disappointing results because in many cases,images with similar colors do not have similar content. Colormethods incorporating spatial information have beenproposed to solve this problem, however, these methods oftenresult in very high dimensions of features which drasticallyslow down the retrieval speed. In this paper, a methodcombining both color and texture features of image isproposed to improve the retrieval performance. Given aquery, images in the database are firstly ranked using colorfeatures. Then the top ranked images are re-ranked accordingto their texture features. Results show the second processimproves retrieval performance significantly.Keywords: CBIR, color, texture, image retrieval.

1. Introduction

In recent years, Content Based Image Retrieval (CBIR)has been proposed in attempt to index image automaticallyand objectively [1, 2]. In CBIR, images in database arerepresented using such low level image features as color,texture and shape, which are extracted from imagesautomatically. Among these low level features, color featuresare the most widely used features for image retrieval becausecolor is the most intuitive feature and can be extracted fromimages conveniently [3, 4]. However, image retrieval usingcolor features often gives disappointing results, because inmany cases, images with similar colors do not have similarcontent. This is due to the global color features computedoften fails to capture color distributions or textures within the

image. Several methods have been proposed to incorporatespatial color information in attempt to avoid color confusionby machine [5, 6, 7, 8]. However, these methods often resultsin very high dimensions of features which drastically slowdown the retrieval speed. In this paper, we propose a methodcombining both color and texture features to improve retrievalperformance. We compute both the color and texture featuresfrom the images and images in the database are indexed usingboth types of features. During the retrieval process, given aquery image, images in the database are firstly ranked usingcolor features. Then in the second process, a number of topranked images are selected and re-ranked according to theirtexture features. Because the texture features are extractedglobally from the image, they are not an accurate description

of the image in some cases. Therefore, we provide twoalternatives to user, one is the retrieval based on colorfeatures, the other is retrieval based on combined features.When the retrieval based on color fails, the user will use theother alternative which is the combined retrieval. Byintegrating these two alternatives, retrieval performance is

improved significantly.The rest of the paper is organized as following. In

Section 2, the color descriptor is presented. Section 3 brieflydescribes the rotation invariant Gabor texture features. Section4 describes the indexing and retrieval process. Retrievalresults and performance will be reported in Section 5, and thepaper is concluded in Section 6.

2. Robust Color Features

To extract robust color features, we use the perceptuallyweighted histogram or PWH proposed by Lu et al [9].Basically, the PWH is acquired by using CIEL*u*v* colorspace and assigning a perceptual weight to each histogram binaccording to the distance between the color of the pixel andthe color of the bin. We briefly describe the details of PWH inthe following.

The first step to extract color features is to select anappropriate color space. Several color spaces are available,such as RGB, CMYK, HSV and CIEL*u*v*. Most digitalimages are stored in RGB color space. However, RGB colorspace is not perceptually uniform, which implies that twocolors with larger distance can be perceptually more similarthan another two colors with smaller distance, or simply put,the color distance in RGB space does not represent perceptualcolor distance. In view of this drawback, CIEL*u*v* space ischosen because it is an uniform color space in terms of colordistance. In order to use L*u*v* space, color values are firstconverted from RGB space into CIEXYZ space with a lineartransform and then from CIEXYZ space into L*u*v* colorspace using the following transform:

)'(*13*

)'(*13*

008856.0/)/(3.903*

008856.0/16/116*

'

'

3

n

n

nn

nn

vvLu

uuLu

whereYYYYL

YYYYL

=

=

=

>=

)315/(9

)315/(4

)315/(9'

)315/(4'

'

'

nnnnn

nnnnn

ZYXYv

ZYXXu

ZYXYv

ZYXXu

++=

++=

++=

++=

and (Xn,Yn, Zn) is the reference white inXYZspace.There are usually millions of colors in the color space, in

order to build the color histogram from an image, the colorspace is normally quantized into a number of color bins witheach bin represents a number of neighboring colors. However,in the L*u*v* space, representative colors are used instead of

7/29/2019 10.1.1.67.2754

2/4

quantizing each color channel by a constant step. The numberof representative colors is given by the combinations (512) ofthe following three components in L*u*v* space [10]. Theserepresentative colors are uniformly distributed in L*u*v*space.

L component ucomponent vcomponent

6.25 -110.875 -123.62518.75 -66.625 -90.87531.25 -22.375 -58.12543.75 21.875 -25.37556.25 66.125 7.37568.75 110.375 40.12581.25 154.625 72.87593.75 198.875 105.625

In contrast to the conventional histogram building whichassigns the color of each pixel to a single color bin, the PWH

assigns the color of each pixel to 10 neighboring color binsbased on the following weight:

1021/1.../1/1

/1

ddd

dw i

i

+++=

where 20

2

0

2

0)()()(

iiiivvuuLLd ++=

and (L0, u0, v0) is the color of the pixel to be assigned, (Li, ui,vi) is the color of bin i. The use of PWH overcomes thedrawback of conventional histogram methods which would inmany situations assign a pixel color to a bin of a quitedifferent color, e.g., assign a pixel color (6.4, 21, 40) to bin(18.75, 21.875, 40.125). The PWH also overcomes thedrawback of having to assign two quite different colors to asame color bin, e.g., assigning pixel color (6.4, 21, 40) and

(30, 21, 40) to bin (18.75, 21.875, 40.125) in conventionalhistogram building. As the result, PWH is much moreaccurate in representing the image than conventionalhistograms.

3. Rotation Invariant Texture Features

Texture is an important feature of objects in an image.Two images with different content can usually bedistinguished by their texture features even when the imagesshare similar colors.

Gabor filters (or Gabor wavelets) are widely adopted toextract texture features from the images for image retrieval[11]. Basically, Gabor filters are a group of wavelets, with

each wavelet capturing energy at a specific frequency and aspecific direction. Expanding a signal using this basisprovides a localized frequency description, thereforecapturing local features/energy of the signal. Texture featurescan then be extracted from this group of energy distributions.The scale (frequency) and orientation tunable property ofGabor filter makes it especially useful for texture analysis.

For a given image I(x, y) with size PQ, its discreteGabor wavelet transform is given by a convolution:

= =

=K

s

K

tmnmn

tsgtysxIyxG0 0

* ),(),(),(

where, K is the filter mask size, and *mng is the complex

conjugate of gmn which is a class of self-similar functions

generated from dilation and rotation of the following motherwavelet:

)2exp()](

2

1exp[

2

1),(

2

2

2

2

Wxjyx

yxg

yxyx

+=

whereW is called the modulation frequency.After applying Gabor filters on the image with different

orientation at different scale, an array of magnitudes isobtained:

1-,1,0,1;-,1,0,,|),(|),(0 0

NnMmyxGnmEP

x

Q

ymn ===

= =

These magnitudes represent the energy content atdifferent scale and orientation of the image.

The main purpose of texture-based retrieval is to findimages or regions with similar texture. It is assumed that weare interested in images or regions that have homogenoustexture, therefore the mean mn and standard deviation mn ofthe magnitude of the transformed coefficients are used torepresent the homogenous texture feature of the region. Afeature vector f(texture representation) is created using mnand mn as the feature components [11]. Five scales and sixorientations are used in common implementation and theGabor texture feature vector is thus given by: f=(00, 00 ,01 , 01 , , 45, 45). The similarity between two texturepatterns is measured by the Euclidean distance between theirGabor feature vectors. To make the above extracted texturefeatures robust to image rotation, a simple circular shift on thefeature map is used [12].

4. Image Retrieval Using Both Color andTexture Features

In this section, image indexing and retrieval using colorand combined features are described. Each image in thedatabase is indexed using both color and texture featureswhich have been described above. In other words, in theindexed database, images are represented by the indices offeatures. In the retrieval, images in the database, called targetimages, are ranked in descending order of similarity to thequery image. A number of the top ranked images arepresented to the user. The ranking of similarity is determinedby the distance between the feature vector of the query imageand the feature vector of the target image. For two imageswith features (f11, f12, , f1n) and (f21, f22, , f2n) respectively,the distance between the two images is given by

221

22212

22111 )(...)()( nn ffffffd +++=

Here, the fij can be either the color features or the texturefeatures.

Given a query image, the images are first retrieved byusing PWH color features. Although PWH is a more accuratecolor histogram representation of images than conventionalhistogram, it is still a global representation of images. As theresult, the retrieval result given by PWH usually reflects theoverall color tone of the images, rather than the actual imagecontent expected by the user. In this situation, texture featuresof the images can be used to help ranking images with similarcontent closer. However, the texture features captured are alsoglobal features, it may fail to retrieve similar images in some

7/29/2019 10.1.1.67.2754

3/4

situations. Solution to this problem can be segmenting imageinto regions and extracting texture features from the regions.However, segmentation is a complex problem which has nodesirable result at this moment. Therefore, we let the user todecide if texture features is needed. If the color retrieval resultis not satisfactory, the user then selects the retrieval usingcombined features. In the combined retrieval, the images inthe database are first ranked using color features according tothe distance between the query and the target images. Then anumber of the top ranked images are selected and they arethen re-ranked using their texture features.

5. Retrieval Experiments and Results

In the experiment, we use the database which is used byMPEG-7 for natural image retrieval test [13]. The databaseconsists of 5,466 images from large varieties. The types of

images in the database include animals, news photos, sceneryphotos, buildings, cartoons, flowers, landscapes, group ofimages from video sequence etc. Images with differentorientations have also been included.

A Java-based client-server retrieval framework has beendeveloped to conduct the retrieval test. The server side is aJava application which handles online indexing and multi-userquery. The client side is a Java applet which presents retrievalresult and also provides useful user interface such as query bysketch, query by example, selecting query from list andnavigation tools.

To evaluate the retrieval performance, the commonevaluation method, i.e., precision-recall pair, is used.Precision P is defined as the ratio of the number of retrieved

relevant images r to the total number of retrieved images n,i.e. P =r/n. PrecisionP measures the accuracy of the retrieval.Recall R is defined as the ratio of the number of retrievedrelevant images r to the total number mof relevant images inthe whole database, i.e. R = r/m. Recall R measures therobustness of the retrieval.

30 images are randomly selected from the database asqueries, they are shown in Figure 1. For each query, theprecision of the retrieval at each level of the recall is obtained.This is done for all the 30 queries, the 30 retrieval results areaveraged to give the final precision-recall chart of Figure 2.

Among the 30 queries tested, 17 of the retrieval resultshave been improved significantly by using combined features.Some of the retrieval screen shots are shown in Figure 3 to

demonstrate the comparison between retrieval using combinedfeatures and retrieval using color features. The screen shotsare arranged in pairs, each pair is labeled as Ri-ct and Ri-c (i=1, 2, , 6), where Ri-ct stands for retrieval using combinedfeatures and Ri-c for retrieval using color features. As can beobserved, in all the cases demonstrated, the combinedretrievals give much better results than retrievals using colorfeatures only. In some cases, retrievals using combinedfeatures give very good results while the same retrievals usingcolor features fail almost completely, e.g., R2-ct,c, and R4-ct,c. As the choice of retrieval using combined features isdetermine by users, it always helps to improve the retrievalresults in our system. This is different from most of otherapproaches which integrates the two features into one and

provides users with only one alternative, therefore, the overallperformance may not be improved in those approaches.

From the precision-recall chart of Figure 2, it can beobserved that the retrieval precision using color features has asharp drop from 40% of recalls. However, retrieval precisionusing combined features is very robust and has a smooth dropof performance. Overall, the retrieval performance ofcombined features outperforms that of color featuressignificantly.

Figure 1. The 30 random selected query images.

6. Conclusions

In this paper, an image retrieval method using both colorfeatures and combined features has been proposed.Experiment results show, our method outperforms thecommon color feature retrieval significantly. We have usedthe PWH color features to improve conventional histogramfeatures. A robust texture feature which is suitable for imageretrieval has also been presented in the paper. In contrast tomost conventional combined approaches which may not givebetter performance than individual features, our approachprovides users with two alternatives, i.e., retrieval using color

features only and retrieval using combined features. Since wegive the users control to select the type of retrieval, theimprovement of retrieval performance is guaranteed.

In the future, we plan to segment image automaticallyinto homogenous texture regions using split and mergingtechnique. By using regional features instead of globalfeatures, we will be able to improve retrieval performance ofcombined features further. We also plan to use more queriesto test the retrieval performance.

References:[1] G. Lu, Multimedia Database Management Systems, Artech

House, 1999.[2] J. M. Martnez, MPEG-7 Overview (version 7), ISO/IEC

JTC1/SC29/WG11 N4674, Jeju, March 2002.[3] W. Niblack et al. The QBIC Project: Querying Images By

Content Using Color, Texture and Shape. SPIE Conf. OnStorage and Retrieval for Image and Video Databases,Vol.1908, San Jose, CA, pp.173-187, 1993.

[4] M. Swain and D. Ballard, Color Indexing, InternationalJournal of Computer Vision, 7(1):11-32, 1991.

[5] J . R. Simith and S. F. Chang, Querying by Color RegionsUsing the VisualSEEK Content-Based Visual QuerySystem, In M. T. Maybury, ed., Intelligent MultimediaInformation Retrieval, AAAI/MIT Press, 1996.

[6] T. S. Chua, K. L. Tan and B. C. Ooi, Fast Signature-basedColor-spatial Image Retrieval, In Proc. of IEEE

7/29/2019 10.1.1.67.2754

4/4

International Conf. on Multimedia Computing and Systems,pp362-369, Ontario, Canada, 1997.

[7] W. Hsu, T. S. Chua and H. K. Pung, Integrated Color-spatialApproach to Content-based Image Retrieval, In Proc. of

ACM Multimedia, pp305-313, 1995.[8] G. Pass, R. Zabih and J. Miller, Comparing Images Using

Color Coherence Vectors, In Proc. of the 4th ACMMultimedia Conference, pp65-73, 1996.

[9] G. Lu and J. Phillips, Using Perceptually WeightedHistograms for Color-Based Image Retrieval, In Proc. ofInternational Conference on Signal Processing, pp1150-1153, Beijing, China, 1998.

[10] B. Y. Chua, Color-based Image Retrieval IncorporatingRelevance Feedback, Honors Thesis, Monash University,2002.

[11] B. S. Manjunath and W. Y . Ma, Texture Features forBrowsing and Retrieval of Large Image Data, IEEE Trans.PAMI-18 (8):837-842, August, 1996.

[12] D. S. Zhang and G. Lu, Content-based Image RetrievalUsing Gabor Texture Features, In Proc. of First IEEEPacific-Rim Conference on Multimedia (PCM'00), pp.392-395, Sydney, Australia, December 13-15, 2000.

[13] B. S. Manjunath et al, Color and Texture Descriptors,IEEE Transactions on Circuits and Systems for VideoTechnology, 11(6):703-715, 2001.

Precision-Recall Chart

0

10

20

30

40

50

60

70

80

90

100

0 20 40 60 80 100

Recall

Precision

Color-Texture

Color

Figure 2. Average precision-recall of 30 queries.

R1-ct R1-c R2-ct R2-c

R3-ct R3-c R4-ct R4-c

R5-ct R5-c R6-ct R6-c

Figure 3. Sample screen shots of retrievals using color and combined features.