bagged textural and color features for melanoma skin ...keywords: malignant melanoma, skin cancer...

Citation: Alfed, Naser and Khelifi, Fouad (2017) Bagged textural and color features for melanoma skin cancer detection in dermoscopic and standard images. Expert Systems with Applications, 90. pp. 101-110. ISSN 0957-4174

Published by: Elsevier

URL: https://doi.org/10.1016/j.eswa.2017.08.010 <https://doi.org/10.1016/j.eswa.2017.08.010>

This version was downloaded from Northumbria Research Link: http://nrl.northumbria.ac.uk/31833/

Northumbria University has developed Northumbria Research Link (NRL) to enable users to access the University’s research output. Copyright © and moral rights for items on NRL are retained by the individual author(s) and/or other copyright owners. Single copies of full items can be reproduced, displayed or performed, and given to third parties in any format or medium for personal research or study, educational, or not-for-profit purposes without prior permission or charge, provided the authors, title and full bibliographic details are given, as well as a hyperlink and/or URL to the original metadata page. The content must not be changed in any way. Full items must not be sold commercially in any format or medium without formal permission of the copyright holder. The full policy is available online: http://nrl.northumbria.ac.uk/policies.html

This document may differ from the final, published version of the research and has been made available online in accordance with publisher policies. To read and/or cite from the published version of the research, please visit the publisher’s website (a subscription may be required.)

http://nrl.northumbria.ac.uk/policies.html

Bagged Textural and Color Features for Melanoma Skin Cancer Detectionin Dermoscopic and Standard Images

Naser Alfed1 , Fouad Khelifi1

Abstract

Early detection of malignant melanoma skin cancer is crucial for treating the disease and saving lives. Manycomputerized techniques have been reported in the literature to diagnose and classify the disease with satis-factory skin cancer detection performance. However, reducing the false detection rate is still challenging andpreoccupying because false positives trigger the alarm and require intervention by an expert pathologist forfurther examination and screening. In this paper, an automatic skin cancer diagnosis system that combinesdifferent textural and color features is proposed. New textural and color features are used in a bag-of-featuresapproach for efficient and accurate detection. We particularly claim that the Histogram of Gradients (HG)and the Histogram of Lines (HL) are more suitable for the analysis and classification of dermoscopic andstandard skin images than the conventional Histogram of Oriented Gradient (HOG) and the Histogram ofOriented Lines (HOL), respectively. The HG and HL are bagged separately using a codebook for each andthen combined with other bagged color vector angles and Zernike moments to exploit the color information.The overall system has been assessed through intensive experiments using different classifiers on a dermo-scopic image dataset and another standard dataset. Experimental results have shown the superiority of theproposed system over state-of-the-art techniques.

Keywords: Malignant melanoma, skin cancer diagnosis, dermoscopic images, standard skin images, texturaland color features.

1. Introduction

Malignant melanoma is one of the most aggres-sive form of skin cancer and its incidence has beenrapidly increasing over the last few decades, caus-ing the majority of deaths related to skin cancer (Ko-rotkov and Garcia, 2012; Silveira et al., 2009). For-tunately, if melanoma skin cancer is detected at itsearly stages, it can be curative for the patient. How-ever, distinguishing melanoma in its early stagesfrom other pigmented skin lesions is still challeng-ing. Many techniques have been used to predict andclassify melanoma skin cancer. Image processingtools for skin cancer detection usually require pre-processing operations for enhancing images and seg-

1Department of Computer and Information Sciences,Northumbria University at Newcastle, NE2 1XE, UK. E-mail:{naser.alfed, fouad.khelifi}@northumbria.ac.uk

menting the regions of interest to extract efficientfeatures. Dermoscopic and standard images capturedfrom skin usually have some noisy artifacts such asapplied oil and hair and this should be removed priorto segmentation. In this context, Dull Razor medi-cal software was first developed by Lee et al. (1997)to remove hairs from pigmented areas. However, thesystem has been criticized for disrupting the normalskin pattern over the area covered by the hairs (Sheet al., 2006). The fast median filtering was lateradopted to remove noise from the acquired skin im-ages (Tanaka et al., 2004). In (Kiani and Sharafat,2011), an improved version of the Dull Razor med-ical software, called E-shaver, has been proposed.The technique mainly enhances hair detection andremovals by identifying light-colored hairs in addi-tion to dark hairs. Barata et al. (2012) proposed twoimportant steps for reflective artifacts and hair detec-

Preprint submitted to Expert Systems with Applications June 13, 2017

tion and removals using a bank of directional filters.Once a skin lesion image is enhanced via artifactsand hair removal techniques, the lesion area is seg-mented (Lee and Chen, 2015). Many segmentationtechniques have been developed in the literature formelanoma skin cancer diagnosis in dermoscopic im-ages (Silveira et al., 2009). Among these techniques,the region-based approach selects a set of seed pointsand from each point a region grows up if the neigh-bouring pixels have similar properties to that of theseed point. Region growing often generates irregularboundaries and small holes (Zhu and Yuille, 1996). Italso has some drawbacks such as sensitivity to noiseand often results in over segmentation (Tobias andSeara, 2002). In (Nock and Nielsen, 2004), a statis-tical region merging (SRM) algorithm has been pro-posed.One of the common features used by dermatologistsfor diagnosing melanoma skin cancer is the rule-based approach, called ABCD, and based on mor-phological analysis of lesions in dermoscopic im-ages. This is used to distinguish between melanomaskin cancer and non melanomas (Nachbar et al.,1994). It is a medical diagnostic method that isbased on four criteria, i.e., asymmetry, border ir-regularity, color variegation, and different structure.The 7-point checklist is another medical diagnosticmethod that is widely used by researchers and isbased on a set of different characteristics depend-ing on color, shape, and texture (Argenziano et al.,1998). These features can be categorized into colorand texture features. In (Barata et al., 2014a), theauthors investigated the role of color and texture fea-tures for skin melanoma cancer detection and con-cluded that color features are more efficient than tex-ture in dermoscopic images. They also showed thatthe features which are locally extracted from the im-ages bring more information about the lesion thanglobal features. In (Barata et al., 2014b), the authorsdemonstrated, via experiments, that color descriptorsdeliver better performance in detecting melanomaskin lesions than texture descriptors. The authorsin (Barata et al., 2012) adopted a set of directionalfilters and a connected component analysis to extractfive different features for pigment network detectionin dermoscopic images. In (Barata et al., 2013),the role of key-point sampling in a bag of features

approach was investigated. The authors suggestedthat performance of the system can be influencedby the number of detected key-points. In (Barataet al., 2015), color constancy has been explored toovercome the problem of changes that may occurduring the skin image acquisition process. The au-thors in (Riaz et al., 2014) proposed a combinationof texture and color features for the classification ofmelanoma and non-melanoma skin images. A vari-ation of the local binary patterns (LBP) was usedfor the texture features to extract scale adaptive pat-terns. As for the color information, the histogramsof the HSV (Hue, Saturation, Value) color space wasadopted. More recently, Ruela et al. (2015) have ex-plored the importance of shape and symmetry fea-tures in Melanoma diagnosis in order to determinethe type of features that play a crucial role in clas-sification. In (Abuzaghleh et al., 2015), the authorsproposed a combination of Lesion Variation PatternFeatures (LVPF) with some extracted shape, colorand texture features including the pigment networkfeature set, the lesion shape feature, the lesion ori-entation feature, the lesion margin feature, the lesionintensity pattern feature, and the lesion variation pat-tern feature. In (Vasconcelos et al., 2015), color fea-tures have been derived from the ABCD rule wherethe authors proposed a clustering approach to ad-just the system to different datasets and image types.In (Kruk et al., 2015), different texture and statisti-cal features were adopted, including the numericaldescriptors based on the Kolmogorov-Smirnov (KS)statistical distance, the classical Haralick descriptorsand fractal texture analysis-based descriptors. In (Gi-otis et al., 2015), physician annotations for skin le-sions, referred to as visual diagnostic attributes, werecombined with lesion color and lesion texture formelanoma skin detection in non-dermoscopic im-ages. Very recently, three types of features have beenused in (Chakravorty et al., 2016), namely, geome-try features, color features, and finally structural fea-tures. More recently, the authors in (Oliveira et al.,2016) adopted asymmetry, border, color and texturefeatures followed by an SVM classifier for the clas-sification of pigmented skin lesions in macroscopic(standard) images.

In this paper, an automatic skin cancer diagnosissystem that combines different textural and color fea-

2

tures is proposed. New textural and color features areintroduced in a bag of features approach for efficientand accurate skin cancer detection. We particularlyclaim that the Histogram of Gradients (HG) and theHistogram of Lines (HL) are more suitable for theanalysis and classification of dermoscopic and stan-dard skin images than the conventional Histogramof Oriented Gradient (HOG) and the Histogram ofOriented Lines (HOL), respectively. This is becausethe orientation of melanoma edges and texture isnot a discriminating feature when compared to nonmelanoma lesions. Therefore, the use of edge andline orientation in skin images reduces the inter classdissimilarity, and this causes an adversary effect onclassification. The HG and HL are bagged sepa-rately using a codebook for each and then combinedwith other bagged Color Vector Angles (CVA) andZernike moments to exploit the color information.Experimental results demonstrate the efficiency ofthe proposed texture and color features as well as thesuperiority of the overall system over state-of-the-art melanoma skin cancer detection techniques. Therest of the paper is organized as follows. Section 2describes the proposed system and the features ex-tracted whereas section 3 provides a discussion of ex-perimental results obtained. Conclusions are drawnin section 4.

2. Proposed system

The proposed system consists of six main stages atthe training phase, namely: preprocessing, key-pointdetection, segmentation, patch extraction and selec-tion (region of interest), feature extraction, codebookgeneration, histogram building, features concatena-tion, and classification. Fig. 1 illustrates the pro-posed approach.

2.1. Pre-processing

For efficient feature extraction, a process of imageenhancement is first conducted. The reason for thisis that the original acquired images may have someair bubbles and artifacts caused by gel applied be-fore the capture of the images in addition to hairsand other noise. The same procedures that have beenimplemented in (Alfed et al., 2015) which is basedon method proposed by Barata et al. (2012) which

Extract

Histogram of

Gradients

(HG)

Extract

Histogram of

Lines (HL)

Extract Colour

Vector Angles

(CVA) &

Zernike

Moments (ZM)

Code book

generation

for HG

Code book

generation

for HL

Code book

generation for

CVA combined

with ZM

Building

histograms

for each

image using

HG codebook

Building

histograms

for each

image using

HG codebook

Building

histograms

using CVA&ZM

codebookConcatenation

Training of a classifier

Patch extraction and selection

Segmentation

Key-point

detection (SIFT)

Enhanced image

Parameters of the

trained classifier

Training images

Descriptor

Figure 1: The training phase of the proposed system

consists of detecting and removing two undesirablepatterns: reflective artifacts and hair. To detect ar-tifacts reflection, a simple thresholding algorithm isadopted where every pixel with a certain brightnessin the image is compared against its neighbourhood.If the pixel of interest is distinguished from its neigh-bourhood with a clearly larger luminance, the pixelis said to be an artifact reflection. Once artifacts aredetected, an inpainting operation is applied accord-ingly. Fig. 2 illustrates the process of converting theoriginal color image (a) into a grey-scale image (b)and then removal of artifacts is obtained in (c).

(a) (b) (c)

Figure 2: Pre-processing for artifacts removal; (a):original im-age, (b):grey-scale image, (c ):after artifacts removal

Hair detection and removal is another big chal-lenge especially when hair lines are not accuratelydetected. The directional Gabor filters are first ap-

3

plied and then followed by finding the local max-imum at each pixel location. The parameters ofthe Gaussian filters used here have been adoptedfrom (Barata et al., 2012). The hair detection algo-rithm uses a bank of 64 directional filters to performthe detection. Again, once hair is detected, an in-painting operation is applied on the correspondingarea. Fig. 3 illustrates hair detection and removal us-ing the directional Gabor filers.

(a) (b) (c)

Figure 3: Pre-processing for artefacts and hair removal; (a):original image, (b): blue component image, (c ): after hair de-tection and removal

2.2. Key-points detectionLowe (2004) presented an efficient method for de-

tecting key points from images that are invariant toimage scale and rotation to perform reliable match-ing between different views of an object. The pro-posed Lowe’s approach was named the Scale In-variant Feature Transform (SIFT). The SIFT methodcan be used for transforming image data into scale-invariant coordinates relative to local features. Itgenerates a good number of features over the wholerange of scales and locations that fully describe theimage (Lowe, 1999). Cascade filtering is widelyused for Key-points detection. It uses efficient algo-rithms to identify the locations of the feature-pointcandidates, subsequently; these candidates are sub-jected to further analysis. Thus, searching for stablekey-points through all continuous possible scales isan important step to determine the locations, whichare known as scale invariant features. First, denoteby L(x, y, σ) the convolution of an image I(x, y) witha Gaussian kernel G(x, y, σ) at scale SIGMA as:

L(x, y, σ) = G(x, y, σ) ∗ I(x, y) (1)

where ∗ denotes the convolution process and theGaussian operator is defined by:

G(x, y, σ) =1

2πσ2 e−(x2+y2)/2σ2(2)

With the standard deviation σ of the Gaussian distri-bution. Lowe proposed searching for the scale-spaceextrema in a difference-of-Gaussian function (DoG)to detect stable key-point locations efficiently. DoGcan be computed by finding the differences of the ad-jacent scales of image I and separated by a constantmultiplicative factor k which is defined as:

D(x, y, σ) = (G(x, y, kσ) −G(x, y, σ)) ∗ I(x, y) (3)

D(x, y, σ) = L(x, y, kσ) − L(x, y, σ) (4)

The detection of local extrema in the DoG image canbe obtained by comparing each sample pixel to its26 neighbours: 8 at the same image scale and 9 pix-els at the above and below immediately scales (seeFig. 4). If the pixel value is the maximum or mini-mum among all compared pixels, it is selected as acandidate key-point.

Figure 4: Key-point detection, Copyright OpenCV-Python

Once candidate key points are detected, the keypoints with low contrast are discarded to maintainonly stable points. This is performed by comparingthe second-order Taylor expansion of the DoG at thecorresponding key points location against a certainthreshold.

2.3. Segmentation

In this work, it is meant by segmentation the detec-tion of the lesion area in a skin image. Here, the Sta-tistical Region Merging (SRM) algorithm has beenadopted. This is an unsupervised learning approachfor border detection proposed by Nock and Nielsen(2004). This method has become well known andis widely used in the segmentation process due to

4

its efficiency, simplicity, and accurate performancewithout the use of quantization or colour space trans-formations. The SRM algorithm is used to evaluatethe pixel values within a regional area and groupedthem based on homogeneous properties attaining asmaller list. Two important components that to de-fine this algorithm are merging (test) and order inmerging. It basically accumulates couples of adja-cent pixels in the image I to be in one set S I . Thesecouples are then sorted in an ascending order of areal function f(p, p′), with p and p′ being pixels of Iand traversing this order only once. The current re-gion to which p belongs to, is taken as the test. Thenthe R(p) and R(p′) are merged if the correspondingfunction f returns true. Fig. 5 illustrates skin lesiondetection from a dermoscopic image.

Figure 5: Image lesion detection (ROI). First column: Originalimages. Second column: Skin lesion.

2.4. Patches extraction and selectionIn this section, the patches are extracted based on

detected key-points. Each of these key-points is to beconsidered as the center of a candidate square patch.The selection of only the patches with areas of morethan 50% inside the segmented lesion is performed.The selected patches that are extracted from the en-hanced image have been used for the next step of ex-tracting local features.

2.5. Feature extractionIn this work, a number of efficient features have

been extracted from each selected patch in order todescribe skin lesions efficiently. This consists of theHG, HL, CVA, and the 3rd order Zernike moments.As will be demonstrated later, HG and HL outper-form the conventional HOG and HOL, respectively.

2.5.1. Histogram of Gradients (HG)Lowe (2004) has first used a variant of the his-

togram of oriented gradients (HOG) as a descriptorwith orientation alignment. Dalal and Triggs (2005)

later proposed the HOG descriptor for pedestriandetection. This has then been widely applied inobject detection and image recognition applications.The rational behind the HOG descriptor is that localfeatures in an image can be described by the dis-tribution of intensity gradients and edge directions.In this work, however, we ignore the orientation ofedges and texture and propose to use the histogramof gradients for each selected patch. The idea isbasically to compute the gradient magnitude m(x, y)for each patch as

∂xI = I(x + 1, y) − I(x − 1, y) (5)

∂yI = I(x, y + 1) − I(x, y − 1) (6)

m(x, y) =

√(∂xI)2 + (∂yI)2 (7)

Recall that a skin image is divided into a numberof patches. Each patch can be described by a HGaccordingly where the boundaries used to calculatethe histograms are determined by the minimum andmaximum values of all the gradient magnitudes ofthe image. Then, each HG is L2-normalized. Weemphasize here that the gradient orientation is ne-glected because it does not bring any useful infor-mation on dermoscopic and standard skin images.Indeed, regardless of the lesion type and class, itsgradient orientation cannot be discriminative as illus-trated by Fig. 6 and Fig. 7 (see third column), respec-tively. As a result, the use of the gradient orientationmay reduce the inter class dissimilarity and, hence,deteriorate the classification performance.

Figure 6: Gradient magnitude and orientation for normal andabnormal dermoscopic images. First row: Normal images. Sec-ond row: Abnormal images. First column: Original image.Second column: Gradient magnitude. Third column: Gradientorientation.

5

Figure 7: Gradient magnitude and orientation for normal andabnormal standard skin images. First row: Normal images.Second row: Abnormal images. First column: Original image.Second column: Gradient magnitude. Third column: Gradientorientation.

On the other hand, the gradient magnitude can dis-criminate normal lesions from abnormal ones in thesense that skin cancer lesions are less textured thannormal lesions (see the second column of Fig. 6 andFig. 7). In Fig. 8, the HOG and HG of a block ofmelanoma and non melanoma lesion are illustrated.The corresponding Chi-square measures between thehistograms of melanoma and non melanoma blocksare depicted in Table 1 for both HG and HOG. As canbe seen, the dissimilarity with HG is more significantthan that with HOG.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 150

0.05

0.1

0.15

0.2

0.25

1 2 3 4 5 6 7 8 9 10 11 12 13 14 150

0.05

0.1

0.15

0.2

0.25

1 2 3 4 5 6 7 8 9 10 11 12 13 14 150

0.05

0.1

0.15

0.2

0.25

1 2 3 4 5 6 7 8 9 10 11 12 13 14 150

0.05

0.1

0.15

0.2

0.25

Figure 8: HOG and HG for normal and abnormal dermoscopicimages. First row: Normal image. Second row: Abnormalimage. First column: HOG. Second column: HG.

Table 1: Difference of HOG and HG between a skin cancer le-sion and another normal lesion in terms of the Chi-square mea-sure.

Histogram HOG HGChi-square 0.34 0.45

2.5.2. Histogram of Lines (HL)

The Histogram of Oriented Lines (HOL) was ini-tially adopted in (Jia et al., 2014) for palm-print iden-tification. It basically uses the same approach as theHOG but computes the directed lines in the imageinstead of gradients. Practically, the summation ofimage pixels over a certain set of lines can be ob-tained via the finite Radon transform (FRAT) (Doand Vetterli, 2003). Indeed, in a textured image, aline can be detected by the FRAT from the neighbor-hood of each pixel in a small local area. However,because the FRAT treats the input image as a peri-odic signal, Huang et al. (2008) proposed a ModifiedFinite Radon transform (MFRAT) to highlight linepatterns effectively through a filtering-like process.The authors of (Jia et al., 2014) used the MFRAT im-age to generate a HOL descriptor for palm-prints byadopting the same steps of the HOG descriptor. Onebeneficial property of the HOL is that it has good in-variance to changes of illumination than the HOG.Moreover, the HOL is robust against transformationsbecause slight translations and rotations lead to onlysmall histogram value changes (Jia et al., 2014).

The MFRAT of a real function f (x,y) on the finitegrid Z2

p centered at (x0, y0) is defined as

r(Lk(x0, y0)) =∑

x,yεLk

f (x, y) (8)

where f (x, y) is the pixel value located in (x, y) andLk denotes the set of points that make up a line onthe lattice Z2

p, which means that:

Lk(x0, y0) = {(x, y) : y = k(x−x0)+y0, (x, y)εZ2p} (9)

where (x0, y0) denotes the centre point of the latticeZ2

p and k is the corresponding slope of Lk. If the cen-tre of lattice Z2

p moves over an image pixel by pixel(or pixels by pixels), the energies and directions ofall pixels are calculated. Two new images, directionand magnitude images respectively can be created,if the values of all pixels for an image I(x, y) of sizen×m are replaced by their directions and energies. Inthis work, we compute the energy function for eachpixel at (x0, y0) as

E(x0, y0) = min(r(Lk(x0, y0)) (10)6

The equation below show how the new energy im-ages can be created (Huang et al., 2008):

FE =

∣∣∣∣∣∣∣∣∣∣∣E(1, 1) E(1, 2) ... E(1,m)E(2, 1) E(2, 2) ... E(2,m)... ... ... ...

E(n, 1) E(n, 2) ... E(n,m)

∣∣∣∣∣∣∣∣∣∣∣ , (11)

where FE is the energy matrix. The same steps forcomputing the histogram of gradients will be usedhere for computing the histogram of lines, denotedby HL. The HL extracted from each patch is adoptedas a feature vector instead of HOL. Note that the lineorientation is neglected because it does not bring anyuseful information on dermoscopic and standard skinimages as discussed earlier in section 2.5.1. Obvi-ously, as can be seen in Fig. 9 the dissimilarity withHL is better than that with HOL for normal and ab-normal images.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 150

0.05

0.1

0.15

0.2

0.25

1 2 3 4 5 6 7 8 9 10 11 12 13 14 150

0.05

0.1

0.15

0.2

0.25

1 2 3 4 5 6 7 8 9 10 11 12 13 14 150

0.05

0.1

0.15

0.2

0.25

1 2 3 4 5 6 7 8 9 10 11 12 13 14 150

0.05

0.1

0.15

0.2

0.25

Figure 9: HOL and HL for normal and abnormal dermoscopicimages. First row: Normal image. Second row: Abnormalimage. First column: HOL. Second column: HL.

Furthermore, the analysis of the two previous le-sions using the Chi-Square measure has been in-cluded in Table 2. As shown, the dissimilarity withHL is more significant than that with HOL.

Table 2: Difference of HOL and HL between a skin cancer le-sion and another normal lesion in terms of the Chi-square mea-sure.

Histogram HOL HLChi-square 0.31 0.55

2.5.3. Color vector angle (CVA)Vector angle is effective in evaluating color con-

trast and can be used for extracting color features.Color Vector Angles (CVA) have been widely usedin edge detection, image hashing and image re-trieval (Tang et al., 2014). CVA is used for identify-ing color edges because it is insensitive to variationsin intensity while being sensitive to differences inhue and saturation (Dony and Wesolkowski, 1999).In this work, CVA is adopted in extracting color fea-tures from dermoscopic and standard skin images.As angle calculation needs two colors, a referencecolor is generated from Pre f = [Rre f ,Gre f , Bre f ]T ,where Rre f , Gre f and Bre f are the means of thered, green and blue components of all pixels. There-fore, for each pixel, the color vector angle betweenits RGB vector and Pre f is calculated. The advan-tage of color vector angle is attributed to its sensi-tivity to hue differences. Let P1 = [R1,G1, B1]T andP2 = [R2,G2, B2]T be the vectors of two colors. Theangle ∆ between two pixels P1 and P2 can be com-puted by

∆ = arcsin

1 −

(PT

1 P2

)2

PT1 P1PT

2 P2

12 (12)

where arcsin(.) is the operation arcsin. Once a ma-trix of angles ∆ is extracted from an image patch, ahistogram is constructed using a certain number ofbins. This constitutes the feature vector correspond-ing to the CVA.

2.5.4. Zernike momentsZernike moments have been used in shape-based

image retrieval and edge detection as they are essen-tially retrieved from geometric moments by replac-ing the conventional transform kernel with orthog-onal Zernike polynomials (Khotanzad and Hong,1990). In general, orthogonal moments are bet-ter than other types of moments in terms of in-formation redundancy and image representation andthey have important advantages such as the rota-tional invariance, robustness to noise and expres-sion efficiency (Zhao et al., 2013). In fact, becauseZernike polynomials are orthogonal to each other,Zernike moments can represent the properties of an

7

image with no redundancy or overlap of information.Therefore, a skin lesion can be comprehensively de-scribed by the Zernike moments. Zernike Moments(ZM) with different orders have been implemented inthis work according to the following equation. Thebest performance is obtained when using the 3rd or-der moment, and therefore it has been used here.Zernike Moments are defined as:

Zα,β =α + 1π

∑(p,θ)ε unit disk

∑I (p, θ) V∗α,β (p, θ) (13)

where Vα,β(p, θ) is a Zernike polynomial of order αand repetition β and the symbol ∗ denotes the com-plex conjugate. The form of these polynomials isgiven as (Zhao et al., 2013)

Vα,β (p, θ) = Rα,β (p) e jβθ (14)

where α is either a positive integer or zero and βis subject to the constraint that α − |β| is even, and0 ≤ |β| ≤ α, and Rα,β (p) are the radial polynomi-als (Zhao et al., 2013). Table 3 lists for each or-der α the corresponding number of moments whichwere experimentally tested. Five Zernike momentsat the third order are extracted from each color planeof a patch in the RGB color representation. The re-spective set of features consists of the L2-normalizedfifteen Zernike moments (e.g. five from each colorchannel).

Table 3: List of Zernike moments up to order threeα Moments No. of moments1 Z1,1 12 Z2,0,Z2,2 23 Z3,1,Z3,3 2

2.6. Codebook generationAt the training stage, a large number of feature

vectors, where each is extracted from a patch ofa training image, are used to generate a codebook.This will then serve as a dictionary to represent eachtest image as a histogram since each patch corre-sponds to a codeword. Three codebooks have beengenerated for each type of features, i.e. HG, HL, andCVA+ZM. The first codebook is for HG features, thesecond one is for the HL features, and the third code-book is generated using a concatenation of CVA and

the 3rd order Zernike Moments (ZM). The K-meansmethod is adopted in this work for generating thecentroids (codewords) in each codebook. The ideaof codebook generation is that each feature vectorextracted from a patch of any training image is usedto form a codebook via K-means with a certain num-ber of centroids (also called codewords).

2.7. Building histograms and concatenation

Once the codebooks are generated, each patch ofan image is assigned a label corresponding to theclosest centroid according to the Euclidean distance.For each codebook, the image is described by thefrequency of labels in the form of a histogram. Asmentioned earlier, this applies to the three types offeatures. Therefore, the final descriptor Fd is a con-catenation of three histograms as

Fd = [CHG,CHL,CCVA+ZM] (15)

where CHG, CHL, and CCVA+ZM represent the his-togram of gradients, the histogram of lines, and thehistogram of CVA and ZM corresponding to their re-spective features, accordingly.

2.8. Classification

Given a number of descriptors, Fd, obtained frommedically annotated images, a binary classifier istrained on two classes, i.e., melanoma and nonmelanoma.

Extract

Histogram

of Gradients

(HG)

Extract

Histogram

of Lines

(HL)

Extract Colour

Vector Angles

(CVA) &

Zernike

Moments (ZM)

Building

histograms

using HG

codebook

Building

histograms

using HG

codebook

Building

histograms

using CVA&ZM

codebookConcatenation

Classification

?

No

n-m

ela

no

ma Melanoma

Patch extraction and selection

Segmentation

Key point

detection (SIFT)

Enhanced image

Testing image

Descriptor

Figure 10: The testing phase of the proposed system

8

Once trained, the classifier can be used formelanoma skin cancer diagnosis. Once the code-books are generated and the classifier is trained, thetesting phase follows the same steps as shown inFig. 10. When a test image is presented to the sys-tem, it is first subjected to the pre-processing stage,segmentation, path extraction and selection, and fea-ture extraction prior to the obtention of the descriptorvia the constructed codebooks. The descriptor is thenpresented to the trained classifier in order to detectthe presence of melanoma skin cancer.

3. Experimental results

In our experiments, five-fold cross-validationis carried out for training and testing the pro-posed system. A dataset of 200 medically an-notated dermoscopic images (40 melanomas and160 non-melanomas) with a truth table obtainedfrom the database of the Hospital Pedro Hispano(HPH) (Mendonc et al., 2015) has been used herefor classification purposes. The system has alsobeen tested on the Dermofit database where 236 stan-dard images were used (76 melanomas and 160 non-melanomas). This has been obtained from the Uni-versity of Edinburgh (Ballerini et al., 2013). Unlessotherwise stated, the setting used in our experimentsfor the process of feature and descriptor extraction islisted in Table 4.

Table 4: Parameters setting of the systemFeatures Parameter Value

codebook size 100CVA patch size 30 × 30

bins per patch 15codebook size 100

ZM patch size 30 × 30moments per patch 15

codebook size 100HG patch size 120 × 120

bins per patch 15codebook size 100

HL patch size 120 × 120bins per patch 15

It is worth mentioning that only the patches whosearea covers more than 50% of the lesion were se-lected for feature extraction. The measures used inour experiments consist of the False Positive rate

(FP), the False Negative rate (FN), Sensitivity (SE),Specificity (SP), and Accuracy (Acc).

3.1. System analysis

In the first set of experiments, the combination ofcolor moments and color histograms, as proposedin (Barata et al., 2014a,b), have been assessed in thebag of features approach as opposed to our combinedCVA+ZM features. This is to illustrate the contribu-tion of the proposed color features and Zernike mo-ments. To this end, three different classifiers wereused, namely the SVM, AdaBoost, and an ArtificialNeural Network (ANN) with 120 neurons in the hid-den layer and the Radial Basis Function (RBF) as anactivation function. The results on the dermoscopyHPH dataset are depicted in Tables 5 and 6.

Table 5: Comparisons of color featuresCRGB+M COpp+M

FN FP SE% SP% FN FP SE% SP%SVM 0.21 0.35 78.75 65.33 0.16 0.31 84.29 69.21AdaBoost 0.14 0.39 86.26 60.67 0.11 0.35 89.36 64.62ANN (RBF) 0.11 0.28 89.45 71.89 0.09 0.29 91.39 70.51

Table 6: Comparisons of color featuresCCVA+ZM

FN FP SE% SP%SVM 0.15 0.32 85 68.39AdaBoost 0.09 0.33 90.96 67.29ANN (RBF) 0.06 0.14 93.73 86.29

As can be seen, the proposed CVA+ZM featuresappear more powerful than the conventional combi-nation of color histograms (using the Opponent colormodel) and color moments (using the first, secondand third order statistical moments). Next, the per-formance of the Histogram of Gradients (HG) andthe Histogram of Lines (HL) is evaluated in compar-ison to the HOG and HOL, respectively on the samedataset. This is to verify our claim on the negative af-fect of texture and edge orientation in the bag of fea-tures approach. Note that the orientation of the gra-dient was used in exiting techniques such as (Barataet al., 2014a,b; Alfed et al., 2016). The classificationresults are shown in Tables 7 and 8, respectively.

9

Table 7: Comparisons of HOG and HGCHOG CHG


Table 8: Comparisons of HOL and HLCHOL CHL


It is clear that the orientation of gradients and linesin HOG and HOL, respectively, decreases the perfor-mance when compared to HG and HL. This can alsobe noticed in Table 9 where the HOG was combinedwith HOL and compared to the other combination ofHG and HL. Interestingly, one can observe that thecombination of HG and HL enhances the accuracy.

Table 9: Combination of texture features[CHOG,CHOL] [CHG,CHL]

FN FP SE% SP% FN FP SE% SP%SVM 0.1 0.22 90.43 77.56 0.08 0.23 92.41 77AdaBoost 0.07 0.18 93.22 82.1 0.04 0.18 95.6 82.43ANN (RBF) 0.04 0.1 95.76 90.14 0.02 0.08 97.5 91.51

To gauge the performance of the complete system,the previously assessed features, i.e., CVA+ZM, HGand HL are combined as proposed in (15). The cor-responding results are depicted in Table 10 where acombination of conventional texture and color fea-tures is also considered.

Table 10: Combinations of texture and color features[CHOG,CHOL,COpp+M] [CHG,CHL,CCVA+ZM]


Obviously, the proposed combination of featuressignificantly outperforms the competing one. This isadopted in the rest of the paper for further analysisand comparisons with state-of-the-art systems.

3.2. Comparison with existing systemsFor comparison purposes, the authors have repro-

duced a number of melanoma skin cancer detec-tion systems that used the bag of features approach,

in particular (Barata et al., 2014a), (Barata et al.,2014b), (Alfed et al., 2015) and (Alfed et al., 2016).Furthermore, since the HPH dataset has been widelyused in the literature with the same settings, we havecollected the results of other related systems as re-ported in their original papers on 200 dermoscopyimages. The reported techniques are listed in Ta-ble 11.

Table 11: List of acronyms used for describing existing tech-niques and strategies

Existing techniques AcronymPigment Network and Bank of directional Filters PNBF (Barata et al., 2012)Sparse key-point sampling using Harris Laplace HLaplace (Barata et al., 2013)Color Moments CM (Barata et al., 2014b)Local color-based features LCB (Barata et al., 2014a)Color Constancy (shades of gray) CC (Barata et al., 2015)Scale Adaptive Local Binary Patterns LBP − based (Riaz et al., 2014)Combination of shape, color and texture features SCT (Abuzaghleh et al., 2015)Mean color vector-based symmetry features MCV (Ruela et al., 2015)Pigment network-based Standard deviations PNStds (Alfed et al., 2015)Detection of Pigment Networks DPN (Eltayef et al., 2017)

Table 12 lists the results of the aforementionedtechniques on the HPH dataset (Mendonc et al.,2015). Note that local features normally yield bet-ter performance than global features as illustratedin (Barata et al., 2014a). It is also worth mentioningthat the combination of color and texture plays a ma-jor role in melanoma skin lesion diagnosis. This canbe justified by the low performance of the systemsthat used color features only, in particular (Barataet al., 2015) and (Barata et al., 2014b) or, on the otherhand, the systems that used texture features only suchas in (Barata et al., 2012) and (Riaz et al., 2014). Ob-viously, the performance has been improved whencolor, shape and texture features are combined suchas in (Abuzaghleh et al., 2015) or even by combin-ing color and texture features such as the system pre-sented in this work.

Table 12: Performance of existing techniques on the Der-moscopy HPH image database

Technique FN FP SE% SP% Acc%PNBF (Barata et al., 2012) 0.09 0.18 91.1 82.1 86.6HLaplace (Barata et al., 2013) 0.02 0.14 98 86 92CM (Barata et al., 2014b) 0.07 0.12 93 88 90.5LCB (Barata et al., 2014a) 0 0.25 100 75 87.5CC (Barata et al., 2015) 0.07 0.24 92.5 76.3 84.4LBP − based (Riaz et al., 2014) 0.16 0.06 84 94 89SCT (Abuzaghleh et al., 2015) 0 0.08 100 91.5 95.75MCV (Ruela et al., 2015) 0.04 0.17 96 83 89.5PNStds (Alfed et al., 2015) 0.05 0.08 95.45 92.33 93.89[CHOG,COpp+M] (Alfed et al., 2016) 0.09 0.15 91 85 88DPN (Eltayef et al., 2017) 0.08 0.05 92.3 95.0 90.0This work 0.01 0.02 99.41 98.18 98.79

10

Overall, the proposed system delivers the thirdlowest score on the HPH dataset in terms of FN,slightly outperformed by SCT and LCB that seemto detect all test cancer images. However, thesetwo systems perform poorly in terms of FP wherea good number of false skin cancer detections werenoticed. As a consequence, the proposed systemappears more reliable in terms of the overall accu-racy, beating the closest competitor by 3% approxi-mately. Finally, 256 standard images have been usedfrom the Dermofit dataset, namely 76 of malignantmelanoma and 180 non-melanoma images. The non-melanoma images have been selected from 9 differ-ent types of lesions as depicted in Table 13.

Table 13: Categories of lesions and number of selected imagesused in the Dermofit database

Dermofit databaseLesion Type Total images Selected images

Actinic Keratosis 45 20Basal Cell Carcinoma 239 20Melanocytic Nevus (mole) 231 20Seborrhoeic Keratosis 257 20Squamous Cell Carcinoma 88 20Intraepithelial Carcinoma 78 20Pyogenic Granuloma 24 20Haemangioma 97 20Dermatofibroma 65 20Malignant Melanoma 76 76

In Table 14, the performance is displayed for theproposed system as well as the competing techniquesthat use a bag of features approach.

Table 14: Performance of existing techniques using standardimages from the Dermofit database

FN FP SE% SP% Acc%LCB (Barata et al., 2014a) 0.04 0.28 95.59 72.02 88.27CM (Barata et al., 2014b) 0.05 0.30 94.62 70.49 87.11PNStds (Alfed et al., 2015) 0.08 0.27 92.23 73.28 85.94[CHOG,COpp+M] (Alfed et al., 2016) 0.19 0.29 81.32 70.86 78.13This work 0.04 0.15 96.04 84.78 92.96

For the Dermofit standard database, the proposedsystem clearly has the upper hand in every aspectand outperforms the competing techniques by morethan 4.5% of overall accuracy. It is also worth notingthat standard images appear less efficient than der-moscopic images for melanoma skin cancer diagno-sis as can be seen from the drop in performance. Thiscan be justified by the fact that dermoscopic images

are characterised by more details and texture, hencemore discriminative features, than standard images.

4. Conclusion

In this paper, a melanoma skin cancer detectionsystem has been presented. The system relies ona bag of features approach using multiple code-books where new color and textural features are pro-posed for describing skin cancer lesions efficiently,namely, the histogram of gradients (HG), the his-togram of lines (HL), the 3rd order Zernike moments,and color vector angles. It has been claimed anddemonstrated that the orientation information in theconventional histogram of oriented gradients (HOG)and the histogram of oriented lines (HOL) reducesthe inter class dissimilarity and consequently de-creases the discriminative nature of the respectivefeatures in skin cancer detection. The histogram ofgradients and the histogram of lines are used sepa-rately to create two respective codebooks whereasthe color vector angles and the 3rd order Zernikemoments are combined for the creation of a thirdcodebook. The automated skin cancer detection sys-tem has been assessed on two different datasets,namely the HPH dermoscopy database and the Der-mofit standard database. Results have shown thatthe proposed system can reach 98.79% of overall ac-curacy on HPH and 92.96% on the standard Der-mofit. Compared with related state-of-the-art tech-niques, the system outperforms its closest competitorby, approximately, 3% on HPH and 4% on Dermofit.

References

Abuzaghleh, O., Barkana, B. D., and Faezipour, M. (2015).Noninvasive real-time automated skin lesion analysis systemfor melanoma early detection and prevention. IEEE Journalof Translational Engineering in Health and Medicine, 3:1–12.

Alfed, N., Khelifi, F., and Bouridane, A. (2016). Improvinga bag of words approach for skin cancer detection in der-moscopic images. In Proc. IEEE International Conferenceon Decision and Information Technologies (CoDIT), pages228–232, Saint Julian’s, Malta.

Alfed, N., Khelifi, F., Bouridane, A., and Seker, H. (2015). Pig-ment network-based skin cancer detection. In Proc. IEEEInternational Conference on Engineering in Medicine andBiology Society (EMBS), pages 7214–7217, Milan, Italy.

11

Argenziano, G., Fabbrocini, G., Carli, P., Giorgi, V. D., Sam-marco, E., and Delfino, M. (1998). Epiluminescence mi-croscopy for the diagnosis of doubtful melanocytic skin le-sions: comparison of the ABCD rule of dermatoscopy and anew 7-point checklist based on pattern analysis. Archives ofDermatology, 134(12):1563–1570.

Ballerini, L., Fisher, R. B., Aldridge, B., and Rees, J. (2013).A color and texture based hierarchical k-nn approach to theclassification of non-melanoma skin lesions. In Color Med-ical Image Analysis, pages 63–86. Springer.

Barata, C., Celebi, M. E., and Marques, J. S. (2015). Im-proving dermoscopy image classification using color con-stancy. IEEE Journal of Biomedical and Health Informatics,19(3):1146–1152.

Barata, C., Marques, J. S., and Rozeira, J. (2012). A systemfor the detection of pigment network in dermoscopy imagesusing directional filters. IEEE Transactions on BiomedicalEngineering, 59(10):2744–2754.

Barata, C., Marques, J. S., and Rozeira, J. (2013). The roleof keypoint sampling on the classification of melanomas indermoscopy images using bag-of-features. In Proc. SpringerIberian Conference on Pattern Recognition and Image Anal-ysis (IbPRIA), pages 715–723, Madeira, Portugal.

Barata, C., Ruela, M., Francisco, M., Mendonca, T., and Mar-ques, J. S. (2014a). Two systems for the detection ofmelanomas in dermoscopy images using texture and colorfeatures. IEEE Systems Journal, 8(3):965–979.

Barata, C., Ruela, M., Mendonca, T., and Marques, J. S.(2014b). A bag-of-features approach for the classificationof melanomas in dermoscopy images: the role of color andtexture descriptors. In Computer Vision Techniques for theDiagnosis of Skin Cancer, pages 49–69. Springer.

Chakravorty, R., Liang, S., Abedini, M., and Garnavi, R.(2016). Dermatologist-like feature extraction from skin le-sion for improved asymmetry classification in ph2 database.In Proc. IEEE International Conference on Engineering inMedicine and Biology Society (EMBC), pages 3855–3858,USA.

Dalal, N. and Triggs, B. (2005). Histograms of oriented gra-dients for human detection. In Proc. IEEE Computer Soci-ety Conference on Computer Vision and Pattern Recognition(CVPR), volume 1, pages 886–893, San Diego, USA.

Do, M. N. and Vetterli, M. (2003). The finite ridgelet trans-form for image representation. IEEE Transactions on ImageProcessing, 12(1):16–28.

Dony, R. and Wesolkowski, S. (1999). Edge detection oncolor images using rgb vector angles. In Proc. IEEE Cana-dian Conference on Electrical and Computer Engineering(CCECE), volume 2, pages 687–692, Edmonton, Canada.

Eltayef, K., Li, Y., and Liu, X. (2017). Detection of pigmentnetworks in dermoscopy images. In Proc. IOP InternationalConference on Communication, Image and Signal Process-ing (CCISP), volume 787, page 012034, Dubai.

Giotis, I., Land, N. M. S., Biehl, M., Jonkman, M. F.,and Petkov, N. (2015). MED-NODE: a computer-assistedmelanoma diagnosis system using non-dermoscopic images.

Expert Systems with Applications, 42:6578–6585.Huang, D., Jia, W., and Zhang, D. (2008). Palmprint ver-

ification based on principal lines. Pattern Recognition,41(4):1316–1328.

Jia, W., Hu, R. X., Lei, Y. K., Zhao, Y., and Gui, J. (2014).Histogram of oriented lines for palmprint recognition. IEEETransactions on Systems, Man, and Cybernetics: Systems,44(3):385–395.

Khotanzad, A. and Hong, Y. H. (1990). Invariant image recog-nition by Zernike moments. IEEE Transactions on PatternAnalysis and Machine Intelligence, 12(5):489–497.

Kiani, K. and Sharafat, A. R. (2011). E-shaver: an improvedDullRazor R© for digitally removing dark and light-coloredhairs in dermoscopic images. Computers in Biology andMedicine, 41(3):139–145.

Korotkov, K. and Garcia, R. (2012). Computerized analysis ofpigmented skin lesions: a review. Artificial Intelligence inMedicine, 56(2):69–90.

Kruk, M., Swiderski, B., Osowski, S., Kurek, J., Slowinska,M., and Walecka, I. (2015). Melanoma recognition usingextended set of descriptors and classifiers. EURASIP Journalon Image and Video Processing, 2015(1):1–10.

Lee, H. and Chen, Y. P. P. (2015). Image based computer aideddiagnosis system for cancer detection. Expert Systems withApplications, 42:5356–5365.

Lee, T., Ng, V., Gallagher, R., Coldman, A., and McLean,D. (1997). Dullrazor R©: a software approach to hair re-moval from images. Computers in Biology and Medicine,27(6):533–543.

Lowe, D. G. (1999). Object recognition from local scale-invariant features. In Proc. IEEE International Conferenceon Computer Vision (ICCV), volume 2, pages 1150–1157,Kerkyra, Greece.

Lowe, D. G. (2004). Distinctive image features from scale-invariant keypoints. International Journal of Computer Vi-sion, 60(2):91–110.

Mendonc, T., Ferreira, P. M., Marcal, A. R. S., Barata, C.,Rocha, J. S. M. J., and Rozeira, J. (2015). PH2: A Pub-lic Database for the Analysis of Dermoscopic Images. PhDthesis, Universidade do Porto.

Nachbar, F., Stolz, W., Merkle, T., Cognetta, A. B., Vogt, T.,Landthaler, M., Bilek, P., Braun-Falco, O., and Plewig, G.(1994). The ABCD rule of dermatoscopy: high prospec-tive value in the diagnosis of doubtful melanocytic skin le-sions. Journal of the American Academy of Dermatology,30(4):551–559.

Nock, R. and Nielsen, F. (2004). Statistical region merging.IEEE Transactions on pattern analysis and machine intelli-gence, 26(11):1452–1458.

Oliveira, R. B., Marranghello, N., Pereira, A. S., and Tavares,J. M. (2016). A computational approach for detecting pig-mented skin lesions in macroscopic images. Expert Systemswith Applications, 61:53–63.

Riaz, F., Hassan, A., Javed, M. Y., and Coimbra, M. T.(2014). Detecting melanoma in dermoscopy images usingscale adaptive local binary patterns. In Proc. IEEE Interna-

12

tional Conference on Engineering in Medicine and BiologySociety (EMBS), pages 6758–6761, Chicago, USA.

Ruela, M., Barata, C., Marques, J. S., and Rozeira, J. (2015).A system for the detection of melanomas in dermoscopy im-ages using shape and symmetry features. Computer Methodsin Biomechanics and Biomedical Engineering: Imaging andVisualization, pages 1–11.

She, Z., Duller, A., Liu, Y., and Fish, P. J. (2006). Simulationand analysis of optical skin lesion images. Skin Researchand Technology, 12(2):133–144.

Silveira, M., Nascimento, J. C., Marques, J. S., Marcal,A. R. S., Mendonca, T., Yamauchi, S., Maeda, J., andRozeira, J. (2009). Comparison of segmentation methods formelanoma diagnosis in dermoscopy images. IEEE Journalof Selected Topics in Signal Processing, 3(1):35–45.

Tanaka, T., Yamada, R., Tanaka, M., Shimizu, K., and Oka,H. (2004). A study on the image diagnosis of melanoma.In Proc. IEEE International Conference on Engineering inMedicine and Biology Society (EMBS), volume 1, pages1597–1600, San Francisco, Ca, USA.

Tang, Z., Dai, Y., Zhang, X., Huang, L., and Yang, F. (2014).Robust image hashing via colour vector angles and discretewavelet transform. The Institution of Engineering and Tech-nology (IET), Image Processing, 8(3):142–149.

Tobias, O. J. and Seara, R. (2002). Image segmentation by his-togram thresholding using fuzzy sets. IEEE Transactions onImage Processing, 11(12):1457–1465.

Vasconcelos, M., Rosado, L., and Ferreira, M. (2015). Anew color assessment methodology using cluster-based fea-tures for skin lesion analysis. In Proc. IEEE InternationalConvention on Information and Communication Technology,Electronics and Microelectronics (MIPRO), pages 373–378,Opatija, Croatia.

Zhao, Y., Wang, S., Zhang, X., and Yao, H. (2013). Robusthashing for image authentication using zernike moments andlocal features. IEEE Transactions on Information Forensicsand Security, 8(1):55–63.

Zhu, S. C. and Yuille, A. (1996). Region competition: Unifyingsnakes, region growing, and bayes/mdl for multiband imagesegmentation. IEEE Transactions on Pattern Analysis andMachine Intelligence, 18(9):884–900.

13

bagged textural and color features for melanoma skin ...keywords: malignant melanoma, skin cancer...

Documents