Download - talk
![Page 1: talk](https://reader034.vdocuments.site/reader034/viewer/2022051818/54bb0e1a4a795932048b4652/html5/thumbnails/1.jpg)
1
Spatially Constrained Segmentation of Dermoscopy Images
Howard Zhou1, Mei Chen2, Le Zou2, Richard Gass2,
Laura Ferris3, Laura Drogowski3, James M. Rehg1
1School of Interactive Computing, Georgia Tech2Intel Research Pittsburgh
3Department of Dermatology, University of Pittsburgh
![Page 2: talk](https://reader034.vdocuments.site/reader034/viewer/2022051818/54bb0e1a4a795932048b4652/html5/thumbnails/2.jpg)
2
Skin cancer and melanoma Skin cancer : most common of all cancers
[ Image courtesy of “An Atlas of Surface Microscopy of Pigmented Skin Lesions: Dermoscopy” ]
![Page 3: talk](https://reader034.vdocuments.site/reader034/viewer/2022051818/54bb0e1a4a795932048b4652/html5/thumbnails/3.jpg)
3
Skin cancer and melanoma Skin cancer : most common of all cancers Melanoma : leading cause of mortality (75%)
[ Image courtesy of “An Atlas of Surface Microscopy of Pigmented Skin Lesions: Dermoscopy” ]
![Page 4: talk](https://reader034.vdocuments.site/reader034/viewer/2022051818/54bb0e1a4a795932048b4652/html5/thumbnails/4.jpg)
4
Skin cancer and melanoma Skin cancer : most common of all cancers Melanoma : leading cause of mortality (75%) Early detection significantly reduces mortality
[ Image courtesy of “An Atlas of Surface Microscopy of Pigmented Skin Lesions: Dermoscopy” ]
![Page 5: talk](https://reader034.vdocuments.site/reader034/viewer/2022051818/54bb0e1a4a795932048b4652/html5/thumbnails/5.jpg)
5
[ Image courtesy of “An Atlas of Surface Microscopy of Pigmented Skin Lesions: Dermoscopy” ]
Clinical ViewDermoscopy view
![Page 6: talk](https://reader034.vdocuments.site/reader034/viewer/2022051818/54bb0e1a4a795932048b4652/html5/thumbnails/6.jpg)
6
Dermoscopy Improve diagnostic accuracy by 30% in the hands
of trained physicians May require as much as 5 year experience to have
the necessary training Motivation for Computer-aided diagnosis (CAD) in
this area
Clinical view Dermoscopy view
![Page 7: talk](https://reader034.vdocuments.site/reader034/viewer/2022051818/54bb0e1a4a795932048b4652/html5/thumbnails/7.jpg)
7
First step of analysis:Segmentation Separating lesions from surrounding skin Resulting border
Gives lesion size and border irregularity Crucial to the extraction of dermoscopic features for
diagnosis
Previous Work : PDE approach – Erkol et al. 2005, … Histogram thresholding – Hintz-Madsen et al. 2001, … Clustering – Schmid 1999, Melli et al. 2006… Statistical region merging – Celebi et al. 2007, …
![Page 8: talk](https://reader034.vdocuments.site/reader034/viewer/2022051818/54bb0e1a4a795932048b4652/html5/thumbnails/8.jpg)
8
Domain specific constraints Spatial constraints
Four corners are skin (Melli et al.2006, Celebi et al. 2007) Implicitly enforcing Local neighborhood constraints on image
Cartesian coordinates (Meanshift)
![Page 9: talk](https://reader034.vdocuments.site/reader034/viewer/2022051818/54bb0e1a4a795932048b4652/html5/thumbnails/9.jpg)
9
Domain specific constraints Spatial constraints
Four corners are skin (Melli et al.2006, Celebi et al. 2007) Implicitly enforcing Local neighborhood constraints on image
Cartesian coordinates (Meanshift)
Meanshift (c = 32, s = 8)
![Page 10: talk](https://reader034.vdocuments.site/reader034/viewer/2022051818/54bb0e1a4a795932048b4652/html5/thumbnails/10.jpg)
10
We explore … Spatial constraints arise from the growth
pattern of pigmented skin lesions
Meanshift (c = 32, s = 8)
![Page 11: talk](https://reader034.vdocuments.site/reader034/viewer/2022051818/54bb0e1a4a795932048b4652/html5/thumbnails/11.jpg)
11
We explore … Spatial constraints arise from the growth
pattern of pigmented skin lesions –
radiating pattern
Meanshift (c = 32, s = 8)
![Page 12: talk](https://reader034.vdocuments.site/reader034/viewer/2022051818/54bb0e1a4a795932048b4652/html5/thumbnails/12.jpg)
12
Embedding constraints
Meanshift (c = 32, s = 8) Polar (k = 6)
Radiating pattern from lesion growth Embedding constraints as polar coords
improves segmentation performance
![Page 13: talk](https://reader034.vdocuments.site/reader034/viewer/2022051818/54bb0e1a4a795932048b4652/html5/thumbnails/13.jpg)
13
Polar (k = 6)Meanshift Polar
Embedding constraints Radiating pattern from lesion growth Embedding constraints as polar coords
improves segmentation performance
![Page 14: talk](https://reader034.vdocuments.site/reader034/viewer/2022051818/54bb0e1a4a795932048b4652/html5/thumbnails/14.jpg)
14
Meanshift Polar
Comparison to the Doctors Radiating pattern from lesion growth Embedding constraints as polar coords
improves segmentation performance White: Dr. FerrisRed : Dr. ZhangBlue : computer
![Page 15: talk](https://reader034.vdocuments.site/reader034/viewer/2022051818/54bb0e1a4a795932048b4652/html5/thumbnails/15.jpg)
15
Dermoscopy images Common radiating appearance
![Page 16: talk](https://reader034.vdocuments.site/reader034/viewer/2022051818/54bb0e1a4a795932048b4652/html5/thumbnails/16.jpg)
16
Growth pattern of pigmented skin lesions lesions grow in both radial and vertical direction Skin absorbs and scatters light. Appearance of pigmented cells varies with depth
Dark brown tan blue-gray
Common radiating appearance pattern on skin surface
[ Image courtesy of “Dermoscopy : An Atlas of Surface Microscopy of Pigmented Skin Lesions]
![Page 17: talk](https://reader034.vdocuments.site/reader034/viewer/2022051818/54bb0e1a4a795932048b4652/html5/thumbnails/17.jpg)
17
Radiating growth pattern on skin surface Difference in appearance: more significant
along the radial direction than any other direction.
![Page 18: talk](https://reader034.vdocuments.site/reader034/viewer/2022051818/54bb0e1a4a795932048b4652/html5/thumbnails/18.jpg)
18
Radiating growth pattern on skin surface Difference in appearance: more significant
along the radial direction than any other direction.
![Page 19: talk](https://reader034.vdocuments.site/reader034/viewer/2022051818/54bb0e1a4a795932048b4652/html5/thumbnails/19.jpg)
19
Each pixel feature vector in R4 3D: R,G,B or L, a, b in the color space 1D: polar radius measured from the center of
the image (normalized by w)
Embedding spatial constraintsFeature vectors
original
r
{R, G, B}
![Page 20: talk](https://reader034.vdocuments.site/reader034/viewer/2022051818/54bb0e1a4a795932048b4652/html5/thumbnails/20.jpg)
20
Each pixel feature vector in R4 Clustering pixels in the feature space Replace pixels with mean for compact
representation
Embedding spatial constraintsGrouping features
filteredoriginal
r
{R, G, B}
![Page 21: talk](https://reader034.vdocuments.site/reader034/viewer/2022051818/54bb0e1a4a795932048b4652/html5/thumbnails/21.jpg)
21
Radiating pattern Dermoscopy vs. natural images
Derm dataset (216)
… …
BSD dataset (300)
![Page 22: talk](https://reader034.vdocuments.site/reader034/viewer/2022051818/54bb0e1a4a795932048b4652/html5/thumbnails/22.jpg)
22
Mean per-pixel residue: average per-pixel color difference of each pair
Embedding spatial constraintsGrouping features
original
{Ro, Go, Bo}
polar
{Rp, Gp, Bp}
Cartesian
{Rc, Gc, Bc}
![Page 23: talk](https://reader034.vdocuments.site/reader034/viewer/2022051818/54bb0e1a4a795932048b4652/html5/thumbnails/23.jpg)
23
Dermoscopy vs. natural images Polar vs. Cartesion
BSD dataset (300)
Residue (polar)
Residue (Cartesian)
Derm dataset (216)
Residue (Cartesian)
Residue (polar)
Mean per-pixel residue (k-means++, k = 30)
![Page 24: talk](https://reader034.vdocuments.site/reader034/viewer/2022051818/54bb0e1a4a795932048b4652/html5/thumbnails/24.jpg)
24
Dermoscopy vs. natural images Polar vs. Cartesion Mean per-pixel residue (k-means++, k = 30)
![Page 25: talk](https://reader034.vdocuments.site/reader034/viewer/2022051818/54bb0e1a4a795932048b4652/html5/thumbnails/25.jpg)
25
Polar vs. Cartesian The regions appear more blocky in the
Cartesian case
Polar (k = 30) Cartesian (k = 30)
![Page 26: talk](https://reader034.vdocuments.site/reader034/viewer/2022051818/54bb0e1a4a795932048b4652/html5/thumbnails/26.jpg)
26
Six super-regions 30 clusters 6 super clusters (K-means++)
Polar (k = 6) Cartesian (k = 6)
![Page 27: talk](https://reader034.vdocuments.site/reader034/viewer/2022051818/54bb0e1a4a795932048b4652/html5/thumbnails/27.jpg)
27
Final segmentation
Polar Cartesian
![Page 28: talk](https://reader034.vdocuments.site/reader034/viewer/2022051818/54bb0e1a4a795932048b4652/html5/thumbnails/28.jpg)
28
Polar vs. Meanshift The regions appear more blocky in the
Meanshift case
Polar (k = 6) Meanshift (c = 32, s = 8)
![Page 29: talk](https://reader034.vdocuments.site/reader034/viewer/2022051818/54bb0e1a4a795932048b4652/html5/thumbnails/29.jpg)
29
Final segmentation
Polar Meanshift
![Page 30: talk](https://reader034.vdocuments.site/reader034/viewer/2022051818/54bb0e1a4a795932048b4652/html5/thumbnails/30.jpg)
30
Given a dermoscopy image
Algorithm overview
![Page 31: talk](https://reader034.vdocuments.site/reader034/viewer/2022051818/54bb0e1a4a795932048b4652/html5/thumbnails/31.jpg)
31
Given a dermoscopy image
Algorithm overview
original
![Page 32: talk](https://reader034.vdocuments.site/reader034/viewer/2022051818/54bb0e1a4a795932048b4652/html5/thumbnails/32.jpg)
32
1. First round clustering: K-means++ (k = 30)
Algorithm overview
original 30 clusters
![Page 33: talk](https://reader034.vdocuments.site/reader034/viewer/2022051818/54bb0e1a4a795932048b4652/html5/thumbnails/33.jpg)
33
2. Second round: clusters(30) super-regions(6)
Algorithm overview
original 30 clusters 6 Super-regions
![Page 34: talk](https://reader034.vdocuments.site/reader034/viewer/2022051818/54bb0e1a4a795932048b4652/html5/thumbnails/34.jpg)
34
3. Apply texture gradient filter (Martin, et al. 2004)
Algorithm overview
original 30 clusters 6 Super-regions
Texture edge map
![Page 35: talk](https://reader034.vdocuments.site/reader034/viewer/2022051818/54bb0e1a4a795932048b4652/html5/thumbnails/35.jpg)
35
4. Find optimal boundary (color+texture)
Algorithm overview
original 30 clusters 6 Super-regions
Texture edge map Final segmentation
![Page 36: talk](https://reader034.vdocuments.site/reader034/viewer/2022051818/54bb0e1a4a795932048b4652/html5/thumbnails/36.jpg)
36
First round clustering: K-means++ (k = 30) Reduce noise Groups pixels into homogenous regions – a
more compact representation of the image Artuhur and Vassilvitskii, 2007
R4 : {L*a*b* (3D), w * polar radius (1D)}
1. First round clustering
original
![Page 37: talk](https://reader034.vdocuments.site/reader034/viewer/2022051818/54bb0e1a4a795932048b4652/html5/thumbnails/37.jpg)
37
First round clustering: K-means++ (k = 30) Reduce noise Groups pixels into homogenous regions – a
more compact representation of the image Artuhur and Vassilvitskii, 2007
R4 : {L*a*b* (3D), w * polar radius (1D)}
1. First round clustering
original 30 clusters
![Page 38: talk](https://reader034.vdocuments.site/reader034/viewer/2022051818/54bb0e1a4a795932048b4652/html5/thumbnails/38.jpg)
38
K = 6 : clusters(30) super-regions(6) Account for intra-skin and intra-lesion variations Avoid a large k
Super-regions correspond to meaningful regions such as skin, skin-lesion transition, and inner lesion, etc.
2. Second round clustering
original 30 clusters
![Page 39: talk](https://reader034.vdocuments.site/reader034/viewer/2022051818/54bb0e1a4a795932048b4652/html5/thumbnails/39.jpg)
39
K = 6 : clusters(30) super-regions(6) Account for intra-skin and intra-lesion variations Avoid a large k
Super-regions correspond to meaningful regions such as skin, skin-lesion transition, and inner lesion, etc.
2. Second round clustering
original 30 clusters 6 super-regions
![Page 40: talk](https://reader034.vdocuments.site/reader034/viewer/2022051818/54bb0e1a4a795932048b4652/html5/thumbnails/40.jpg)
40
3. Color-texture integration Incorporating texture information can
improve segmentation performance. Severely sun damaged skin; texture variations
at boundaries in addition to color variations
original
![Page 41: talk](https://reader034.vdocuments.site/reader034/viewer/2022051818/54bb0e1a4a795932048b4652/html5/thumbnails/41.jpg)
41
3. Color-texture integration Incorporating texture information can
improve segmentation performance. Severely sun damaged skin; texture variations
at boundaries in addition to color variations Apply texture gradient filter (Martin, et al. 2004)
original
![Page 42: talk](https://reader034.vdocuments.site/reader034/viewer/2022051818/54bb0e1a4a795932048b4652/html5/thumbnails/42.jpg)
42
3. Color-texture integration Incorporating texture information can
improve segmentation performance. Severely sun damaged skin; texture variations
at boundaries in addition to color variations Apply texture gradient filter (Martin, et al. 2004)
Texture edge map: pseudo-likelihood
original Texture edge map
![Page 43: talk](https://reader034.vdocuments.site/reader034/viewer/2022051818/54bb0e1a4a795932048b4652/html5/thumbnails/43.jpg)
43
Optimal skin-lesion boundary Color: Earth Mover’s Distance (EMD) between every
pair of super-regions
4. Optimal boundary
6 super-regions
![Page 44: talk](https://reader034.vdocuments.site/reader034/viewer/2022051818/54bb0e1a4a795932048b4652/html5/thumbnails/44.jpg)
44
Optimal skin-lesion boundary Color: Earth Mover’s Distance (EMD) between every pair
of super-regions Texture: Texture edge map
4. Optimal boundary
Texture edge map6 super-regions
![Page 45: talk](https://reader034.vdocuments.site/reader034/viewer/2022051818/54bb0e1a4a795932048b4652/html5/thumbnails/45.jpg)
45
Optimal skin-lesion boundary Color: Earth Mover’s Distance (EMD) between every pair
of super-regions Texture: Texture edge map Minimizing the integrated color-texture measure
4. Optimal boundary
Texture edge map6 super-regions
![Page 46: talk](https://reader034.vdocuments.site/reader034/viewer/2022051818/54bb0e1a4a795932048b4652/html5/thumbnails/46.jpg)
46
Our collaborating dermatologist Dr. Ferris manually outline the lesions in 67 dermoscopy images
The border error is given by
Computer : binary image obtained by filling the automatic detected border
ground-truth : obtained by filling in the boundaries outlined by Dr. Ferris
Validation and results
![Page 47: talk](https://reader034.vdocuments.site/reader034/viewer/2022051818/54bb0e1a4a795932048b4652/html5/thumbnails/47.jpg)
47
Typical segmentation result
Error = 12.96%
White: Dr. FerrisRed : Dr. ZhangBlue : computer
![Page 48: talk](https://reader034.vdocuments.site/reader034/viewer/2022051818/54bb0e1a4a795932048b4652/html5/thumbnails/48.jpg)
48
ComparisonCompared to ground-truth outlined by Dr. Ferris
11.32
20.64 21.41
16.9219.49 20.13
15.91 14.93
0
5
10
15
20
25
30
35
none Cartesian polar Dr. Zhang
Spatial constraints
Per
ce
nta
ge
err
or Dr. Zhang
RGB
CIELAB
Color + texture
To account for inter-operator variation, we also asked Dr. Alex Zhang to manually outline boundaries on the same dataset
![Page 49: talk](https://reader034.vdocuments.site/reader034/viewer/2022051818/54bb0e1a4a795932048b4652/html5/thumbnails/49.jpg)
49
Additional results
Error = 5.80%
White: Dr. FerrisRed : Dr. ZhangBlue : computer
![Page 50: talk](https://reader034.vdocuments.site/reader034/viewer/2022051818/54bb0e1a4a795932048b4652/html5/thumbnails/50.jpg)
50
Additional results
Error = 13.61%
White: Dr. FerrisRed : Dr. ZhangBlue : computer
![Page 51: talk](https://reader034.vdocuments.site/reader034/viewer/2022051818/54bb0e1a4a795932048b4652/html5/thumbnails/51.jpg)
51
Additional results
Error = 16.60%
White: Dr. FerrisRed : Dr. ZhangBlue : computer
![Page 52: talk](https://reader034.vdocuments.site/reader034/viewer/2022051818/54bb0e1a4a795932048b4652/html5/thumbnails/52.jpg)
52
Additional results
Error = 34.09%
White: Dr. FerrisRed : Dr. ZhangBlue : computer
![Page 53: talk](https://reader034.vdocuments.site/reader034/viewer/2022051818/54bb0e1a4a795932048b4652/html5/thumbnails/53.jpg)
53
Limitation
Assumption that lesions appear relatively near the center may not hold
Fairly low number of super regions (6) may limit the algorithm to perform well on lesions with more colors
![Page 54: talk](https://reader034.vdocuments.site/reader034/viewer/2022051818/54bb0e1a4a795932048b4652/html5/thumbnails/54.jpg)
54
Conclusion
Growth pattern of pigmented skin lesions can be used to improve lesion segmentation accuracy in dermoscopy images.
An unsupervised segmentation algorithm incorporating these spatial constraints
We demonstrate its efficacy by comparing the segmentation results to ground-truth segmentations determined by an expert.
![Page 55: talk](https://reader034.vdocuments.site/reader034/viewer/2022051818/54bb0e1a4a795932048b4652/html5/thumbnails/55.jpg)
55
Future work
Extend to meanshift?
![Page 56: talk](https://reader034.vdocuments.site/reader034/viewer/2022051818/54bb0e1a4a795932048b4652/html5/thumbnails/56.jpg)
56
Comparison to other methods
Compared to ground-truth outlined by Dr. Ferris
26.74
20.43 20.77 20.13
14.93
11.32
0
5
10
15
20
25
30
Meanshift JSEG (Celebi2006)
SRM (Celebi2007)
SCS Cartesian SCS polar Dr. Zhang
Segmentation methods
Pe
rce
nta
ge
err
or
![Page 57: talk](https://reader034.vdocuments.site/reader034/viewer/2022051818/54bb0e1a4a795932048b4652/html5/thumbnails/57.jpg)
57
Color and texture cue integration
Apply texture gradient filter (Martin, et al. 2004)
Pseudo-likelihood map - edge caused by texture variation is present at a certain location