lecture: k-means & mean-shift clustering - stanford...
Post on 14-May-2018
230 Views
Preview:
TRANSCRIPT
Lecture 11 -Stanford University
Lecture:k-means&mean-shiftclustering
JuanCarlosNiebles andRanjayKrishnaStanfordVisionandLearningLab
26-Oct-171
Lecture 11 -Stanford University
Recap:ImageSegmentation
• Goal:identifygroupsofpixelsthatgotogether
26-Oct-172
Lecture 11 -Stanford University
Recap:GestaltTheory• Gestalt:wholeorgroup
– Wholeisgreaterthansumofitsparts– Relationshipsamongpartscanyieldnewproperties/features
• Psychologistsidentifiedseriesoffactorsthatpredisposesetofelementstobegrouped(byhumanvisualsystem)
Untersuchungen zur Lehre von der Gestalt,Psychologische Forschung, Vol. 4, pp. 301-350, 1923http://psy.ed.asu.edu/~classics/Wertheimer/Forms/forms.htm
“I stand at the window and see a house, trees, sky. Theoretically I might say there were 327 brightnessesand nuances of colour. Do I have "327"? No. I have sky, house, and trees.”
Max Wertheimer(1880-1943)
26-Oct-173
Lecture 11 -Stanford University
Recap:GestaltFactors
• Thesefactorsmakeintuitivesense,butareverydifficulttotranslateintoalgorithms.
26-Oct-174
Lecture 11 -Stanford University
Whatwillwelearntoday?
• K-meansclustering• Mean-shiftclustering
26-Oct-175
Reading:[FP] Chapters:14.2,14.4D.Comaniciu andP.Meer,MeanShift:ARobustApproachtowardFeatureSpaceAnalysis,PAMI2002.
Lecture 11 -Stanford University
Whatwillwelearntoday?
• K-meansclustering• Mean-shiftclustering
26-Oct-176
Reading:[FP] Chapters:14.2,14.4D.Comaniciu andP.Meer,MeanShift:ARobustApproachtowardFeatureSpaceAnalysis,PAMI2002.
Lecture 11 -Stanford University
ImageSegmentation:ToyExample
• Theseintensitiesdefinethethreegroups.• Wecouldlabeleverypixelintheimageaccordingtowhich
oftheseprimaryintensitiesitis.– i.e.,segmenttheimagebasedontheintensityfeature.
• Whatiftheimageisn’tquitesosimple?
intensityinput image
blackpixels graypixels
whitepixels
1 23
Slide credit: Kristen Grauman
26-Oct-177
Lecture 11 -Stanford University
Pixelcou
nt
Inputimage
InputimageIntensity
Pixelcou
nt
Intensity
Slide credit: Kristen Grauman
26-Oct-178
Lecture 11 -Stanford University
• Nowhowtodeterminethethreemainintensitiesthatdefineourgroups?
• Weneedtocluster.
InputimageIntensity
Pixelcou
nt
Slide credit: Kristen Grauman
26-Oct-179
Lecture 11 -Stanford University
• Goal:choosethree“centers”astherepresentativeintensities,andlabeleverypixelaccordingtowhichofthesecentersitisnearestto.
• BestclustercentersarethosethatminimizeSumofSquareDistance(SSD)betweenallpointsandtheirnearestclustercenterci:
Slid
e cr
edit
: Kr
iste
n G
raum
an
0 190 255
1 23
Intensity
26-Oct-1710
SSD = x − ci( )2x∈clusteri∑
clusteri∑
Lecture 11 -Stanford University
ClusteringforSummarization
Goal:clustertominimizevarianceindatagivenclusters– Preserveinformation
Whether is assigned to
Cluster center Data
Slide:DerekHoiem
c*, δ* = argminc, δ
1N
δiji
K
∑ ci − x j( )2
j
N
∑
x j ci
26-Oct-1711
Lecture 11 -Stanford University
Clustering• Withthisobjective,itisa“chickenandegg”problem:– Ifweknewtheclustercenters,wecouldallocatepointstogroupsbyassigningeachtoitsclosestcenter.
– Ifweknewthegroupmemberships,wecouldgetthecentersbycomputingthemeanpergroup.
Slid
e cr
edit
: Kr
iste
n G
raum
an
26-Oct-1712
Lecture 11 -Stanford University
K-meansclustering1. Initialize():clustercenters
2. Compute:assigneachpointtotheclosestcenter– denotesthesetofassignmentforeachtoclusteratiterationt
1. Computer:updateclustercentersasthemeanofthepoints
1. Update,RepeatStep2-3tillstopped
Slide:DerekHoiem
c1,...,cK
δ t = argminδ
1N
δ t−1iji
K
∑ ct−1i − x j( )2
j
N
∑
ct = argminc
1N
δ tiji
K
∑ ct−1i − x j( )2
j
N
∑
t = t +1
x j ciδ t
t = 0
δ t
ct
26-Oct-1713
Lecture 11 -Stanford University
K-meansclustering1. Initialize():clustercenters
2. Compute:assigneachpointtotheclosestcenter
1. Computer:updateclustercentersasthemeanofthepoints
2. Update,RepeatStep2-3tillstopped
Slide:DerekHoiem
c1,...,cKt = 0
δ t
ct
• Commonlyused:randominitialization• OrgreedilychooseKtominimizeresidual
• Typicaldistancemeasure:• Euclidean• Cosine• Others
• doesn’tchangeanymore.ct
sim(x, !x ) = xT !xsim(x, !x ) = xT !x x ⋅ !x( )
t = t +1
ct = argminc
1N
δ tiji
K
∑ ct−1i − x j( )2
j
N
∑
26-Oct-1714
Lecture 11 -Stanford University
K-meansclustering
Illustration Source: wikipedia
1. Initialize Cluster Centers
2. Assign Points to Clusters
3. Re-compute Means
Repeat (2) and (3)
• Javademo:http://home.dei.polimi.it/matteucc/Clustering/tutorial_html/AppletKM.html
26-Oct-1715
Lecture 11 -Stanford University
• Convergestoalocalminimum solution– Initializemultipleruns
• Betterfitforsphericaldata
• NeedtopickK(#ofclusters)
26-Oct-1716
K-meansclustering
Lecture 11 -Stanford University
SegmentationasClustering
2clusters
26-Oct-1717
Originalimage
3clusters
Lecture 11 -Stanford University
K-Means++
• Canwepreventarbitrarilybadlocalminima?
1. Randomlychoosefirstcenter.2. Picknewcenterwithprob.proportionalto
– (Contributionofx tototalerror)
3. RepeatuntilK centers.
• Expectederror*optimal
Arthur & Vassilvitskii 2007
Slid
e cr
edit
: St
eve
Seit
z
26-Oct-1718
x − ci( )2
=O log K( )
Lecture 11 -Stanford University
FeatureSpace• Dependingonwhatwechooseasthefeaturespace,wecangrouppixelsindifferentways.
• Groupingpixelsbasedonintensitysimilarity
• Featurespace:intensityvalue(1D)Slide credit: Kristen Grauman
26-Oct-1719
Lecture 11 -Stanford University
FeatureSpace• Dependingonwhatwechooseasthefeaturespace,wecan
grouppixelsindifferentways.
• Groupingpixelsbasedon colorsimilarity
• Featurespace:colorvalue(3D)
R=255G=200B=250
R=245G=220B=248
R=15G=189B=2
R=3G=12B=2
R
GB
Slide credit: Kristen Grauman
26-Oct-1720
Lecture 11 -Stanford University
FeatureSpace• Dependingonwhatwechooseasthefeaturespace,wecan
grouppixelsindifferentways.
• Groupingpixelsbasedon texturesimilarity
• Featurespace:filterbankresponses(e.g.,24D)
Filterbankof24filters
F24
F2
F1
…Slide credit: Kristen Grauman
26-Oct-1721
Lecture 11 -Stanford University
SmoothingOutClusterAssignments
• Assigningaclusterlabelperpixelmayyieldoutliers:
• Howcanweensuretheyarespatiallysmooth? 1 2
3?
Original Labeledbyclustercenter’sintensity
Slide credit: Kristen Grauman
26-Oct-1722
Lecture 11 -Stanford University
SegmentationasClustering• Dependingonwhatwechooseasthefeaturespace,wecangrouppixelsindifferentways.
• Groupingpixelsbasedonintensity+position similarity
ÞWaytoencodebothsimilarity andproximity.Slide credit: Kristen Grauman
X
Intensity
Y
26-Oct-1723
Lecture 11 -Stanford University
K-MeansClusteringResults• K-meansclusteringbasedonintensityorcolorisessentiallyvectorquantizationoftheimageattributes– Clustersdon’thavetobespatiallycoherent
Image Intensity-basedclusters Color-basedclusters
Imag
e so
urce
: Fo
rsyt
h &
Pon
ce
26-Oct-1724
Lecture 11 -Stanford University
K-MeansClusteringResults
• K-meansclusteringbasedonintensityorcolorisessentiallyvectorquantizationoftheimageattributes– Clustersdon’thavetobespatiallycoherent
• Clusteringbasedon(r,g,b,x,y)valuesenforcesmorespatialcoherence
Imag
e so
urce
: Fo
rsyt
h &
Pon
ce
26-Oct-1725
Lecture 11 -Stanford University
Howtoevaluateclusters?
• Generative– Howwellarepointsreconstructedfromtheclusters?
• Discriminative– Howwelldotheclusterscorrespondtolabels?
• Canwecorrectlyclassifywhichpixelsbelongtothepanda?
– Note:unsupervisedclusteringdoesnotaimtobediscriminativeaswedon’thavethelabels.
Slide:DerekHoiem26-Oct-1726
Lecture 11 -Stanford University
Howtochoosethenumberofclusters?Trydifferentnumbersofclustersinavalidationsetandlookatperformance.
Slide:DerekHoiem26-Oct-1727
Lecture 11 -Stanford University
K-Meansprosandcons• Pros
• Findsclustercentersthatminimizeconditionalvariance(goodrepresentationofdata)
• Simpleandfast,Easytoimplement• Cons
• NeedtochooseK• Sensitivetooutliers• Pronetolocalminima• Allclustershavethesameparameters
(e.g.,distancemeasureisnon-adaptive)
• *Canbeslow:eachiterationisO(KNd)forNd-dimensionalpoints
• Usage• Unsupervisedclustering• Rarelyusedforpixelsegmentation
26-Oct-1728
Lecture 11 -Stanford University
Whatwillwelearntoday?
• K-meansclustering• Mean-shiftclustering
26-Oct-1729
Reading:[FP] Chapters:14.2,14.4D.Comaniciu andP.Meer,MeanShift:ARobustApproachtowardFeatureSpaceAnalysis,PAMI2002.
Lecture 11 -Stanford University
Mean-ShiftSegmentation
• Anadvancedandversatiletechniqueforclustering-basedsegmentation
http://www.caip.rutgers.edu/~comanici/MSPAMI/msPamiResults.html
D.Comaniciu andP.Meer,MeanShift:ARobustApproachtowardFeatureSpaceAnalysis,PAMI2002. Slid
e cr
edit
: Sv
etla
na L
azeb
nik
26-Oct-1730
Lecture 11 -Stanford University
Mean-ShiftAlgorithm
• IterativeModeSearch1. Initializerandomseed,andwindowW2. Calculatecenterofgravity(the“mean”)ofW:3. Shiftthesearchwindowtothemean4. RepeatStep2untilconvergence
Slid
e cr
edit
: St
eve
Seit
z
26-Oct-1731
Lecture 11 -Stanford University
Region ofinterest
Center ofmass
Mean Shiftvector
Mean-Shift
SlidebyY.Ukrainitz &B.Sarel
26-Oct-1732
Lecture 11 -Stanford University
Region ofinterest
Center ofmass
Mean Shiftvector
Mean-Shift
SlidebyY.Ukrainitz &B.Sarel
26-Oct-1733
Lecture 11 -Stanford University
Region ofinterest
Center ofmass
Mean Shiftvector
Mean-Shift
SlidebyY.Ukrainitz &B.Sarel
26-Oct-1734
Lecture 11 -Stanford University
Region ofinterest
Center ofmass
Mean Shiftvector
Mean-Shift
SlidebyY.Ukrainitz &B.Sarel
26-Oct-1735
Lecture 11 -Stanford University
Region ofinterest
Center ofmass
Mean Shiftvector
Mean-Shift
SlidebyY.Ukrainitz &B.Sarel
26-Oct-1736
Lecture 11 -Stanford University
Region ofinterest
Center ofmass
Mean Shiftvector
Mean-Shift
SlidebyY.Ukrainitz &B.Sarel
26-Oct-1737
Lecture 11 -Stanford University
Region ofinterest
Center ofmass
Mean-Shift
SlidebyY.Ukrainitz &B.Sarel
26-Oct-1738
Lecture 11 -Stanford University
Tessellate the space with windows Run the procedure in parallel SlidebyY.Ukrainitz&B.Sarel
RealModalityAnalysis
26-Oct-1739
Lecture 11 -Stanford University
The blue data points were traversed by the windows towards the mode. SlidebyY.Ukrainitz&B.Sarel
RealModalityAnalysis
26-Oct-1740
Lecture 11 -Stanford University
Mean-ShiftClustering
• Cluster:alldatapointsintheattractionbasinofamode
• Attractionbasin:theregionforwhichalltrajectoriesleadtothesamemode
SlidebyY.Ukrainitz &B.Sarel
26-Oct-1741
Lecture 11 -Stanford University
Mean-ShiftClustering/Segmentation• Findfeatures(color,gradients,texture,etc)• Initializewindowsatindividualpixellocations• Performmeanshiftforeachwindowuntilconvergence• Mergewindowsthatendupnearthesame“peak”ormode
Slid
e cr
edit
: Sv
etla
na L
azeb
nik
26-Oct-1742
Lecture 11 -Stanford University
Mean-ShiftSegmentationResults
http://www.caip.rutgers.edu/~comanici/MSPAMI/msPamiResults.html Slid
e cr
edit
: Sv
etla
na L
azeb
nik
26-Oct-1743
Lecture 11 -Stanford University
• Needtoshiftmanywindows…• Manycomputationswillberedundant.
Problem:ComputationalComplexity
Slid
e cr
edit
: Ba
stia
n Le
ibe
26-Oct-1746
Lecture 11 -Stanford University
Speedups:BasinofAttraction
r
Slid
e cr
edit
: Ba
stia
n Le
ibe
26-Oct-1747
1. Assignallpointswithinradiusrofendpointtothemode.
Lecture 11 -Stanford University
Speedups
r =c
Slid
e cr
edit
: Ba
stia
n Le
ibe
26-Oct-1748
2. Assignallpointswithinradiusr/cofthesearchpathtothemode->reducethenumberofdatapointstosearch.
Lecture 11 -Stanford University
TechnicalDetails
26-Oct-1750
Comaniciu &Meer,2002
• Term1:thisisproportionaltothedensityestimateatx(similartoequation1fromthepreviousslide).
• Term2:thisisthemean-shiftvectorthatpointstowardsthedirectionofmaximumdensity.
Lecture 11 -Stanford University
TechnicalDetails
26-Oct-1751
Comaniciu &Meer,2002
Finally,themeanshiftprocedurefromagivenpointxt is:1. Computerthemeanshirtvectorm:
2. Translatethedensitywindow:
3. Iteratesteps1and2untilconvergence.
Lecture 11 -Stanford University
SummaryMean-Shift• Pros
– General,application-independenttool– Model-free,doesnotassumeanypriorshape(spherical,
elliptical,etc.)ondataclusters– Justasingleparameter(windowsizeh)
• hhasaphysicalmeaning(unlikek-means)– Findsvariablenumberofmodes– Robusttooutliers
• Cons– Outputdependsonwindowsize– Windowsize(bandwidth)selectionisnottrivial– Computationally(relatively)expensive(~2s/image)– Doesnotscalewellwithdimensionoffeaturespace
Slid
e cr
edit
: Sv
etla
na L
azeb
nik
26-Oct-1752
top related