graph cuts by using local texture features of wavelet coefficient for image segmentation
DESCRIPTION
GRAPH CUTS BY USING LOCAL TEXTURE FEATURES OF WAVELET COEFFICIENT FOR IMAGE SEGMENTATION. Keita Fukuda, Tetsuya Takiguchi, Yasuo Ariki Graduate School of Engineering, Kobe University, Japan. Table of contents. Introduction conventional method, problem with graph cuts Proposed method - PowerPoint PPT PresentationTRANSCRIPT
2008 International Conference on Multimedia & Expo
GRAPH CUTS BY USING LOCAL TEXTURE GRAPH CUTS BY USING LOCAL TEXTURE FEATURES OF WAVELET COEFFICIENT FEATURES OF WAVELET COEFFICIENT
FOR IMAGE SEGMENTATIONFOR IMAGE SEGMENTATION
Keita Fukuda, Tetsuya Takiguchi, Yasuo ArikiKeita Fukuda, Tetsuya Takiguchi, Yasuo ArikiGraduate School of Engineering, Kobe University, Graduate School of Engineering, Kobe University,
JapanJapan
2008 International Conference on Multimedia & Expo 2
Table of contentsTable of contents
• Introductionconventional method, problem with graph cuts
• Proposed methoditerated graph cuts using texture and smoothing
and detail of each method• Experiments
• Conclusion and Future Work
2008 International Conference on Multimedia & Expo 3
IntroductionIntroduction
Extracting the foreground objects in static images is one of the most fundamental tasks (Image Segmentation Problem).
“bkg”“obj”
2008 International Conference on Multimedia & Expo 4
BackgroundBackground
Recently, the image segmentation problem is formalized as an optimal solution problem.
e.g. Snakes, Level Set Method, Graph CutsY.Boykov, M.P.Jolly,“Interactive graph cuts for optimal boundary & region segmentation of object in N-D images”, IEEE International Conference on Computer Vision and Pattern Recognition
possible to compute global optimal solution
The cost function is general enough to include both region and boundary properties of the segments
Advantage of Graph Cuts
S
T
O
B
2008 International Conference on Multimedia & Expo 5
Graph CutsGraph Cuts
S
T
O
B
min cut
O
B
Assigned the corresponding label
Searching th
e
minimum co
st cu
tt-link
n-link
Background likelihood
S:”obj”
T:”bkg”
O
B
Neighbor similarity
A 3-by-4 image
Segmentation result
Object likelihood
Constructa graph
2008 International Conference on Multimedia & Expo 6
Problem (1) with GCProblem (1) with GC
It has been difficult to segment images that include complexnoisy edges.
To solve this problem,using iterated graph cuts based on smoothing.
T. Nagahashi, H.Fujiyoshi, and T.Kanade ,“ Image Segmentation Using Iterated Graph Cuts Based on Multi-scale Smoothing , ” ACCV2007
Noise exists in strong edge
2008 International Conference on Multimedia & Expo 7
Problem (2) with GCProblem (2) with GC
It is difficult to segment images with an object whose color is similar to the background.
To solve this problem,employing the texture likelihood as well as color likelihood.
The color of object is similar to that of background
Iterated Graph Cuts based on local texture features as well as low frequency features of wavelet coefficient.
Proposed method
2008 International Conference on Multimedia & Expo 8
To solve problem (2) with GC, high pass subbands (LHk,HLk,HHk) are used for t-link. Local texture features are defined from them.
The likelihood are derived from local texture features as well as color. And the prior probabilities are defined using the previous segmentation result.After initializing the level k, the input image is decomposed into subbands
using multiresolution wavelet analysis.To solve problem (1) with GC, low pass subband (LLk) is used for n-link.
The obtained neighbor similarity is set to n-link as edge cost.
Proposed MethodProposed Method
LL,LH,HL,HHare obtained
Graph Cuts Segmentation
Prior probability
GMM(color+texture)update
k ← k-1
LL: Smoothing
LH,HL,HH: Local texture features
Multiresolution analysis
(level k)
seed
Input Output
t-link
n-link
Graph Cuts segmentation is carried out, and these processes are repeated until k = 0.
2008 International Conference on Multimedia & Expo 9
Multiresolution Multiresolution analysisanalysis
downsampledHH1LH1
HL1LH2 HH2
HL2LL2
Level 2
LH1
HL1
HH1
LL1
Level 1
LL: low-pass information
LH: vertical
HL: horizontal
HH: diagonal
orientation
Subbands information
for n-link
for t-link
Input imageLevel 1Level 2
Input image downsampled
2008 International Conference on Multimedia & Expo 10
n-linkn-link
p q
},{ qpB
),(1
2)(
exp 2
2
},{ qpdistII
B qpqp
B{p,q} is large when p and q are similar, it is close to 0 when p and q are very different.
Global to local segmentation can be performed by iterated Graph Cuts with multiresolution analysis using from coarse to fine level for n-link .
finecoarse
Level 3 Level 2 Level 1
2008 International Conference on Multimedia & Expo 11
Local texture featuresLocal texture features
LH (level 1) HL (level 1)
Local texture features are defined by averaging the absolute wavelet coefficient in the window (3×3) surrounding pixel p
HH (level 1)
HHkHLkLHkfordTNqp
kp ,,||
91
,
)(
Local texture features
d: wavelet coefficient
k: level
Local texture features Tp are larger in complex region, and smaller in flat region.
2008 International Conference on Multimedia & Expo 12
t-linkt-link
Distance trans.
Pr(O)
Distance trans.
Pr(B)
S:”obj”
T:”bkg”
p
- ln Pr(B|Yp)
- ln Pr(O|Yp)
Color (RGB) :Cp
Local texture:Tp
Pr(Yp|O)
Color (RGB) :Cp
Local texture:Tp
Pr(Yp|B)
Background
Object
GMM
GMM
Object likelihood
Background likelihood
2008 International Conference on Multimedia & Expo 13
Graph Cuts Graph Cuts SegmentationSegmentation
Edge cost of the graph
)|Pr(ln)"("
)|Pr(ln)"("
pp
pp
YBbkgR
YOobjR
The boundary between the object and the background is found by searching the
minimum cost cut on the graph
S
T
O
B
mincut
Yuri Boykov and Vladimir Kolmogorov,“An Experimental Comparison of Min-Cut/Max-Flow Algorithms for Energy Minimization in Computer Vision”
2008 International Conference on Multimedia & Expo 14
Summarize proposed methodSummarize proposed method
S:”obj”
T:”bkg”
O
B
t-link (texture with color)n-link
Segmentation result
S:”obj”
T:”bkg”
O
B
Multiresolution at level kk Multiresolution at level k-1k-1
Multiresolution level is set to k-1k-1
The prior probability
2008 International Conference on Multimedia & Expo 15
Experimental ConditionExperimental Condition We compared using the same labels:method (1) Interactive Graph Cuts [Boykov 04]
method (2) Iterated Graph Cuts using smoothing [Nagahashi 07]
Proposed Iterated Graph Cuts using smoothing and texture Berkley Image database (50 images) The segmentation error rate is defined as follow:
001size image
background in the pixels detected misssize image
object in the pixels detectedmiss(%)
Err
Mask imageOutput imageError (in red and blue )
=
2008 International Conference on Multimedia & Expo 16
Experimental ResultExperimental Result
Level kk method (1) method (2) Proposed 1 4.62 3.78 3.172 4.62 2.95 2.623 4.62 2.95 1.98
The proposed method improved the error rate compared to two conventional methods.
In this database, two conventional methods could achieve the low error rate.
⇒ The examples with high error rate are shown from next page.
Err (%) in segmentation result at multiresolution level k
2008 International Conference on Multimedia & Expo 17
Experimental result (1)Experimental result (1) Effect of smoothing
6.27 % 1.16 % 0.85 %
0.79 %3.69 % 0.83 %
The method (2) and proposed method can achieve the better image segmentation for the images with complex edges than the method (1).
Method (1) Method (2) Proposed
2008 International Conference on Multimedia & Expo 18
Experimental result (2)Experimental result (2)
10.44 % 8.28 % 2.86 %
11.38 % 16.64 % 2.15 %
Effect of local texture featuresMethod (1) Method (2) Proposed
The proposed method can achieve the better image segmentation for the images with object colors similar to background than method (1) and(2)
2008 International Conference on Multimedia & Expo 19
DiscussionDiscussion To solve problem (1) : method (2), proposed
Prior probability The brief shape information can be given from previous segmentation result
Smoothing removing local strong edges.
To solve problem (2) : proposed Local texture features
for the image with object colors similarto background.
2008 International Conference on Multimedia & Expo 20
SummarySummary
Graph Cuts by using local texture features of wavelet coefficient for image segmentation.
New concept:Graph Cuts segmentation based on local texture features as well as smoothing process.
In future work: The weight to texture and color for segmentation Others texture features.
2008 International Conference on Multimedia & Expo 21
Thank you for your attention !Thank you for your attention !
2008 International Conference on Multimedia & Expo 22
The cost functionThe cost function
Nqp
qpqpPp
pp ffVfDfE,
, ),()()(
propertyregion the:)(Pp
pp fD
propertyboundary :),(,
,Nqp
qpqp ffV
The soft constraints that we impose of label f f are described by the cost function E(f) E(f) as follow:
The regional term assumes that the individual penalties for
assigning pixel p to labels.
The boundary term assumes that a penalty for a
discontinuity between p and q
2008 International Conference on Multimedia & Expo 23
Minimum cost cutMinimum cost cut
S
T
10
2
2
2
6
65
3
97
S
T
10
2
2
2
6
65
3
97
3/34/105/5
0/6
5/6
8/92/7
2/2
2/2
0/2
1. Graph Construct
Edge label = capacity
Edge = pipe
Goal : max flow from S to T
S
T
2
2
2
6
65
3
97
2 8
Example of a graphResidual GraphCurrent flow
3. Current flow from residual graph
e.g. 8/9 = (flow) / (capacity)
2. Residual graph
Find augmenting paths until finished.
2008 International Conference on Multimedia & Expo 24
Minimum cost cutMinimum cost cut
S
T
10
2
2
2
6
65
3
97
1. Graph Construct
Edge label = capacity
Edge = pipe
Goal : max flow from S to T
3. Current flow from residual graph
e.g. 8/9 = (flow) / (capacity)
4. Minimum cost cut
Saturated edges are found from 2.
Each node is assigned for the corresponding labels.
S
T
10
2
2
2
6
65
3
97
3/34/105/5
0/6
5/6
8/92/7
2/2
2/2
0/2
mincut
S T T
Yuri Boykov and Vladimir Kolmogorov,“An Experimental Comparison of Min-Cut/Max-Flow Algorithms for Energy Minimization in Computer Vision”
2. Residual graph
Find augmenting paths until finished.
2008 International Conference on Multimedia & Expo 25
Experimental result (3)Experimental result (3) for the unclear boundary images.
The proposed method is effective for images which have the unclear boundary between object and background.
Method (1) Method (2) Proposed
2008 International Conference on Multimedia & Expo 26
Time costTime cost
The size of image : 650×400
method(1) method(2) Proposed
10.8 sec 56.8 sec 82.5 sec
The proposed method carried out 3 times graph cuts segmentation.
And Gaussian Mixture Model is derived from 6 dimensional features.
It is because the proposed method is slow.
Time cost
2008 International Conference on Multimedia & Expo 27
Discussion (2)Discussion (2)
Remained problemThe proposed method is ineffective for a flat
image such as artificial images due to a few edges.
Future work include optimization of the weight to texture and color for segmentation.
2008 International Conference on Multimedia & Expo 28
Noisy edge problemNoisy edge problem
If the strong edges exist ,
n-link edge cost is small.
(easy to pass the n-link edge)
p q
S
T
p q
S
T
n-link edge cost is large.
(hardly pass the n-link edge)
Else ,
2008 International Conference on Multimedia & Expo 29
Influence of seeds Influence of seeds positionposition
• Graph cuts technique is quite stable and normally produces the same results regardless of particular seed positioning within the same image.
2008 International Conference on Multimedia & Expo 30
Prior probabilityPrior probability
• Distance dd from the border between the object and background is normalized from 0.5 to 1.0 .
• The prior probability is defined by using dojb and dbkg.
Prior probability
otherwised
ddifd
bkg
bkgobjobj
1)Pr(O
)Pr(1)Pr( OB
2008 International Conference on Multimedia & Expo 31
First time First time segmentationsegmentation
S:”obj”
T:”bkg”
p
- ln Pr(Yp|B)
- ln Pr(Yp|O)
Pr(Yp|O)
Pr(Yp|B)
Color (RGB) :Cp
Local texture:TpGMM
Color (RGB) :Cp
Local texture:TpGMM
“obj” seeds
“bkg” seeds
2008 International Conference on Multimedia & Expo 32
Multiresolution levelMultiresolution level
At level 3
The low pass subband LL3 is 1 / 64 * the original image.
The low pass subband LL4 is 1 / 256 * the original image.
At level 2
2008 International Conference on Multimedia & Expo 33
ParameterParameter
Graph Cuts parameter :
λ = 0.05
σ = 12.0
The constant KK is larger than sum of all n-link costs not to label the opposite label.
Gaussian Mixture Model:
5 components
)|Pr(ln)"("
)|Pr(ln)"("
max1
),(1
2)(
exp
},{:},{
2
2
},{
pp
pp
NqpqqpPp
qpqp
YBbkgR
YOobjR
BK
qpdistII
B
2008 International Conference on Multimedia & Expo 34
Local Texture featuresLocal Texture features
2008 International Conference on Multimedia & Expo 35
Results of proposed Results of proposed methodmethod
2008 International Conference on Multimedia & Expo 36
Result of proposed Result of proposed method method
2008 International Conference on Multimedia & Expo 37
QuestionQuestion
• Please repeat the question.• Thank you for your question. Are you asking about … ?• What you see here is …• I believe that …• To be honest , we never looked into possibility.• The question was …• There is not the particular deep meaning.• Not yet• Thank you for your advice.
2008 International Conference on Multimedia & Expo 38
追加スライド追加スライド• n-link の問題について• ユーザの引き方に問題は?• 数パーセントでも違いがはっきりしている• パラメータの説明• Level4 がない理由• 問題点• 一回目のセグメンテーション
2008 International Conference on Multimedia & Expo 39
Minimum cost cutMinimum cost cut
S
T
10
2
2
2
6
65
3
97
10+5+6+9 = 30
2+2+5+3 = 12 (mincut)
10+2+5+6+9 = 32
2008 International Conference on Multimedia & Expo 40
S
T
2
2
2
6
65
3
97
2 8
input labeled mask output error