岩田 覚

(京都大学数理解析研究所)

劣モジュラ最適化

Outline• Submodular Functions

ExamplesDiscrete Convexity

• Minimizing Submodular Functions • Symmetric Submodular Functions• Maximizing Submodular Functions• Approximating Submodular Functions

Submodular Functions

• Cut Capacity Functions • Matroid Rank Functions • Entropy Functions

)()()()( YXfYXfYfXf ∪+∩≥+

VYX ⊆∀ ,

VX Y

R2: →Vf:V Finite Set

Cut Capacity Function

Cut Capacity ∑= } leaving :|)({)( XaacXκ

X

s t

Max Flow Value＝Min Cut Capacity

0)( ≥ac

Matroid Rank Functions

:)()(

||)(,

ρρρ

ρYXYX

XXVX≤⇒⊆≤⊆∀

],[rank)( XUAX =ρMatrix Rank Function

Submodular

=A

X

V

U

Whitney (1935)

Entropy Functions

:)(Xh

0)( =φh

X

Information Sources

0≥

)()()()( YXhYXhYhXh ∪+∩≥+

Entropy of the Joint Distribution

Conditional Mutual Information

Positive Definite Symmetric Matrices

0)( =φf

=A ][XA][detlog)( XAXf =

X

)()()()( YXfYXfYfXf ∪+∩≥+

Ky Fan’s Inequality

Extension of the Hadamard Inequality∏∈

≤Vi

iiAAdet

Discrete Concavity

})({}){()(}){( ufuTfSfuSfTS

−∪≥−∪⇒⊆

V

T

Diminishing Returns

S u

Discrete Convexity

)()()()( YXfYXfYfXf ∪+∩≥+

V

X Y

Convex Function

x

y

Discrete Convexity

},{ ba

},,{ cbaV =

}{b

}{a}{c

},{ cb

φ

Lovász (1983)

f̂

f

f̂

： Linear Interpolation

： Convex

： Submodular

Murota (2003)

Discrete Convex Analysis

Submodular Function Minimization

X

)(Xf

?}|)(min{ VYYf ⊆

MinimizationAlgorithm

Evaluation Oracle

Minimizer

0)( =φfAssumption:

Ellipsoid MethodGrötschel, Lovász, Schrijver (1981)

Base Polyhedra

Submodular Polyhedron

)}()(,,|{)( YfYxVYxxfP V ≤⊆∀∈= R

∑∈

=Yv

vxYx )()(

}|{ RR →= VxV

)}()(),(|{)( VfVxfPxxfB =∈=Base Polyhedron

Greedy Algorithm

v

⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢

⎣

⎡

=

⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢

⎣

⎡

⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢

⎣

⎡

))((

))(())((

)(

)()(

1110

11001

2

1

2

1

nn vLf

vLfvLf

vy

vyvy

MM

L

OMM

MO

L

}){)(())(()( vvLfvLfvy −−=

Edmonds (1970)Shapley (1971)

Extreme Base

)(vL

)( Vv∈

:y

2v1v nv

Theorem

Min-Max Theorem

)}(|)(max{ )(min fBxVxYfVY

∈= −

⊆

)}(,0min{:)( vxvx =−

)()()( YfYxVx ≤≤−

Edmonds (1970)

Combinatorial Approach

Extreme Base

Convex Combination

Cunningham (1985)

LL

L yx ∑Λ∈

= λ

)( fByL ∈

x

|)(|max XfMVX⊆

=

)log( 6 nMMnO γ

Submodular Function Minimization

)( 87 nnO +γ)log( 5 MnO γ

)log( 7 nnO γ

Grötschel, Lovász, Schrijver (1981, 1988)

Iwata, Fleischer, Fujishige (2000) Schrijver (2000)

Iwata (2003)

Fleischer, Iwata (2000)

)log)(( 54 MnnO +γOrlin (2007)

)( 65 nnO +γ

Iwata (2002)

Fully Combinatorial

Ellipsoid Method

Cunningham (1985)

Iwata, Orlin (2009)

)( subject to Minimize 2

fBxx∈

The Minimum-Norm Base

sol. opt. :∗x}0)(|{: <= ∗

− vxvX}0)(|{:0 ≤= ∗ vxvX

Minimal MinimizerMaximal Minimizer

∗x

Theorem Fujishige (1984)

Wolfe’s Algorithm Practically Efficient

Evacuation Problem （Dynamic Flow）

:)(Xo

:)(aτ

Hoppe, Tardos (2000)

:)(vb

:)(ac

SX ∩ XT \

TSXXoXb ∪⊆∀≤ ),()(

TSCapacity

Transit Time

Supply/Demand

Maximum Amount of Flow from to .

Feasible

Multiterminal Source Coding

X XR

Y

Encoder

Encoder YRDecoder

)(YH

)(XH

)|( XYH

)|( YXH

),( YXH

),( YXHXR

YR

Slepian, Wolf (1973)

Multiclass Queueing Systems

iμiλ

Server

Control Policy

Arrival Rate Service Rate

Multiclass M/M/1Preemptive

Arrival Interval Service Time

Performance Region

VXS

Xii

Xiii

Xiii ⊆∀

−≥

∑∑

∑∈

∈

∈

,1 ρ

μρρ

:s

,: iii μλρ =

YRjs

is

:js Expected Staying Time of a Job in j

Achievable

Coffman, Mitrani (1980)

:s1<∑

∈Viiρ

A Class of Submodular FunctionsVzyx +∈R,,

))(()()()( XxhXyXzXf −=

:h

VXS

Xii

Xiii

Xiii ⊆∀

−≥

∑∑

∑∈

∈

∈

,1 ρ

μρρ

Nonnegative, Nondecreasing, Convex

)( VX ⊆

i

iiy

μρ

=:

xxh

−=

11:)(

iii Sz ρ=:

iix ρ=:

Submodular

Itoko & Iwata (2005)

Zonotope in 3D))(),(),(()( XzXyXxXw =

}|)({conv VXXwZ ⊆=

)(),,(~ xyhzzyxf −=

Zonotope

} ofPoint ExtremeLower :),,(|),,(~min{

}|)(min{

Zzyxzyxf

VXXf

=

⊆

Remark: ),,(~ zyxf is NOT concave!

z

Line Arrangementβ

iii zyx =+ βα

α

Topological Sweeping MethodEdelsbrunner, Guibas (1989) )( 2nO

Enumerating All the Cells

Symmetric Submodular Functions

. ),\()( VXXVfXf ⊆∀=

?},|)(min{ VXVXXf ≠⊂≠φ

)()()()( ,

YXfYXfYfXfVYXYX

∩+∪≥+⇒≠∪≠∩ φ

R→Vf 2:

Symmetric

Crossing Submodular

Symmetric Submodular Function Minimization

Maximum Adjacency Ordering

• Minimum Cut Algorithm by MA-orderingNagamochi & Ibaraki (1992)

• Simpler ProofsFrank (1994), Stoer & Wagner (1997)

• Symmetric Submodular Functions Queyranne (1998)

• Alternative ProofsFujishige (1998), Rizzi (2000)

Minimum Degree OrderingNagamochi (2007)

ISAAC’07, Sendai, Japan

Finding the family of all extreme sets for symmetric crossing submodular functionsin time. )( 3γnO

Symmetric Submodular Function Minimization

Extreme Sets

.: ),()( XZXZXfZf ≠≠⊂∀> φ

)\()\()()( XYfYXfYfXf +≥+

:f Symmetric Crossing Submodular Function

Extreme Set:X

The family of all extreme sets forms a laminar.

X YX Y

Flat Pair for Symmetric Submodular Functions

,}|)(min{)( XxxfXf ∈≥

.1|},{| s.t. =∩⊆∀ vuXVX

)( },{ vuVvu ≠⊆Flat Pair

uv

X

},{ vu

No Extreme SetsSeparate andu .v

Shrink into a single vertex.

},{ vu

MD-Ordering for Symmetric Submodular Functions

X

)\( )()(:)( iii VVXXVfXfXf ⊆∪+=

iV

Symmetric, Crossing Submodular

MD-orderingVvvvv nn ∈− ,,...,, 121

jvEach has minimum value of among )(1 vf j− .\ 1−∈ jVVv

},...,{: 1 ii vvV =

MD-Ordering for Symmetric Submodular Functions

nn vv ,1−

1−nv2−= ni

The last two vertices of an MD-ordering form a flat pair.

Proof by Induction:

:},{ 1 nn vv − if

.0,1,...,2−= ni

Flat Pair for on

nv

iVV \

Time Complexity

• Finding an MD-ordering in time.

• Finding all the extreme sets in time.

• Minimizing symmetric submodular functions in time.

)( 3γnO

)( 2γnO

)( 3γnO

Application to Clustering

A

V

AV \nXX ,...,1

),()()( )|()();(

BABA

ABBBA

XXHXHXHXXHXHXXI−+=

−=

Random variables

VPartition into andas independent as possible

);( \ AVA XXIMinimize subject to VA ≠≠φ

AV \A

Application to Clustering

)( jiVV ji ≠=∩ φ

∑=

k

iiVf

1)(Minimize

subject tokVVV ∪∪= L1

Greedy Split

)12( k− -Approximation

Submodurlar Function Maximization

Approximation AlgorithmsNemhauser, Wolsey, Fisher (1978)

Monotone SF / Cardinality Constraint(1-1/e)-Approximation

Feige, Mirrokni, Vondrák (FOCS 2007)Nonnegative SF2/5-Approximation

Vondrák (STOC 2008) Monotone SF / Matroid Constraint(1-1/e)-Approximation

Approximate Maximization

)()(:

}{:

}){(maxarg::

1

1

1

0

−

−

−

−=

∪=

∪==

jjj

jjj

jj

TfTf

vTT

vTfvT

ρ

φ

1−jv1vGreedy Algorithm

)(SfMaximize subject to kS ≤||

kj ,...,1=

1−jT

( ) kSSSfTf k ≤∀−≥ |:| ),(e11)(

Approximate Maximization

),...,1( kj =

)(: *Sf=η:*S

∑−

=

+≤1

1

j

iijk ρρη

[ ]

jj

TSujjj

j

kTf

TfuTfTf

TSfSf

j

ρ+≤

−∪+≤

∪≤

−

∈−−−

−

∑−

)(

)(}){()(

)()( )

1

\111

1**

1*

Q

Optimal Solution

∑−

=

1

1

j

iiρ

∑=

=k

iikTf

1)( ρ

Approximate Maximization

),...,1( 0 kjj =≥ρ

111:ˆ −

⎟⎠⎞

⎜⎝⎛ −=

j

j kkηρ

),...,1( kj =

⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢

⎣

⎡

≥

⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢

⎣

⎡

⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢

⎣

⎡

η

ηη

ρ

ρρ

MM

L

OOM

MO

L

kk

kk

2

1

110

100

Minimize ∑=

k

ii

1ρ

subject to

⎟⎠⎞

⎜⎝⎛ −≥

⎥⎥⎦

⎤

⎢⎢⎣

⎡⎟⎠⎞

⎜⎝⎛ −−=∑

= e11111ˆ

1ηηρ

kk

ii k

)(: *Sf=η

∑=

=k

iki Tf

1)(ρ

Feature Selection

nXXY ,...,, 1

AX

)|()(:);( AA XYHYHYXI −=

Random Variables

1XYPredict from subset

2X 3X

Y“Sick”

“Fever” “Rash” “Cough”);( YXI AMaximize

subject to kA ≤|| :X Conditionally Independent

)|()( YXHXH AA −= Submodular

Krause & Guestrin (2005)

Submodular Welfare Problem

(Monotone, Submodular)kff ,...,1Utility Functions

)( jiVV ji ≠=∩ φ

∑=

k

iiVf

1)(Maximize

subject tokVVV ∪∪= L1

e)11( − -ApproximationVondrák (2008)

Approximating Submodular Functions

X

)(XfAlgorithm Evaluation

Oracle

Function f̂ Y)(ˆ Yf

Construct a set function such that

For what function is this possible?

Approximating Submodular Functions

. ),(ˆ)()()(ˆ VXXfnXfXf ⊆∀≤≤ α

α

,0)( =φfAssumption . 0,)( VXXf ⊆∀≥Problem

f̂

1)( =nαnn =)(α

Remarksfor cut capacity functions

for general monotonesubmodular functions

Approximating Submodular Functions

• Algorithm with for matroid rank functions.

• Algorithm with for monotone submodular functions.

• No polynomial algorithm can achieve a factor better than even for matroid rank functions.

)log()( nnOn =α

)log()( nnn Ω=α

1)( += nnα

Goemans, Harvey, Iwata, Mirrokni (2009)

Submodular Load Balancing

?)(maxmin},...,{ 1

jjjVVVf

m

∑∈

=Xv

jvj pXf :)(

:,...,1 mffSvitkina & Fleischer (2008)

Monotone Submodular Functions

Scheduling

2-Approximation AlgorithmLenstra, Shmoys, Tardos (1990)

)log( nnO -Approximation Algorithm

Pointers

• S. Fujishige: Submodular Functions and Optimization, Elsevier, 2005.

• ICML 2008 Tutorial Session http://www.submodularity.org

• 永野清仁，河原吉伸，岡本吉央：離散凸最適化手法による機械学習の諸問題へのアプローチ，回路とシステム軽井沢ワークショップ，2009.