· introduction gauge problems optimality conditions and examples a few examples a few ideas on...

IntroductionGauge problems

Optimality conditions and examples

MATHEMATICAL PROGRAMMING (I)

Emilio Carrizosa

Doc-Course, March 2010

Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)

A few examplesA few ideas on convexityGauges

Probabilities approximately proportional to a vector

U = u1, . . . , uN (records)

Each ui : associated ci > 0.

Sought: for each i , πi , with

0 ≤ πi ≤ 1 for all i = 1, 2, . . . ,N.∑Ni=1 πi = n0.

πi ∝ ci for all i = 1, 2, . . . ,N.

min∑N

(πici− k)2

s.t. 0 ≤ πi ≤ 1 ∀i = 1, 2, . . . ,N∑Ni=1 πi = n0

k ≥ 0

U = u1, . . . , uN (records)

πi ∝ ci for all i = 1, 2, . . . ,N.

min∑N

(πici− k)2

s.t. 0 ≤ πi ≤ 1 ∀i = 1, 2, . . . ,N∑Ni=1 πi = n0

k ≥ 0

U = u1, . . . , uN (records)

πi ∝ ci for all i = 1, 2, . . . ,N.

min∑N

(πici− k)2

s.t. 0 ≤ πi ≤ 1 ∀i = 1, 2, . . . ,N∑Ni=1 πi = n0

k ≥ 0

U = u1, . . . , uN (records)

πi ∝ ci for all i = 1, 2, . . . ,N.

min∑N

(πici− k)2

s.t. 0 ≤ πi ≤ 1 ∀i = 1, 2, . . . ,N∑Ni=1 πi = n0

k ≥ 0

U = u1, . . . , uN (records)

πi ∝ ci for all i = 1, 2, . . . ,N.

min∑N

(πici− k)2

s.t. 0 ≤ πi ≤ 1 ∀i = 1, 2, . . . ,N∑Ni=1 πi = n0

k ≥ 0

U = u1, . . . , uN (records)

πi ∝ ci for all i = 1, 2, . . . ,N.

min∑N

(πici− k)2

s.t. 0 ≤ πi ≤ 1 ∀i = 1, 2, . . . ,N∑Ni=1 πi = n0

k ≥ 0

U = u1, . . . , uN (records)

ci: constant.

min∑N

(πici− k)2

s.t. 0 ≤ πi ≤ 1 ∀i = 1, 2, . . . ,N∑Ni=1 πi = n0

k ≥ 0

The Fermat-Weber problem

Let A ⊂ Rn, finite (set of users, asking for service). For eacha ∈ A, let ωa > 0 be the demand power of a.

minx∈Rn

f (x) :=∑a∈A

ωa‖x − a‖

minx∈Rn

f (x) :=∑a∈A

ωa‖x − a‖

minx∈Rn

f (x) :=∑a∈A

ωa‖x − a‖

Discrete p-median problem

Ingredients:

A, finite set of users. Each a ∈ A, with demand ωa.

Finite set F of possible locations to open facilities.

The locations of p identical facilities are sought.

Distance d(a, f ) from user at a and potential location f :known.

Each user goes to his/her nearest open facility.

Objective: minimize the total transportion cost (or,equivalently, the average from a random user to his/hernearest open facility).

Ingredients:

Binary variables

For each f ∈ F , define y(f ) =

1, if plant at f is open0, else

For each f ∈ F , a ∈ A, define variable

x(a, f ) =

1, if a goes to f0, else

Cost for a, if (s)he goes to f : ωad(a, f )

Cost for a : ωa

∑f∈F d(a, f )x(a, f )

Total cost:∑

∑f∈F d(a, f )x(a, f )

)Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)

Binary variables

x(a, f ) =

Cost for a : ωa

∑f∈F d(a, f )x(a, f )

Total cost:∑

∑f∈F d(a, f )x(a, f )

Binary variables

x(a, f ) =

Cost for a : ωa

∑f∈F d(a, f )x(a, f )

Total cost:∑

∑f∈F d(a, f )x(a, f )

Binary variables

x(a, f ) =

Cost for a : ωa

∑f∈F d(a, f )x(a, f )

Total cost:∑

∑f∈F d(a, f )x(a, f )

Binary variables

x(a, f ) =

Cost for a : ωa

∑f∈F d(a, f )x(a, f )

Total cost:∑

∑f∈F d(a, f )x(a, f )

Binary variables

x(a, f ) =

Exactly p plants are open: ∑f ∈F

y(f ) = p

Binary variables

x(a, f ) =

Each user goes to just one facility:∑f ∈F

x(a, f ) = 1 ∀a ∈ A

Binary variables

x(a, f ) =

Nobody can go to closed facilities:

x(a, f ) ≤ y(f ) for all a ∈ A, ∀f ∈ F

min∑

(ωa∑

f ∈F d(a, f )x(a, f ))

s.t.∑

f ∈F y(f ) = p∑f ∈F x(a, f ) = 1 ∀a ∈ A

x(a, f ) ≤ y(f ) ∀a ∈ A, f ∈ Fx(a, f ) ∈ 0, 1 ∀a ∈ A, f ∈ Fy(f ) ∈ 0, 1 ∀f ∈ F

(`∞) Distance to co-hyperplanarity

A ⊂ Rn, finite nonempty.

Aim: perturb as few as possible the points in A so that theperturbed set is co-hyperplanar

For hyperplane H(u, β) := u>x + β = 0, the perturbation is

δ(u, β) = max |u>a+β|‖u‖

H(u, β) is sought minimizing the maximum perturbationδ(u, β)

A ⊂ Rn, finite nonempty.

Aim: perturb as few as possible the points in A so that theperturbed set is co-hyperplanar

For hyperplane H(u, β) := u>x + β = 0, the perturbation is

δ(u, β) = max |u>a+β|‖u‖

H(u, β) is sought minimizing the maximum perturbationδ(u, β)

min(u,b)∈Rn×R, u 6=0

maxa∈A

|u>a + b|‖u‖

min(u,b)∈Rn×R, u 6=0

maxa∈A

|u>a + b|‖u‖

Support Vector Machines

Population: O.Classes: C = −1, 1.

u ∈ O →

xu ∈ Rn ( features )yu ∈ C ( class labels )

I ⊂ O : training sample: individuals u with (xu, yu) known.

I+ := u ∈ I : yu = +1, I− := u ∈ I : yu = −1. Bothassumed to be non-empty

score function f (x) := ω>x + β sought to assign labels toobjects u for which just xu (and not yu!) is known:

x ∈ Rn → f (x) :=

1, if ω>x + β > 0−1, if ω>x + β < 0

u ∈ O →

x ∈ Rn → f (x) :=

1, if ω>x + β > 0−1, if ω>x + β < 0

u ∈ O →

x ∈ Rn → f (x) :=

1, if ω>x + β > 0−1, if ω>x + β < 0

u ∈ O →

x ∈ Rn → f (x) :=

1, if ω>x + β > 0−1, if ω>x + β < 0

u ∈ O →

x ∈ Rn → f (x) :=

1, if ω>x + β > 0−1, if ω>x + β < 0

u ∈ O →

x ∈ Rn → f (x) :=

1, if ω>x + β > 0−1, if ω>x + β < 0

How to chose then one linear classifier?

Halfspace of misclassification

for i ∈ I+ : H(ω, β)− := x ∈ Rn : ω>x + β ≤ 0for i ∈ I− : H(ω, β)+ := x ∈ Rn : ω>x + β ≥ 0

δi (ω, β) := distance from xi to halfspace of misclassification

margin: mini∈I δi (ω, β)

Maximize the margin

maxω∈Rn\0

(mini∈I

δi (ω, β)

):= max

yi (ω

>xi + β)

‖ω‖, 0

Maximize the margin

maxω∈Rn\0

(mini∈I

δi (ω, β)

):= max

yi (ω

>xi + β)

‖ω‖, 0

Maximize the margin

maxω∈Rn\0

(mini∈I

δi (ω, β)

):= max

yi (ω

>xi + β)

‖ω‖, 0

maxω∈Rn\0

yi (ω

>xi + β)

‖ω‖, 0

For (xi : i ∈ I+, xi : i ∈ I−) : separable, equivalent to

maxω∈Rn\0

mini∈I

yi (ω>xi + β)

‖ω‖

maxω∈Rn\0

yi (ω

>xi + β)

‖ω‖, 0

maxω∈Rn\0

mini∈I

yi (ω>xi + β)

‖ω‖

maxω∈Rn\0

yi (ω

>xi + β)

‖ω‖, 0

maxω∈Rn\0

mini∈I

yi (ω>xi + β)

‖ω‖

min ‖ω‖s.t. yi (ω

>xi + β) ≥ 1 ∀i ∈ Iω ∈ Rn, β ∈ R

maxω∈Rn\0

yi (ω

>xi + β)

‖ω‖, 0

maxω∈Rn\0

mini∈I

yi (ω>xi + β)

‖ω‖

Hard-margin SVM

min ω>ωs.t. yi (ω

>xi + β) ≥ 1 ∀i ∈ Iω ∈ Rn, β ∈ R

maxω∈Rn\0

yi (ω

>xi + β)

‖ω‖, 0

maxω∈Rn\0

mini∈I

yi (ω>xi + β)

‖ω‖

Soft-margin SVM

min ω>ω + C∑

i∈I ηi

s.t. yi (ω>xi + β) + ηi ≥ 1 ∀i ∈ I

ω ∈ Rn, β ∈ Rηi ≥ 0 ∀i ∈ I

Probabilities approximately proportional to a vector(revisited)

ci: constant.

min∑N

ci− k)2

s.t. 0 ≤ πi ≤ 1 ∀i∑Ni=1 πi = n0

k ≥ 0

ci: constant.

min maxNi=1

ci− k)2

s.t. 0 ≤ πi ≤ 1 ∀i∑Ni=1 πi = n0

k ≥ 0

(∑Ni=1

ci− k)2

,maxNi=1

ci− k)2)

s.t. 0 ≤ πi ≤ 1 ∀i = 1, 2, . . . ,N∑Ni=1 πi = n0

k ≥ 0

ci: constant.

min∑N

ci− k)2

s.t. 0 ≤ πi ≤ 1 ∀i∑Ni=1 πi = n0

k ≥ 0

ci: constant.

min maxNi=1

ci− k)2

s.t. 0 ≤ πi ≤ 1 ∀i∑Ni=1 πi = n0

k ≥ 0

(∑Ni=1

ci− k)2

,maxNi=1

ci− k)2)

s.t. 0 ≤ πi ≤ 1 ∀i = 1, 2, . . . ,N∑Ni=1 πi = n0

k ≥ 0

ci: constant.

min∑N

ci− k)2

s.t. 0 ≤ πi ≤ 1 ∀i∑Ni=1 πi = n0

k ≥ 0

ci: constant.

min maxNi=1

ci− k)2

s.t. 0 ≤ πi ≤ 1 ∀i∑Ni=1 πi = n0

k ≥ 0

(∑Ni=1

ci− k)2

,maxNi=1

ci− k)2)

s.t. 0 ≤ πi ≤ 1 ∀i = 1, 2, . . . ,N∑Ni=1 πi = n0

k ≥ 0

Summary of examples

1 Probabilities approximately proportional to a vector

2 Fermat-Weber problem

3 Discrete p-median

4 (`∞) distance to co-hyperplanarity

5 Support Vector Machines

6 Probabilities approximately proportional to a vector (revisited)

minx∈X

Summary of examples

3 Discrete p-median

minx∈X

Optimal solution (f : X −→ R)

x∗ : f (x∗) ≤ f (x) ∀x ∈ X

Summary of examples

3 Discrete p-median

minx∈X

ε-Optimal solution (f : X −→ R)

x∗ : f (x∗) ≤ f (x) + ε∀x ∈ X

Summary of examples

3 Discrete p-median

minx∈X

Ideal solution (f : X −→ Rn)

x∗ : fi (x∗) ≤ fi (x) ∀i , ∀x ∈ X

Summary of examples

3 Discrete p-median

minx∈X

Weakly efficient solution (f : X −→ Rn)

x∗ : 6 ∃x ∈ X : fi (x) < fi (x∗)∀i

Summary of examples

3 Discrete p-median

minx∈X

Efficient solution (f : X −→ Rn)

x∗ : 6 ∃x ∈ X :fi (x) ≤ fi (x∗)∀if (x∗) 6= f (x)

The practitioner’s perspective

http://www-neos.mcs.anl.gov

Convexity

Convex sets

S ⊂ Rn : convex if ∀x , y ∈ S , the segment with endpoints x , y isincluded in S :

x , y ∈ S ⇒ (1− λ) x + λy ∈ S ∀λ ∈ [0, 1].

Operations preserving convexity

Given C ⊂ Rn : convex, int(C ), cl(C ) andA · C + µ := Ac + µ : c ∈ C are convex sets.

If Cii∈I are convex in Rn, the sets ∩i∈I Ci ,∏

i∈I Ci ,∑

i∈I Ci

are also convex.

Convexity

Convex sets

S ⊂ Rn : convex if ∀x , y ∈ S , the segment with endpoints x , y isincluded in S :

x , y ∈ S ⇒ (1− λ) x + λy ∈ S ∀λ ∈ [0, 1].

Operations preserving convexity

Given C ⊂ Rn : convex, int(C ), cl(C ) andA · C + µ := Ac + µ : c ∈ C are convex sets.

If Cii∈I are convex in Rn, the sets ∩i∈I Ci ,∏

i∈I Ci ,∑

i∈I Ci

are also convex.

Convex envelop

Given X ⊂ Rn, its convex envelop conv(X ) is the smallest convexset containing X : conv(X ) =

⋂C⊃X ,C :convex C

Example

For E ⊂ Rn : finite not empty,conv(E ) =

∑e∈E λee : λe ≥ 0 ∀e ∈ E ,

∑e∈E λe = 1

Caratheodory

Let X ⊂ Rn. For any x ∈ conv(X ), ∃X ∗ ⊂ X , |X ∗| ≤ n + 1 s.t.x ∈ conv(X ∗).

Convex envelop

Example

∑e∈E λee : λe ≥ 0 ∀e ∈ E ,

∑e∈E λe = 1

Caratheodory

Convex envelop

Example

∑e∈E λee : λe ≥ 0 ∀e ∈ E ,

∑e∈E λe = 1

Caratheodory

K ⊂ Rn, K 6= ∅ : cone if K = R+ · K :

x ∈ K ⇒ λx ∈ K ∀λ ∈ R+.

Any cone contains the origin.

Convex cones

A cone K ⊂ Rn is convex iff K + K = K .

K ⊂ Rn, K 6= ∅ : cone if K = R+ · K :

x ∈ K ⇒ λx ∈ K ∀λ ∈ R+.

Convex cones

K ⊂ Rn, K 6= ∅ : cone if K = R+ · K :

x ∈ K ⇒ λx ∈ K ∀λ ∈ R+.

Convex cones

Normal cone

Given X ⊂ Rn, x ∈ X , the normal cone of X at x is the closedconvex cone NX (x),

NX (x) =

p ∈ Rn : p>(y − x) ≤ 0 ∀y ∈ X.

Normal cone

NX (x) =

p ∈ Rn : p>(y − x) ≤ 0 ∀y ∈ X.

Example

For X = [a, b] ⊂ R, a < b,

NX (x) =

R−, if x = a0, if a < x < bR+, if x = b.

Normal cone

NX (x) =

p ∈ Rn : p>(y − x) ≤ 0 ∀y ∈ X.

Example

For X = x ∈ Rn : Ax ≤ b,

NX (x) =

A>u : u>(Ax − b) = 0, u ≥ 0.

Separation

Given C ⊂ Rn, C 6= ∅, convex, closed, x ∈ Rn \ C , ∃u ∈ Rn s.t.

u>y < u>x ∀y ∈ C .

Minimax theorem

For closed convex sets X ,Y ⊂ Rn, with Y : compact,

minx∈X

maxy∈Y

x>y = maxy∈Y

minx∈X

Separation

Given C ⊂ Rn, C 6= ∅, convex, closed, x ∈ Rn \ C , ∃u ∈ Rn s.t.

u>y < u>x ∀y ∈ C .

Minimax theorem

For closed convex sets X ,Y ⊂ Rn, with Y : compact,

minx∈X

maxy∈Y

x>y = maxy∈Y

minx∈X

Given X ⊂ Rn, its polar X is

p : p>x ≤ 1 ∀x ∈ X.

Examples

For x0 ∈ Rn, x0 6= 0, x0 is a closed halfspace.

For X = x : ‖x‖2 ≤ c, X = x : ‖x‖2 ≤ 1c .

p : p>x ≤ 1 ∀x ∈ X.

Examples

For X = x : ‖x‖2 ≤ c, X = x : ‖x‖2 ≤ 1c .

p : p>x ≤ 1 ∀x ∈ X.

Examples

For X = x : ‖x‖2 ≤ c, X = x : ‖x‖2 ≤ 1c .

Properties

Given X ⊂ Rn, X is closed and convex with 0 ∈ X .

If X ⊂ Y ⊂ Rn, then X ⊃ Y .

Let X : convex, closed, with 0 ∈ X . Then

(X ) = X .0 ∈ int(X ) iff X : bounded.

Convex functions

Epigraph

The epigraph epi(f ) of function f : S −→ R is defined as

epi(f ) = (x , t) ∈ Rn × R : x ∈ S , t ≥ f (x) .

Let S ⊂ Rn : convex. f : S −→ R is convex if epi(f ) is convex.Equivalently, f is convex iff

f ((1− λ)x + λy) ≤ (1− λ)f (x) + λf (y).

Convex functions

Epigraph

The epigraph epi(f ) of function f : S −→ R is defined as

epi(f ) = (x , t) ∈ Rn × R : x ∈ S , t ≥ f (x) .

Let S ⊂ Rn : convex. f : S −→ R is convex if epi(f ) is convex.Equivalently, f is convex iff

f ((1− λ)x + λy) ≤ (1− λ)f (x) + λf (y).

Algebra of convex functions

Any C2 in Rn with PSD Hessian in Rn is convex.

Given f : convex on the convex set S ⊂ Rn,

λf + µ, with λ ≥ 0, µ ∈ R : convexf (Ax + b) : convex, with A ∈ Rm×n, b ∈ Rm

Given f1, f2, convex on the convex S ⊂ Rn, the followingfunctions are also convex:

f1 + f2.maxf1, f2.

Let f : S ⊂ Rn −→ R, convex on the convex S , y g : R −→ R :non-decreasing. Then g f : convex on S .

Algebra of convex functions

Any C2 in Rn with PSD Hessian in Rn is convex.

Given f : convex on the convex set S ⊂ Rn,

λf + µ, with λ ≥ 0, µ ∈ R : convexf (Ax + b) : convex, with A ∈ Rm×n, b ∈ Rm

Given f1, f2, convex on the convex S ⊂ Rn, the followingfunctions are also convex:

f1 + f2.maxf1, f2.

Let f : S ⊂ Rn −→ R, convex on the convex S , y g : R −→ R :non-decreasing. Then g f : convex on S .

Gauges

Let B ⊂ Rn, convex, compact, with 0 ∈ int(B). Define the gaugeγB with unit ball B as

γB(x) = min t ≥ 0 : x ∈ tB .

Properties

γB(x) ≥ 0 ∀x ∈ Rn, γB(x) = 0 iff x = 0.

γB(λx) = λγB(x) ∀x ∈ Rn, λ ∈ R+.

γB(x + y) ≤ γB(x) + γB(y) ∀x , y ∈ Rn.

γB : convex in Rn.

Gauges

γB(x) = min t ≥ 0 : x ∈ tB .

Properties

γB(x) ≥ 0 ∀x ∈ Rn, γB(x) = 0 iff x = 0.

γB(x + y) ≤ γB(x) + γB(y) ∀x , y ∈ Rn.

γB : convex in Rn.

Gauges

γB(x) = min t ≥ 0 : x ∈ tB .

Properties

γB(x) ≥ 0 ∀x ∈ Rn, γB(x) = 0 iff x = 0.

γB(x + y) ≤ γB(x) + γB(y) ∀x , y ∈ Rn.

γB : convex in Rn.

Gauges

γ(x) ≥ 0∀x ∈ Rn

γ(x) = 0 iff x = 0γ(λx) = λγ(x)∀x ∈ Rn, λ ∈ R+

γ(x + y) ≤ γ(x) + γ(y)∀x , y ∈ Rn

convex compactsets, with 0 :interior point

γ x : γ(x) ≤ 1

Gauges

γ(x) ≥ 0∀x ∈ Rn

γ(x + y) ≤ γ(x) + γ(y)∀x , y ∈ Rn

γ x : γ(x) ≤ 1

Gauges

γ(x) ≥ 0∀x ∈ Rn

γ(x + y) ≤ γ(x) + γ(y)∀x , y ∈ Rn

γ x : γ(x) ≤ 1

Symmetric gauges: norms

B : symmetric w.r.t. 0 iff γB : norm in Rn.

γ(x) ≥ 0∀x ∈ Rn

γ(x) = 0 iff x = 0γ(λx) = |λ|γ(x)∀x ∈ Rn

γ(x + y) ≤ γ(x) + γ(y)∀x , y ∈ Rn

compact convexsets , symmetricw.r.t. 0 with 0 in

the interior

γ x : γ(x) ≤ 1

γ(x) ≥ 0∀x ∈ Rn

γ(x + y) ≤ γ(x) + γ(y)∀x , y ∈ Rn

the interior

γ x : γ(x) ≤ 1

γ(x) ≥ 0∀x ∈ Rn

γ(x + y) ≤ γ(x) + γ(y)∀x , y ∈ Rn

the interior

γ x : γ(x) ≤ 1

Norms. Examples

Euclidean (`2): γ(x) =√

Norms. Examples

Manhattan (`1): γ(x) =∑

i |xi |

Norms. Examples

Chebyshev (`∞): γ(x) = maxi|xi |

Norms. Examples

That’s not a gauge!!!

Norms. Examples

Composite norm

Let γ1, . . . , γk : norms in Rn1 , . . . ,Rnk . Let τ : norm in Rk ,no-decreasing in Rk

+. Then,

function ν, defined as

ν(s1, s2, . . . , sk) = τ(γ1(s1), . . . , γk(sk)) :

norm in Rn1 × . . .× Rnk .

for n1 = n2 = . . . = nk = n, function

ν(x) = τ(γ1(x), . . . , γk(x)) :

norm in Rn.

Norms. Examples

Norm of sorted components

Let c : Rn −→ Rn s.t., for each x ∈ Rn, c(x) is the sorting of(|x1|, . . . , |xn|) with c1(x) ≥ c2(x) ≥ . . . ≥ cn(x). Function

x 7−→ γ(x) =∑

ωjcj(x)

is a norm if ω1 ≥ . . . ≥ ωn ≥ 0, ω1 > 0.

Particular cases:

ω1 ω2 . . . ωn γ1 0 . . . 0 `∞1 1 . . . 1 `1λ µ . . . µ `1,∞

Norms. Examples

Norm of sorted components

Let c : Rn −→ Rn s.t., for each x ∈ Rn, c(x) is the sorting of(|x1|, . . . , |xn|) with c1(x) ≥ c2(x) ≥ . . . ≥ cn(x). Function

x 7−→ γ(x) =∑

ωjcj(x)

is a norm if ω1 ≥ . . . ≥ ωn ≥ 0, ω1 > 0.

Particular cases:

ω1 ω2 . . . ωn γ1 0 . . . 0 `∞1 1 . . . 1 `1λ µ . . . µ `1,∞

Skewed gauge

Given p ∈ Rn, ‖p‖2 < 1, define γ as

γ(x) = ‖x‖2 + p>x .

p = (0, 0)

p = (0.3, 0) p = (0.5, 0)

Skewed gauge

γ(x) = ‖x‖2 + p>x .

p = (0, 0) p = (0.3, 0)

p = (0.5, 0)

Skewed gauge

γ(x) = ‖x‖2 + p>x .

p = (0, 0) p = (0.3, 0) p = (0.5, 0)

Skewed gauge

M. Cera, J.A. Mesa, F.A. Ortega, F. Plastria

”Locating a central hunter on the plane”

JOTA 136 (2008) 155–166.

P. Chaudhuri

”On a geometric notion on quantiles for multivariate data”

Journal of the American Statistical Association 91 (1996) 862–872.

F. Plastria

”On destination optimality in asymmetric distance Fermat-Weberproblems”

Annals of Operations Research 40 (1992) 355–369.

Skewed gauge

Skewed gauge in Rn :

Given p ∈ Rn, ‖p‖2 < 1, define γ :

γ(x) = ‖x‖2 + p>x .

Skewed gauge

Skewed gauge in R :

Given p ∈ R, |p| < 1, define γ :

γ(x) = |x |+ px =

(1 + p)x , si x > 0,−(1− p)x , si x ≤ 0

Skewed gauge

Skewed gauge in R :

Given p ∈ R, |p| < 1, define γ :

γ(x) = |x |+ px =

(1 + p)x , si x > 0,−(1− p)x , si x ≤ 0

Polyhedral gauge

Let E ⊂ Rn : finite, with 0 ∈ conv(E ), affine(E ) = Rn (or,equivalently, 0 ∈ int(conv(E )). Then, the gauge γ,

γ(x) = maxe∈E

e>x = maxe∈conv(E)

is a polyhedral gauge.

Polyhedral gauge

Let E ⊂ Rn : finite, with 0 ∈ conv(E ), affine(E ) = Rn (or,equivalently, 0 ∈ int(conv(E )). Then, the gauge γ,

γ(x) = maxe∈E

e>x = maxe∈conv(E)

is a polyhedral gauge.

Polyhedral gauge

Let E ⊂ Rn : finite, and let γ(x) = maxe∈E e>x . Then γ is a polyhedralgauge iff it holds:

the following LP has optimal value 0

max∑

e∈E λe

s.t. 0 =∑

e∈E λeeλe ≥ 0 ∀e ∈ E .

∀i = 1, 2, . . . , n, δ ∈ −1, 1, the following LP has strictly positiveoptimal value

min zs.t. z ≥ e>x ∀e ∈ E

xi = δx ∈ Rn.

Polyhedral gauge

max∑

e∈E λe

s.t. 0 =∑

xi = δx ∈ Rn.

Polyhedral gauge

max∑

e∈E λe

s.t. 0 =∑

xi = δx ∈ Rn.

Dual of a gauge

Dual gauge

Let γB : gauge, with unit ball B. Its dual γB is

γB(x) = maxp∈B

Dual of a polyhedral gauge

γ(x) = maxe∈E e>x :

γ(x) = maxγ(u)≤1

= maxu>x : e>u ≤ 1∀e ∈ E= min

∑e∈E

λe :∑e∈E

λee = x , λe ≥ 0 ∀e ∈ E

Dual of a gauge

Dual gauge

Let γB : gauge, with unit ball B. Its dual γB is

γB(x) = maxp∈B

Dual of a polyhedral gauge

γ(x) = maxe∈E e>x :

γ(x) = maxγ(u)≤1

= maxu>x : e>u ≤ 1∀e ∈ E= min

∑e∈E

λe :∑e∈E

λee = x , λe ≥ 0 ∀e ∈ E

Cauchy-Schwarz inequality

γB is a gauge with unit ball B :

γB = γB

((γB))

= γB = γB

γB(x) = ((γB))

(x) = maxu∈B

u>x = maxu 6=0

γB(u)

γB = γB

((γB))

= γB = γB

γB(x) = ((γB))

(x) = maxu∈B

u>x = maxu 6=0

γB(u)

γB = γB

((γB))

= γB = γB

γB(x) = ((γB))

(x) = maxu∈B

u>x = maxu 6=0

γB(u)

Cauchy-Schwarz

γB(x)γB(u) ≥ x>u ∀x , u

γB = γB

((γB))

= γB = γB

γB(x) = ((γB))

(x) = maxu∈B

u>x = maxu 6=0

γB(u)

Cauchy-Schwarz for Euclidean norm . . .

‖x‖2‖u‖2 ≥ x>u ∀x , u

Convex functions, continuity and differentiability

Let f : convex in the open convex set S ⊂ Rn.

f is continuous in S .

The set of points of non-differentiability of f has Lebesguemeasure 0.

Subdiferentiability

Let f : convex in the convex set S ⊂ Rn, and x0 ∈ S : punt ofdifferentiability of f . Then

f (x) ≥ f (x0) +∇f (x0)>(x − x0) ∀x ∈ S .

Let f : convex in the convex set S ⊂ Rn. Given x0 ∈ S , thesubdifferential ∂f (x0) of f at x0 is defined as

∂f (x0) =

p : f (x) ≥ f (x0) + p>(x − x0) ∀x.

p ∈ ∂f (x0) iff the affine function x 7−→ f (x0) + p>(x − x0) is alower bound of f .

∂f (x) is convex, closed and non-empty for all x ∈ int(S)

Subdiferentiability

f (x) ≥ f (x0) +∇f (x0)>(x − x0) ∀x ∈ S .

∂f (x0) =

p : f (x) ≥ f (x0) + p>(x − x0) ∀x.

Subdiferentiability

f (x) ≥ f (x0) +∇f (x0)>(x − x0) ∀x ∈ S .

∂f (x0) =

p : f (x) ≥ f (x0) + p>(x − x0) ∀x.

Subdiferentials and directional derivatives

Let f : convex in the open convex S ⊂ Rn. For x0 ∈ S , d ∈ Rn,d 6= 0, ∃∇f (x0; d), the directional derivative of f at x0 in thedirection d . Moreover, it holds:

∇f (x0; d) = maxp∈∂f (x0)

Subdiferential calculus

If f is convex and differentiable at x0, then

∂f (x0) = ∇f (x0)

Let f1, f2, . . . , fk : convex in the open convex S ⊂ Rn. For x ∈ S ,

∂ (f1(x) + f2(x) + . . .+ fk(x)) = ∂f1(x)+∂f2(x)+. . .+∂fk(x).

∂max1≤i≤k fi (x) = conv(⋃

i∈A(x) ∂fi (x), conA(x) = j : fj(x) = max1≤i≤k fi (x)

Subdiferential calculus

If f is convex and differentiable at x0, then

∂f (x0) = ∇f (x0)

Let f1, f2, . . . , fk : convex in the open convex S ⊂ Rn. For x ∈ S ,

∂ (f1(x) + f2(x) + . . .+ fk(x)) = ∂f1(x)+∂f2(x)+. . .+∂fk(x).

∂max1≤i≤k fi (x) = conv(⋃

i∈A(x) ∂fi (x), conA(x) = j : fj(x) = max1≤i≤k fi (x)

Example

Let E ⊂ Rn, finite, aee∈E , f (x) = maxe∈E e>(x − ae).

∂f (x) = conv(

e : f (x) = e>(x − ae))

Let f (t) = |t − a| = maxt − a, a− t. Then

∂f (t) =

−1, si t < aconv(−1, 1) = [−1, 1], si t = a1, si t > a

Example

Let E ⊂ Rn, finite, aee∈E , f (x) = maxe∈E e>(x − ae).

∂f (x) = conv(

e : f (x) = e>(x − ae))

Let f (t) = |t − a| = maxt − a, a− t. Then

∂f (t) =

−1, si t < aconv(−1, 1) = [−1, 1], si t = a1, si t > a

Composition with affine functions

Let f convex in Rm, and let A ∈ Rm×n, b ∈ Rm. Let g be theconvex function in Rn g(x) = f (Ax + b). Then

∂g(x) = A>∂f (y)|y=Ax+b

Subdifferential of a gauge

∂γ(x) =

u ∈ B : u>x = max

u∈Bu>x

∂γ(0) = B

For x 6= 0, ∂γ(x) is an exposed face of B :

p ∈ ∂γ(x) iff γ(p) = 1, p>x = γ(x).

γ(x) = maxp∈∂γ(x)

Examples

∂‖x‖2 =

1‖x‖2 x , si x 6= 0

y : ‖y‖2 ≤ 1, si x = 0

Let p ∈ Rn, ‖p‖2 < 1, γ(x) = ‖x‖2 + p>x . Then

∂γ(x) =

1‖x‖2 x + p, si x 6= 0

y + p : ‖y‖2 ≤ 1, si x = 0

Subdifferentials of monotonic norms

Let γ : norm, monotonic in Rn+. Given x ∈ Rn

1 If x ∈ Rn++, then ∂γ(x) ⊂ Rn

2 If x ∈ Rn+, then ∂γ(x) ∩ Rn

+ 6= ∅.3 If p ∈ Rn

+, then γ(p) = maxγ(x)≤1,x∈Rn+

4 γ is monotonic in Rn+.

Dual of a composite gauge

Let γ1, . . . , γk : gauges in Rn1 , . . . ,Rnk y τ : gauge in Rk ,non-decreasing in Rk

+. The composite gauge,ν(s1, s2, . . . , sk) = τ(γ1(s1), . . . , γk(sk)), has as dual

ν(s1, . . . , sk) = τ(γ1(s1), . . . , γk(sk))

Problem statementDuality

inf γ(Cx + c) + d>xs. t. x ∈ K

γ : gauge in IRm

C : IRm×n matrix

c ∈ IRm, d ∈ IRn

K = M + E ⊆ IRn nonempty asymptotically conical set witha. c. r. (M,E ).

d = 0 : gauge- or homogeneous programming, e.g. Freund,Math Prog, 1987. Glassey, Math Prog, 1976, Gwinner, JOTA,1985, . . .

K = IRn, Kaplan & Yang, Math Prog, 1997.

General: Carrizosa & Fliege, Math Prog, 2002.

inf γ(Cx + c) + d>xs. t. x ∈ K

γ : gauge in IRm

C : IRm×n matrix

c ∈ IRm, d ∈ IRn

K = M + E ⊆ IRn nonempty asymptotically conical set witha. c. r. (M,E ).

d = 0 : gauge- or homogeneous programming, e.g. Freund,Math Prog, 1987. Glassey, Math Prog, 1976, Gwinner, JOTA,1985, . . .

K = IRn, Kaplan & Yang, Math Prog, 1997.

General: Carrizosa & Fliege, Math Prog, 2002.

S ⊆ IRn,S 6= ∅ : asymptotically conical set (a.c.s.) if has the form

S = M + E ,

M : compact convex

E : closed convex cone

(M,E ) : asymptotically conical representation (a. c. r.) of S .

If (M,E ) : a. c. r. of K , then K∞ = E

Let (M1,E1), (M2,E2) : a. c. r. of K1,K2 ⊆ IRn. Then,

(M1 ×M2,E1 × E2) : a. c. r. of K1 × K2

(M1 + M2,E1 + E2) : a. c. r. of K1 + K2

Let A(x) = Ax + b. Then, (AM + b,AE ) : a.c.r. of A(K )

S ⊆ IRn,S 6= ∅ : asymptotically conical set (a.c.s.) if has the form

S = M + E ,

M : compact convex

E : closed convex cone

(M,E ) : asymptotically conical representation (a. c. r.) of S .

If (M,E ) : a. c. r. of K , then K∞ = E

Let (M1,E1), (M2,E2) : a. c. r. of K1,K2 ⊆ IRn. Then,

(M1 ×M2,E1 × E2) : a. c. r. of K1 × K2

(M1 + M2,E1 + E2) : a. c. r. of K1 + K2

Let A(x) = Ax + b. Then, (AM + b,AE ) : a.c.r. of A(K )

A.c.s.:

compact setspolyhedraaffine spacesclosed convex cones. . .

Class of a.c.s. not closed under intersections

The inverse image of an a.c.s. under an affine mapping: notnecessarily a.c.s.

Duality

(P)︷︸︸︷infx∈K

(γ(Cx + c) + d>x

(D)︷︸︸︷max

γ(u)≤1

(u>c + inf

x∈Kx>(C>u + d

S∗ := x ∈ IRn | x>s ≥ 0 for all s ∈ Sδ∗S(x) := sup

x>y | y ∈ S

inf γ(Cx + c) + d>x = max u>c − δ∗M(−C>u − d)s.t. x ∈ M + E s.t. C>u + d ∈ E ∗

γ(u) ≤ 1

Duality

(γ(Cx + c) + d>x

(D)︷︸︸︷max

γ(u)≤1

(u>c + inf

x∈Kx>(C>u + d

x>y | y ∈ S

γ(u) ≤ 1

Duality

(γ(Cx + c) + d>x

(D)︷︸︸︷max

γ(u)≤1

(u>c + inf

x∈Kx>(C>u + d

x>y | y ∈ S

γ(u) ≤ 1

The unconstrained case: K = IRn

inf γ(Cx + c) + d>x = max u>c − δ∗M(−C>u − d)s.t. x ∈ M + E s.t. C>u + d ∈ E∗

γ(u) ≤ 1

M := 0E = IRn

max u>cs.t. C>u + d = 0

γ(u) ≤ 1

Application: distance to a cone

γ(u) ≤ 1

S : closed convex cone

x0 ∈ IRn

‖ · ‖ norm with unit ball B

infx∈S ‖x − x0‖

γ : ‖ · ‖C = I , c = −x0, d = 0

M = 0, E = S

infx∈S‖x − x0‖ = δ∗S∗∩B(−x0)

Application: distance to a cone

γ(u) ≤ 1

S : closed convex cone

x0 ∈ IRn

‖ · ‖ norm with unit ball B

infx∈S ‖x − x0‖

γ : ‖ · ‖C = I , c = −x0, d = 0

M = 0, E = S

infx∈S‖x − x0‖ = δ∗S∗∩B(−x0)

Application: distance to a hyperplane

γ(u) ≤ 1

H = x ∈ IRn : ω>(x − p) = 0

‖ · ‖ gauge

infx∈H ‖x − x0‖

γ : ‖ · ‖

C = I , c = −x0, d = 0

M = p, E = x : ω>x = 0

infx∈H‖x − x0‖ = max

ω>(p − x0)

‖ω‖,ω>(x0 − p)

‖ − ω‖

Application: distance to a hyperplane

γ(u) ≤ 1

H = x ∈ IRn : ω>(x − p) = 0

‖ · ‖ gauge

infx∈H ‖x − x0‖

γ : ‖ · ‖

C = I , c = −x0, d = 0

M = p, E = x : ω>x = 0

infx∈H‖x − x0‖ = max

ω>(p − x0)

‖ω‖,ω>(x0 − p)

‖ − ω‖

Application: single facility location

minx∈IRn

γ(γ1(x−a1), . . . , γm(x−am)) γ(u1, . . . , um) := γ(γ1(u1), . . . , γm(um)

minx∈IRn

...In×n

...−am

γ(u) ≤ 1

minx∈IRn

...In×n

...−am

γ(u) ≤ 1

minx∈IRn

...In×n

...−am

γ(u) ≤ 1

max −∑m

j=1 u>j aj

s.t.∑m

j=1 uj = 0n×1

γ(u1, . . . , um) ≤ 1

minx∈IRn

...In×n

...−am

γ(u) ≤ 1

max −∑m

j=1 u>j aj

s.t.∑m

j=1 uj = 0n×1

γ(γ1 (u1), . . . , γm(um)) ≤ 1

The linearly-constrained case: K = x : Ax ≥ b

infx∈Ax≥b

(γ(Cx + c) + d>x

)= maxγ(u)≤1

(c>u + inf

Ax≥bx>(C>u + d

max c>u + b>vs.t. −C>u + A>v = d

γ(u) ≤ 1v ≥ 0

The linearly-constrained case: K = x : Ax ≥ b

infx∈Ax≥b

(γ(Cx + c) + d>x

)= maxγ(u)≤1

(c>u + inf

Ax≥bx>(C>u + d

γ(u) ≤ 1v ≥ 0

Application: Support Vector Machines

min γ(Cx + c) + d>xs.t. Ax ≥ b

γ(u) ≤ 1v ≥ 0

min ν

((In×n, 01×n)

y1x>1 y1

y2x>2 y2

......

ymx>m ym

11...1

γ(u) ≤ 1v ≥ 0

min ν

((In×n, 01×n)

y1x>1 y1

y2x>2 y2

......

ymx>m ym

11...1

γ(u) ≤ 1v ≥ 0

min ν

((In×n, 01×n)

y1x>1 y1

y2x>2 y2

......

ymx>m ym

11...1

∑mj=1 vj

s.t.∑m

j=1 vjyj = 0∑mj=1 vjyjxj = u

ν(u) ≤ 1v ≥ 0

γ(u) ≤ 1v ≥ 0

min ν

((In×n, 01×n)

y1x>1 y1

y2x>2 y2

......

ymx>m ym

11...1

∑mj=1 vj

s.t.∑m

j=1 vjyj = 0

ν(∑m

j=1 vjyjxj) ≤ 1

v ≥ 0

γ(u) ≤ 1v ≥ 0

min ν

((In×n, 01×n)

y1x>1 y1

y2x>2 y2

......

ymx>m ym

11...1

min ν(a+ − a−)s.t. a+ ∈ conv (xi : i ∈ I +)

a− ∈ conv (xi : i ∈ I−)

SVM dual and closest pairs

1/2 a+* +1/2 a-

Unconstrained problemsUnconstrained problems

The set of optimal solutions of a convex problem

Let f : convex in the closed convex set S ⊂ Rn. The set of optimalsolutions to minx∈S f (x) is closed and convex, though it may beempty.

Optimality conditions. Unconstrained convex problems

minx∈Rn

f (x) x∗ : optimal solution iff 0 ∈ ∂f (x∗)

Under differentiability: ∂f (x∗) = ∇f (x∗)

x∗ : optimal solution iff 0 ∈ ∇f (x∗)

Optimality conditions. Unconstrained convex problems

minx∈Rn

f (x) x∗ : optimal solution iff 0 ∈ ∂f (x∗)

Under differentiability: ∂f (x∗) = ∇f (x∗)

x∗ : optimal solution iff 0 ∈ ∇f (x∗)

Example. Fermat-Weber problem in the line

minx∈R

f (x) :=∑a∈A

ωa|x − a|

w.l.o.g.,∑

a∈A ωa = 1

Optimality conditions:

0 ∈ ∂f (x)=

∑a∈A ωa∂|x − a|

a∈A,a<x ωa −∑

a∈A,a>x ωa +∑

a∈A,a=x ωa[−1, 1].

minx∈R

f (x) :=∑a∈A

ωa|x − a|

w.l.o.g.,∑

a∈A ωa = 1

0 ∈ ∂f (x)=

a∈A,a>x ωa +∑

a∈A,a=x ωa[−1, 1].

minx∈R

f (x) :=∑a∈A

ωa|x − a|

w.l.o.g.,∑

a∈A ωa = 1

0 ∈ ∂f (x)=

a∈A,a>x ωa +∑

a∈A,a=x ωa[−1, 1].

for x 6∈ A :

0 =∑

a∈A,a<x

ωa −∑

a∈A,a>x

minx∈R

f (x) :=∑a∈A

ωa|x − a|

w.l.o.g.,∑

a∈A ωa = 1

0 ∈ ∂f (x)=

a∈A,a>x ωa +∑

a∈A,a=x ωa[−1, 1].

for x 6∈ A : ∑a∈A,a<x

ωa =∑

a∈A,a>x

ωa =1

minx∈R

f (x) :=∑a∈A

ωa|x − a|

w.l.o.g.,∑

a∈A ωa = 1

0 ∈ ∂f (x)=

a∈A,a>x ωa +∑

a∈A,a=x ωa[−1, 1].

for x = a0 ∈ A :

−∑

a∈A,a<a0

ωa +∑

a∈A,a>a0

ωa ∈ ωa0 · [−1, 1]

minx∈R

f (x) :=∑a∈A

ωa|x − a|

w.l.o.g.,∑

a∈A ωa = 1

0 ∈ ∂f (x)=

a∈A,a>x ωa +∑

a∈A,a=x ωa[−1, 1].

for x = a0 ∈ A :

−ωa0 ≤ −∑

a∈A,a<a0

ωa +∑

a∈A,a>a0

ωa ≤ ωa0

minx∈R

f (x) :=∑a∈A

ωa|x − a|

w.l.o.g.,∑

a∈A ωa = 1

0 ∈ ∂f (x)=

a∈A,a>x ωa +∑

a∈A,a=x ωa[−1, 1].

for x = a0 ∈ A : ∑a∈A,a≥a0

ωa ≥∑

a∈A,a<a0ωa∑

a∈A,a≤a0ωa ≥

∑a∈A,a>a0

minx∈R

f (x) :=∑a∈A

ωa|x − a|

w.l.o.g.,∑

a∈A ωa = 1

0 ∈ ∂f (x)=

a∈A,a>x ωa +∑

a∈A,a=x ωa[−1, 1].

for x = a0 ∈ A : ∑a∈A,a≥a0

ωa ≥ 12∑

a∈A,a≤a0ωa ≥ 1

minx∈R

f (x) :=∑a∈A

ωa|x − a|Optimal solution: median(s) ofA.

minx∈R

f (x) :=∑a∈A

minx∈R

f (x) :=∑a∈A

Example. Fermat-Weber problem in the line ( skewedgauge)

minx∈R

f (x) :=∑a∈A

ωa (|x − a|+ p(x − a))

w.l.o.g.,∑

a∈A ωa = 1

minx∈R

f (x) :=∑a∈A

ωa (|x − a|+ p(x − a))

w.l.o.g.,∑

a∈A ωa = 1

0 ∈ ∂f (x)=

∑a∈A ωa∂ (|x − a|+ p(x − a))

a∈A,a>x ωa +∑

a∈A,a=x ωa[−1, 1] + p.

minx∈R

f (x) :=∑a∈A

ωa (|x − a|+ p(x − a))

w.l.o.g.,∑

a∈A ωa = 1

for x 6∈ A : ∑a∈A,a<x ωa = 1−p

2∑a∈A,a>x ωa = 1+p

minx∈R

f (x) :=∑a∈A

ωa (|x − a|+ p(x − a))

w.l.o.g.,∑

a∈A ωa = 1

for x = a0 ∈ A : ∑a∈A,a≤a0

ωa ≥ 1−p2∑

a∈A,a≥a0ωa ≥ 1+p

Euclidean Fermat-Weber in Rn

minx∈Rn

f (x) :=∑a∈A

ωa‖x − a‖2

minx∈Rn

f (x) :=∑a∈A

ωa‖x − a‖2

For x 6∈ A,

0 ∈ ∂f (x) = ∇f (x)Optimality at x :

∑a∈A

ωa‖x−a‖2 (x − a) = 0

x =∑

a∈A Ωa(x)a, with Ωa(x) = ωa/‖x−a‖2∑b∈A ωb/‖x−b‖2

minx∈Rn

f (x) :=∑a∈A

ωa‖x − a‖2

For x = a0 ∈ A,

0 ∈ ∂f (x) =∑

a 6=a0

ωa‖x−a‖2 (x − a) +ωa0B (B : unit ball of `2)

Optimality at x = a0: ‖∑

a∈A1ωa0

ωa‖x−a‖2 (x − a)‖2 ≤ 1.

minx∈Rn

f (x) :=∑a∈A

ωa‖x − a‖2

Fermat problem

minx∈R2

‖x − a‖2 + ‖x − b‖2 + ‖x − c‖2

Fermat problem

minx∈R2

‖x − a‖2 + ‖x − b‖2 + ‖x − c‖2

Fermat problem

minx∈R2

‖x − a‖2 + ‖x − b‖2 + ‖x − c‖2

Fermat problem

For x 6∈ a, b, c

0 =x − a

‖x − a‖+

x − b

‖x − b‖+

x − c

‖x − c‖0 = pa + pb + pc

Fermat problem

For x 6∈ a, b, c

0 =x − a

‖x − a‖+

x − b

‖x − b‖+

x − c

‖x − c‖0 = pa + pb + pc

−1 = p>a pb + p>a pc

−1 = p>a pb + p>b pc

−1 = p>a pc + p>b pc

Fermat problem

For x 6∈ a, b, c

0 =x − a

‖x − a‖+

x − b

‖x − b‖+

x − c

‖x − c‖0 = pa + pb + pc

p>a pb = p>a pc = p>b pc = −1

Fermat problem

minx∈R2

‖x − a‖2 + ‖x − b‖2 + ‖x − c‖2

For x ∈ a, b, c, x = a say

0 ∈ ∂ (‖x − a‖+ ‖x − b‖+ ‖x − c‖)

Fermat problem

minx∈R2

‖x − a‖2 + ‖x − b‖2 + ‖x − c‖2

0 ∈ ∂ (‖x − a‖+ ‖x − b‖+ ‖x − c‖)

∥∥∥∥ a− b

‖a− b‖+

a− c

‖a− c‖

∥∥∥∥ ≤ 1

Fermat problem

minx∈R2

‖x − a‖2 + ‖x − b‖2 + ‖x − c‖2

0 ∈ ∂ (‖x − a‖+ ‖x − b‖+ ‖x − c‖)

∥∥∥∥ a− b

‖a− b‖+

a− c

‖a− c‖

∥∥∥∥2

Fermat problem

minx∈R2

‖x − a‖2 + ‖x − b‖2 + ‖x − c‖2

0 ∈ ∂ (‖x − a‖+ ‖x − b‖+ ‖x − c‖)

∥∥∥∥ a− b

‖a− b‖

∥∥∥∥2

∥∥∥∥ a− c

‖a− c‖

∥∥∥∥2

+ 2(a− b)>(a− c)

‖a− b‖‖a− c‖≤ 1

Fermat problem

minx∈R2

‖x − a‖2 + ‖x − b‖2 + ‖x − c‖2

0 ∈ ∂ (‖x − a‖+ ‖x − b‖+ ‖x − c‖)

(a− b)>(a− c)

‖a− b‖‖a− c‖≤ −1

Fermat-Weber problem in Rn. Different norms yielddifferent solutions

γ : `2, . . .

γ : `1, . . .

Center problem in Rn

Let A ⊂ Rn, finite. Let γ : norm

minx∈Rn

f (x) := maxa∈A

γ(x − a)

Optimality at x∗ :

0 ∈ conv (∂γ(x∗ − a) : f (x∗) = γ(x∗ − a))

Caratheodory:

If x∗ : optimal, then ∃A∗ ⊂ A, with cardinality |A∗| ≤ n + 1 s.t.x∗ : optimal for

minx∈Rn

f (x) := maxa∈A∗

γ(x − a)

minx∈Rn

f (x) := maxa∈A

γ(x − a)

0 ∈ conv (∂γ(x∗ − a) : f (x∗) = γ(x∗ − a))

Caratheodory:

minx∈Rn

γ(x − a)

minx∈Rn

f (x) := maxa∈A

γ(x − a)

0 ∈ conv (∂γ(x∗ − a) : f (x∗) = γ(x∗ − a))

Caratheodory:

minx∈Rn

γ(x − a)

Optimality conditions. Constrained convex problems

minx∈S

x∗ : optimal solution iff 0 ∈ ∂f (x∗) + NS(x∗)

Example. Fermat-Weber problem in a segment

minx∈S

f (x) :=∑a∈A

ωa|x − a|

S = [s1, s2], s1 < s2, s1, s2 6∈ A.∑

a∈A ωa = 1

minx∈S

f (x) :=∑a∈A

ωa|x − a|

S = [s1, s2], s1 < s2, s1, s2 6∈ A.∑

a∈A ωa = 1

NS(x) =

R−, si x = s1

0, si s1 < x < s2

R+, si x = s2.

Optimality conditions in x ∈ (s1, s2) : x : median of A

Optimality at xs1 : 0 ∈∑

a<x ωa −∑

a>x ωa + R−.Optimality at xs2 : 0 ∈

∑a<x ωa −

∑a>x ωa + R+.

minx∈S

f (x) :=∑a∈A

ωa|x − a|

S = [s1, s2], s1 < s2, s1, s2 6∈ A.∑

a∈A ωa = 1

NS(x) =

R−, si x = s1

0, si s1 < x < s2

R+, si x = s2.

Optimality conditions in x ∈ (s1, s2) : x : median of A

Optimality at xs1 : 0 ∈∑

a<x ωa −∑

a>x ωa + R−.Optimality at xs2 : 0 ∈

∑a<x ωa −

∑a>x ωa + R+.

minx∈S

f (x) :=∑a∈A

ωa|x − a|

S = [s1, s2], s1 < s2, s1, s2 6∈ A.∑

a∈A ωa = 1

NS(x) =

R−, if x = s1

0, if s1 < x < s2

R+, if x = s2.

Optimality at x ∈ (s1, s2) : x : median of A

Optimality at xs1 :∑

a>x ωa ≤ 12

Optimality at xs2 :∑

a<x ωa ≤ 12

min∑N

ci− k)2

s.t. 0 ≤ πi ≤ 1 ∀i = 1, 2, . . . ,N∑Ni=1 πi = n0

k ≥ 0

min∑N

ci− k)2

s.t. 0 ≤ πi ≤ 1 ∀i = 1, 2, . . . ,N∑Ni=1 πi = n0

k ∈ R

min∑N

ci− k)2

s.t. 0 ≤ πi ≤ 1 ∀i = 1, 2, . . . ,N∑Ni=1 πi = n0

k ∈ R

ci− k)

+ αi − βi − ϑ = 0 ∀i = 1, 2, . . . ,N1N

∑Ni=1

ci− k = 0∑N

i=1 πi = nπi ≤ 1 ∀i = 1, 2, . . . ,Nπi ≥ 0 ∀i = 1, 2, . . . ,N

αi (1− πi ) = 0 ∀i = 1, 2, . . . ,Nβiπi = 0 ∀i = 1, 2, . . . ,Nαi , βi ≥ 0 ∀i = 1, 2, . . . ,N

k , ϑ ∈ REmilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)

min∑N

ci− k)2

s.t. 0 ≤ πi ≤ 1 ∀i = 1, 2, . . . ,N∑Ni=1 πi = n0

k ∈ R

From this, an O(N log(N))-time algorithm yields an optimal solution.

Hard-margin SVM

min 12ω>ω

s.t. yi (ω>xi + β) ≥ 1∀i ∈ I

ω ∈ Rn, β ∈ R

0 ∈ ∂f (ω, β) + NS(ω, β)

Hard-margin SVM

min 12ω>ω

s.t. yi (ω>xi + β) ≥ 1∀i ∈ I

ω ∈ Rn, β ∈ R

0 = ω −∑i∈I

λiyixi

0 =∑i∈I

0 ≤ λi ∀i ∈ I

0 = λi

(yi (ω

>xi + β)− 1)∀i ∈ I

Hard-margin SVM

I +(λ) =

i ∈ I + : λi > 0

I−(λ) =

i ∈ I− : λi > 0

I (λ) = I +(λ) ∪ I−(λ)

xi : i ∈ I (λ) : support vectors at λ

Let (ω∗, β∗) : optimal.

∃ λ ∈ RI multiplier at (ω∗, β∗), with |I (λ)| ≤ n + 1

For λ ∈ RI multiplier in (ω∗, β∗),

I +(λ) 6= ∅, I−(λ) 6= ∅.(ω∗, β∗) is also optimal to

>xi + β) ≥ 1 ∀i ∈ I (λ)

Hard-margin SVM

I +(λ) =

i ∈ I + : λi > 0

I−(λ) =

i ∈ I− : λi > 0

I (λ) = I +(λ) ∪ I−(λ)

xi : i ∈ I (λ) : support vectors at λ

Let (ω∗, β∗) : optimal.

∃ λ ∈ RI multiplier at (ω∗, β∗), with |I (λ)| ≤ n + 1

For λ ∈ RI multiplier in (ω∗, β∗),

I +(λ) 6= ∅, I−(λ) 6= ∅.(ω∗, β∗) is also optimal to

>xi + β) ≥ 1 ∀i ∈ I (λ)

· introduction gauge problems optimality conditions and examples a few examples a few ideas on...

Documents