1

OR IIOR IIGSLM 52800GSLM 52800

2

OutlineOutline

separable programming

quadratic programming

3

Separable ProgramsSeparable Programs

a separable NLP if f and all gj are separable functions

0 xi i, a finite number1

( ) ( )n

i ii

f f x

x1

( ) ( )n

j ji ii

g g x

x

4

Idea of Separable ProgramIdea of Separable Program

min f(x), s.t. gj(x) 0 for j = 1, …, m. hard NLP but simple LP problems approximating a separable NL

program by a LP a non-linear function by a piecewise

linear one

5

A Fact About Convex FunctionsA Fact About Convex Functions

f: a convex function for any > 0, possible to find a

sequence of piecewise linear convex functions fn

such that |f fn|

6

Example 6.1Example 6.1

a separable program

21 1 2

1 2

1 2

1 2

min 2 ,

. . 2 5,

2x 9,

, 0.

x x x

s t x x

x

x x

7

Example 6.1Example 6.1

approximating by a piecewise linear function

two representations, -form and -form

21 1 2

1 2

1 2

1 2

min 2 ,

. . 2 5,

2x 9,

, 0.

x x x

s t x x

x

x x

21x

points O A B C

x1 0 2 4 4.5

y 0 4 16 20.25

1

C

B

A

O

20.25

16

9

4

4321

8

FormForm

a piecewise linear function with (segment) break points

any point = the convex combination of the two break points of the linear segment

i ( 0) = the weight of break point i

1

C

B

AO

20.2516

94

4321

1 0 2 4 4.5 ,

0 4 16 20.25 ,

1,

at most two adjacent take non-zero values.

O A B C

O A B C

O A B C

i

x

y

9

Example 6.1Example 6.1

the program becomes

1 2

1 2

1 2

1

1 2

min 2 ,

. . 2 5,

2 9,

0 2 4 4.5 ,

0 4 16 20.25 ,

1,

at most two adjacent take non-zero values,

, , ,

O A B C

O A B C

O A B C

i

O

y x x

s t x x

x x

x

y

y x x

, , , 0. A B C

the last but one type of constraints is non-linear

21 1 2

1 2

1 2

1 2

min 2 ,

. . 2 5,

2x 9,

, 0.

x x x

s t x x

x

x x

10

FactFact

nonlinear constraint: at most two adjacent i taking non-zero values possible to have only one i = 1

for convex f and gj: no need to have the non-linear constraint non-optimal to have more than two non-zero i, or

two i not adjacent

11

FactFact

non-optimal to have more than two non-zero i, or two i not adjacent

e.g., f being an objective function any convex combination between two non-adjacent break

points being above the piecewise non-linear function

similarly, the point for three or more non-zero I’s lying above the piecewise non-linear function

think about A = 0.3, B = 0.4, and C = 0.3

1

C

B

AO

20.25

16

94

4321

12

FactFact

non-optimal to have more than two non-zero i, or two i not adjacent

e.g., gj being a constraint

gj(0.3A+0.7B) gj(0.3A+0.7C) bj the feasible set of {0.3A+0.7B} is larger than that by {0.3A+0.7C} the solution from {0.3A+0.7C} cannot be minimum

similar argument for three or more non-zero i’s lying above the piecewise non-linear function

C

B

AO

13

Example 6.1Example 6.1

the program becomes a linear program

1 2

1 2

1 2

1

1 2

min 2 ,

. . 2 5,

2 9,

0 2 4 4.5 ,

0 4 16 20.25 ,

1,

, , , , , , 0.

O A B C

O A B C

O A B C

O A B C

y x x

s t x x

x x

x

y

y x x

14

Example 6.2: Non-Convex ProblemExample 6.2: Non-Convex Problem

min f(x),

s.t. 1 x 3.

approximating f(x) by a piecewise linear function y = 0 + 10A + 6B

x = 0 + 2A + 3B

min 10 6 ,

. . 2 3 3,

2 3 1,

1,

at most two adjacent 's can be positive at the same time,

, , 0.

O A B

O A B

O A B

O A B

O A B

s t

15

Example 6.2: Non-Convex ProblemExample 6.2: Non-Convex Problem

adding slack variable s, surplus variable u, and artificial variable a1 and a2:

1

2

min 10 6 ,

. . 2 3 3,

2 3 1,

+ 1,

at most two adjacent 's can be positive at the

O A B

O A B

O A B

O A B

s t s

u a

a

1 2

same time,

, , , , , 0. O A B s a a

16

Example 6.2: Non-Convex ProblemExample 6.2: Non-Convex Problem

1

2

min 10 6 ,

. . 2 3 3,

2 3 1,

+ 1,

at most two adjacent 's can be positive at the

O A B

O A B

O A B

O A B

s t s

u a

a

1 2

same time,

, , , , , 0. O A B s a a

17

Example 6.2: Non-Convex ProblemExample 6.2: Non-Convex Problem

18

Example 6.2: Non-Convex ProblemExample 6.2: Non-Convex Problem

most negative 0

B in basis only A qualified to enter, not O

19

Example 6.2: Non-Convex ProblemExample 6.2: Non-Convex Problem

20

Example 6.2: Non-Convex ProblemExample 6.2: Non-Convex Problem

21

Example 6.2: Non-Convex ProblemExample 6.2: Non-Convex Problem

22

FormForm

again, the last constraint is unnecessary for a convex program

1 2 2 0.5 ,

4 12 4.25 ,

0 , , 1,

there exists > 0 such that = 1 for and = 0 for .

OA AB BC

OA AB BC

OA AB BC

j i i

x

y

i j i j

1

C

B

AO

20.25

16

94

4321

23

Quadratic ProgrammingQuadratic Programming

24

Quadratic Objective Function Quadratic Objective Function & Linear Constraints & Linear Constraints

Langrangian function

T T12

min ( ) + ,

. . ,

f

s t

x c x x Qx

Ax b x 0.

T T12

( , ) + L x c x x Qx + (Ax b)

25

KKT KKT ConditionsConditions

positive definite Q a convex program

a unique global minimum

the KKT sufficient

otherwise, KKT necessary

26

KKT KKT ConditionsConditions

cT + xTQ + TA 0 Qx + A y = c

Ax b 0 Ax + v = b

xT(c + Qx + A) = 0 xTy = 0

T(Ax b) = 0 Tv = 0

x 0, 0, y 0, v 0

solving the set of equations phase-1 of a linear program

27

Example 7.1Example 7.1(Example 10.14 of JB)(Example 10.14 of JB)

2 21 2 1 2

1 2

1

1 2

min ( ) 8 16 4 ,

. . 5,

3,

, 0.

f x x x x

s t x x

x

x x

x

8

16

c2 0

0 8

Q

1 1

1 0

A

5

3

b

28

Example 7.1Example 7.1(Example 10.14 of JB)(Example 10.14 of JB)

KKT conditions

2x1 + 1 + 2 y1 = 8,

8x2+ 1 y2 = 16,

x1+ x2 + v1 = 5,

x1 + v2 = 3.

x1y1 = x2y2 = 1v1 = 2v2 = 0

x1, y1, x2, y2, 1, v1, 2, v2 0

29

Example 7.1Example 7.1(Example 10.14 of JB)(Example 10.14 of JB)

30

Example 7.1(Example 10.14 of JB)