lecture 8 – nonlinear programming models topics general formulations local vs. global solutions...
TRANSCRIPT
Lecture 8 – Nonlinear Programming Models
Topics
• General formulations
• Local vs. global solutions
• Solution characteristics
• Convexity and convex programming
• Examples
• In LP ... the objective function & constraints are linear and the problems are “easy” to solve.
• Many real-world engineering and business problems have nonlinear elements and are hard to solve.
Nonlinear Optimization
Minimize f(x)
s.t. gi(x) (, , =) bi, i = 1,…,m
x = (x1,…,xn) is the n-dimensional vector of decision variables
f (x) is the objective function
gi(x) are the constraint functions
bi are fixed known constants
General NLP
Example 1 Max f (x) =3x1 + 2x2
4
s.t. x1 + x2 1, x1 0, x2 unrestricted
2
Examples 2 and 3 can be reformulated as LPs
Example 2 Max f (x) = ec1x1 ec2x
2 … ecn xn
s.t. Ax = b, x 0 n
Example 3 Min j =1
fj (xj )
s.t. Ax = b, x 0
where each fj(xj ) is of the form
Problems with“decreasing efficiencies”
fj(xj)
xj
Examples of NLPs
Max f(x1, x2) = x1x2
s.t. 4x1 + x2 8
x1 0, x20
2
8
f(x) = 2
f(x) = 1
x2
Optimal solution will lie on the line g(x) = 4x1 + x2 – 8 = 0.
x1
NLP Graphical Solution Method
• Solution is not a vertex of feasible region. For this particular problem the solution is on the boundary of the feasible region. This is not always the case.
• In a more general case, f (x1, x2) = g (x1,
x2) with 0. (In this case, = 1.)
Gradient of f (x) = f (x1, x2) (f/x1, f/x2)T
This gives f/x1 = x2, f/x2 = x1
and g/x1 = 4, g/x2 = 1
At optimality we have f (x1, x2) = g (x1, x2)
or x1* = 1 and x2
* = 4
Solution Characteristics
f(x)
x
localmin
globalmax stationary
point
localmin
localmax
Let S n be the set of feasible solutions to an NLP.
Definition: A global minimum is any x0 S such than
f (x0) f (x)
for all feasible x not equal to x0.
Nonconvex Function
At (1, 0), f(x1, x2) = 1g1(x1, x2) + 2g1(x1, x2)
or (0, 6) = 1(1, 0) + 2(0, 1), 1 0, 2 0
so 1 = 0 and 2 = 6
If g1 = x1 0 and g2 = x2 0, what is the optimum ?
Function with Unique Global Minimum at x = (1, –3)
Convex function: If you draw a straight line between any two points on f (x) the line will be above or on f (x).
Concave function: If f (x) is convex than – f (x) is concave.
Linear functions are both convex and concave.
Convexity
d2 f (x)dx2
≥ 0 for all x
Convexity condition for univariate f :
x1 x2
f (x)
Definition of Convexity
Let x1 and x2 be two points (vectors) in S n. A function f (x) is convex if and only if
f (x1 + (1–)x2) ≤ f (x1) + (1–)f (x2)
for all 0 < < 1. It is strictly convex if the inequality sign ≤ is replaced with the sign <.
1-dimensional example
f(x)
x1 x2x1+(1−)x2
f(x1+(1−)x2)
f(x1)+(1−)f(x2).
.
..
. . .
d2f
dx12
d2fdx1dx2
. . . d2f
dx1dxn
d2fdx2dx1
d2fdxndx1
. . . d2f
dxn2
Hessian of f at x : 2f (x) =
.
.
.
. . .
.
.
.
A positively weighted sum of convex functions is convex:
If fk(x) is convex for k =1,…,m and 1,…,m 0,
then f (x) = k fk(x) is convex.
m
k =1
Theoretical Result for Convex Functions
Used to determine convexity.
Determining Convexity
One-Dimensional Functions:
A function f (x) C 1 is convex if and only if it is
underestimated by linear extrapolation; i.e.,
f (x2) ≥ f (x1) + (df (x1)/dx)(x2 – x1) for all x1 and x2.
A function f (x) C 2 is convex if and only if its second
derivative is nonnegative.
d2f (x)/dx2 ≥ 0 for all x
If the inequality is strict (>), then f (x) is strictly convex.
x1 x2
f(x)
Multiple Dimensional Functions
Definition: The Hessian matrix H(x) associated with
f (x) is the n n symmetric matrix of second partial
derivatives of f (x) with respect to the components of x.
Example: f (x) = 3(x1)2 + 4(x2)3 – 5x1x2 + 4x1
⎥⎦
⎤⎢⎣
⎡−
−=⎟⎟
⎠
⎞⎜⎜⎝
⎛−
+−=∇
2122
21
245
56)( and
512
456)(
xxx
xxf xHx
When f (x) is quadratic, H(x) has only constant terms;
when f (x) is linear, H(x) does not exist.
f (x) is convex if only if f (x2) ≥ f (x1) + Tf (x1)(x2 – x1) for all x1 and x2.
Properties of the Hessian
•H(x) is positive definite if and only if xTHx > 0 for all x 0.
•H(x) is positive semi-definite if and only if xTHx ≥ 0 for all x and there exists and x 0 such that xTHx = 0.
•H(x) is indefinite if and only if xTHx > 0 for some x, and xTHx < 0 for some other x.
How can we use Hessian to determine whether or not f(x) is convex?
Multiple Dimensional Functions and Convexity
• f (x) is strictly convex (or just convex) if its associated Hessian matrix H(x) is positive definite (semi-definite) for all x.
• f (x) is neither convex nor concave if its associated Hessian matrix H(x) is indefinite
The terms negative definite and negative-semi- definite are also appropriate for the Hessian and provide symmetric results for concave functions. Recall that a function f (x) is concave if –f (x) is convex.
Testing for Definiteness
Definition: The ith leading principal submatrix of H is
the matrix formed taking the intersection of its first i
rows and i columns. Let Hi be the value of the
corresponding determinant:
obtained. is untilon so and , ,2221
12112111 nH
hh
hhHhH ==
Let Hessian, H =
⎥⎥⎥⎥⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢⎢⎢⎢⎢
⎣
⎡
nnnn
n
n
hhh
hhh
hhh
...
..
..
..
...
...
21
22221
11211
, where hij = 2f
(x)/xixj
Rules for Definiteness
• H is positive definite if and only if the determinants of all the leading principal submatrices are positive; i.e., Hi > 0 for i = 1,…,n.
• H is negative definite if and only if H1 < 0 and the remaining leading principal determinants alternate in sign:
H2 > 0, H3 < 0, H4 > 0, . . .
Positive-semidefinite and negative semi-definiteness require that all principal submatrices satisfy the above conditions for the particular case.
Quadratic Functions
Example 1: f (x) = 3x1x2 + x12 +
3x22
⎥⎦
⎤⎢⎣
⎡=⎟⎟
⎠
⎞⎜⎜⎝
⎛++
=∇63
32)( and
63
23)(
21
12 xHxxx
xxf
so H1 = 2 and H2 = 12 – 9 = 3
Conclusion f (x) is convex
because H(x) is positive definite.
Quadratic Functions (cont’d)
Example 2: f (x) = 24x1x2 + 9x12 +
16x22
⎥⎦
⎤⎢⎣
⎡=⎟⎟
⎠
⎞⎜⎜⎝
⎛++
=∇3224
2418)( and
3224
1824)(
21
12 xHxxx
xxf
so H1 = 18 and H2 = 576 – 576 = 0
• Thus H is positive semi-definite (determinants of
all submatrices are nonnegative) so f (x) is
convex.• Note, xTHx = 2(3x1 + 4x2)2 ≥ 0. For x1 = 4, x2 =
3, we get xTHx = 0.
Nonquadratic Functions
Example 3: f (x) = (x2 – x12)2 + (1 – x1)2
⎥⎦
⎤⎢⎣
⎡
−−++−
=24
42124)(
1
1212
x
xxxxH
Thus the Hessian depends on the point under consideration:
At x = (1, 1), which is positive definite.
At x = (0, 1), which is indefinite.
Thus f(x) is not convex although it is strictly convex near (1, 1).
⎥⎦
⎤⎢⎣
⎡−
−=
24
410)1,1(H
2 0(0 1)
0 2,
−⎡ ⎤=⎢ ⎥
⎣ ⎦H