notation or-1 2015 1 2 3 4 backgrounds or-1 2015 5 convex sets nonconvex set
TRANSCRIPT
Notation
: the set of real numbers
: the set of vectors with real components
: the subset of of vectors whose components are all
: the set of integers
: the set of nonnegative integers
: the vector of with components . All vectors are assumed to be column vec-tors unless otherwise specified.
, or : the inner product of and , .
: Euclidean norm of the vector , .
: every component of the vector is larger than or equal to the corresponding component of .
: every component of the vector is larger than the corresponding component of .
OR-1 2015 1
(continued)
, or : transpose of matrix
rank(): rank of matrix
: the empty set (without any element)
: the set consisting of three elements and
: the set of elements such that …
: is an element of the set
: is not an element of the set
: is contained in (and possibly )
: is strictly contained in
: the number of elements in the set , the cardinality of
: the union of the sets and
: the intersection of the sets and
, or : the set of the elements of which do not belong to
OR-1 2015 2
(continued)
such that: there exists an element such that
such that: there does not exist an element such that
: for any element of …
(P) (Q): the property (P) implies the property (Q). If (P) holds, then (Q) holds. (P) is sufficient condition for (Q). (Q) is necessary condition for (P).
(P) (Q): the property (P) holds if and only if the property (Q) holds
, or : graph which consists of the set of nodes and the set of arcs (directed)
, or : graph which consists of the set of nodes and the set of edges (undi-rected)
: maximum value of the numbers and
: the element among which attains the value
OR-1 2015 3
OR-1 2015 4
Backgrounds
Def: line segment joining two points is the collection of points .
(same as
(Generally, , called convex combination)
1x
2x
1x
2x
21212 )1()( xxxxx
)( 21 xx
OR-1 2015 6
Def: The convex hull of a set is the set of all points that are convex combina-tions of points in S, i.e.
conv(S) =
Picture: , for all i, .
(assuming )
x
y
z
OR-1 2015 7
Proposition: Let be a convex set and for , define .
Then is a convex set.
Pf) If , is convex. Suppose .
For any , .
Then .
But , hence .
Hence the property of convexity of a set is preserved under scalar multi-plication.
Consider other operations that preserve convexity.
OR-1 2015 8
Convex function Def: Function is called a convex function if for all and , satisfies , .
Also called strictly convex function if satisfies
, .
1x 2x21 )1( xx
))1(( 21 xxf
)()1()( 21 xfxf
111 ))(,( nRxfx ))(,( 22 xfx
𝑥 (𝑅𝑛)
𝑓 (𝑥 )(𝑅)
Meaning: The line segment joining and is above or on the locus of points of function values.
OR-1 2015 9
OR-1 2015 10
Def: Let . Define epigraph of as epi. Equivalent definition of convex function: is a convex function if and only if epi
is a convex set.
Def: is a concave function if is a convex function.
Def: is an extreme point of a convex set if x cannot be expressed as for dis-tinct
(equivalently, x does not lie on any line segment that joins two other points in the set C.)
: extreme points
OR-1 2015 11
Review-Linear Algebra 2052 4321 xxxx
30545 4321 xxxx
20263 4321 xxxx
2613
5451
1512
A
4
3
2
1
x
x
x
x
x
20
30
20
b
notation vectormatrix,in bAx
inner product of two column vectors : .
If , then are said to be orthogonal. In 3-D, the angle between the two vec-tors is 90 degrees.
( Vectors are column vectors unless specified otherwise. But, our text does not differentiate it.)
OR-1 2015 12
Submatrices multiplication
2221
1211
34333231
24232221
14131211
AA
Aa
aaaa
aaaa
aaaa
A
2
1
4
3
2
1
B
b
b
b
b
b
B
222121
212111
BAbA
BAbaAB
OR-1 2015 13
submatrix multiplications which will be frequently used.
'
'
'
2
1
21
21
22221
11211
m
n
mnmm
n
n
a
a
a
AAA
aaa
aaa
aaa
A
bxAxAxAxA
x
x
x
x
AAAbAx jnj jnnn
12211
4
3
2
1
21
nn AyAyAyAAAyAy ''''' 2121
mi iimm
m
m 'ay'ay'ay'ay
'a
'a
'a
yyyA'y 122112
1
21
:
OR-1 2015 14
Def: is said to be linearly dependent if , not all equal to 0, such that .
( i.e., there exists a vector in which can be expressed as a linear combination of the other vectors. )
Def: linearly independent if not linearly dependent.
In other words, implies for all .
(i.e., none of the vectors in can be expressed as a linear combination of the re-maining vectors.)
Def: Rank of a set of vectors : maximum number of linearly independent vectors in the set.
Def: Basis for a set of vectors : collection of linearly independent vectors from the set such that every vector in the set can be expressed as a linear combination of them. (maximal linearly independent subset, minimal generators of the set)
OR-1 2015 15
Thm: r linearly independent vectors form a basis if and only if the set has rank r.
Def: row rank of a matrix : rank of its set of row vectors
column rank of a matrix : rank of its set of column vectors
Thm: for a matrix A, row rank = column rank
Def : nonsingular matrix : rank = number of rows = number of columns. (determinant of a nonsingular matrix?) Otherwise, called singular
Thm: Let be an matrix. Then has a unique solution if and only if is nonsingular.
Thm: If A is nonsingular, then unique inverse exists.
OR-1 2015 16
Simultaneous Linear Equations
Thm: Ax = b has at least one solution if and only if rank(A) = rank( [A, b] )
Pf) ) rank( [A, b] ) rank(A). Suppose rank( [A, b] ) > rank(A).
Then b is linearly independent of the column vectors of A, i,e., b can’t be expressed as a linear combination of columns of A. Hence does not have a solution.
) There exists a basis in columns of A which generates b. So has a solu-tion.
Thm: Suppose matrix , rank(A) = rank ([A, b]) = r. Then Ax = b has a unique solution if (and only if) r = n.Pf) Let be any two solutions of . Then , or . . Since column vectors of are linearly independent, we have for all j. Hence . (Note that m may be greater than n.)
OR-1 2015 17
Thm: Suppose matrix , rank(A) = rank ([A, b]) = . Then has infinitely many solutions if . (In this case, if , some equations are redundant.)
Pf) Let . Then is nonempty.
Suppose that the first rows of are linearly independent. (Otherwise, we rearrange the rows of without loss of generality.)
Consider . Then .
(Pf: Clearly since any element of automatically satisfies the constraints defining . We will show that .
Since rank(, the row space of has dimension and the rows forms a basis of the row space. Therefore, every row of can be expressed in the form , for some scalars .
Let be an element of and note that
,
(continued)Consider now an element of . We will show that it belongs to .For any , , which establish that and .)Let be expressed as , where is the submatrix of which has linearly inde-pendent rows of as its rows ( is of full row rank). Then and has the same set of solutions. Let after permuting the columns of , where is a matrix with rank and full-column rank. Also let , which corresponds to the partition of as .
Then .
We may assign any values to , then and is uniquely determined since is nonsingular. Hence there exist infinitely many solutions. Note that this also provides a proof for the necessity part of the previous theorem.
OR-1 2015 18
OR-1 2015 19
Operations that do not change the solution set of the linear equations
(Elementary row operations)Change the position of the equationsMultiply a nonzero scalar k to both sides of an equationMultiply a scalar k to an equation and add it to another equation
Outline of the proof for the third operation:
Let ,
Show that implies (which means )
implies (which means )
Hence . Solution sets are same.
The operations can be performed only on the coefficient matrix , for .
OR-1 2015 20
Solving systems of linear equations (Gauss-Jordan Elimination, 변수의 치환 ) (will be used in the simplex method to solve LP problems)
2252
103
104
32
21
321
xx
xx
xxx
2252
2042
104
32
32
321
xx
xx
xxx
2252
102
104
32
32
321
xx
xx
xxx
2
102
206
3
32
31
x
xx
xx
2
6
8
3
2
1
x
x
x
OR-1 2015 21
Infinitely many solutions case
2252
103
104
432
421
4321
xxx
xxx
xxxx
2252
20242
104
432
432
4321
xxx
xxx
xxxx
2252
102
104
432
432
4321
xxx
xxx
xxxx
2
102
2026
43
432
431
xx
xxx
xxx
2
63
88
43
42
41
xx
xx
xx
43
42
41
2
36
88
xx
xx
xx
txtxtx
ttx
2 ,36 ,88
get and arbitrary for Assign
321
4
OR-1 2015 22
43
42
41
2
36
88
xx
xx
xx
method)simplex in the used be (will
0 2, 6, 8, Here
solution. basic a called is variablesdependent for the solving and
0 toariablesindepent v thesettingby obtainedsolution thely,Particular
variables.dependent are ,, and t variableindependen is
4321
3214
xxxx
xxxx
OR-1 2015 23
2
102
2026
43
432
431
xx
xxx
xxx
2
63
88
43
42
41
xx
xx
xx
2
123
248
43
32
31
xx
xx
xx
434241 2 ,36 ,88 xxxxxx 343231 2 ,312 ,824 xxxxxx
different istion representabut solutions, ofset same thehave systemsBoth
0,2,6,8.). 4321 xxxxge
OR-1 2015 24
Elementary row operations are equivalent to premultiplying a nonsingular square matrix to both sides of the equations
2252
103
104
32
21
321
xx
xx
xxx
22
10
10
520
031
411
3
2
1
x
x
x
2252
2042
104
32
32
321
xx
xx
xxx
22
10
10
1
11
1
520
031
411
1
11
1
3
2
1
x
x
x
22
20
10
520
420
411
3
2
1
x
x
x
𝐴𝑥=𝑏
OR-1 2015 25
2252
102
104
32
32
321
xx
xx
xxx
2252
2042
104
32
32
321
xx
xx
xxx
22
10
10
1
11
1
520
031
411
1
11
1
3
2
1
x
x
x
22
10
10
1
11
1
1
2/1
1
520
031
411
1
11
1
1
2/1
1
3
2
1
x
x
x
OR-1 2015 26
22
10
10
1
11
1
1
2/1
1
520
031
411
1
11
1
1
2/1
1
3
2
1
x
x
x
22
10
10
1
2/12/1
1
520
031
411
1
2/12/1
1
3
2
1
x
x
x
22
20
10
1
2/1
1
520
420
411
1
2/1
1
3
2
1
x
x
x
OR-1 2015 27
So if we multiply all elementary row operation matrices, we get the ma-trix having the information about the elementary row operations we per-formed
2252
103
104
32
21
321
xx
xx
xxx
2
6
8
3
2
1
x
x
x
22
10
10
111
25.25.2
65.65.7
520
031
411
111
25.25.2
65.65.7
3
2
1
x
x
x
2
6
8
100
010
001
3
2
1
x
x
x
111
25.25.2
65.65.71A
OR-1 2015 28
Finding inverse of a nonsingular matrix .
Perform elementary row operations (premultiply elementary row opera-tion matrices) to make to for some . Then is .
Let the product of the elementary row operation matrices which converts to be denoted by C.
Then
Hence .
OR-1 2015 29
The form we will see in the simplex method.
Consider , where is matrix with rank . Suppose after permuting col-umns of , where is and nonsingular, and is matrix.
Hence is now expressed as .
Now premultiplying on both sides of the equation is equivalent to per-forming elementary row operations on the equations which converts the coefficient matrix to identity matrix.
Let . Then . The solution is called a basic solution which is considered in the simplex method. By different choice of matrix (nonsingular), we may obtain different solution.