REVIEW OF LINEAR ALGEBRA

INDR 262INTRODUCTION TO OPTIMIZATION METHODS

Metin TürkayDepartment of Industrial Engineering

Koç University, Istanbul

MATRICES

Øm and n are positive integersØOrder of matrix: mxnØThe number in the i th row and jth column of A is

called the ij th element of A and is written aij .

) =+,, +,-+-, +-,

⋯ +,/⋯ +-/

⋮ ⋮+1, +1-

⋱ ⋮⋯ +1/

EXAMPLE

! =1 2 34 5 67 8 9

,-- = 1,./ = 6,/- = 7

EQUAL MATRICES

ØTwo matrices A and B are equal if and only if Aand B are of the same order and for all i and j, aij=bij .

( = 1 23 4 - = . /

0 1

ØIf A=B, then x=1, y=2, w=3, z=4

VECTORS

ØA list of n real numbers, say (a1,a2,…,an) is called an n-dimensional vector. An n-dimensional vector also maybe displayed as a 1 by n matrix.

12 , 1 2 3

SCALAR PRODUCT OF TWO VECTORS

ØThe scalar product of vectors

! = !# !$ ⋯ !& and * =*#*$⋮*&

is !#*# + !$*$ + ⋯ +!&*&

EXAMPLE

uv = 1x2+2x1+3x2 = 10

[ ]úúú

û

ù

êêê

ë

é==

212

321 vu

NOTES

ØIf u = [1 2 3] and , then uv is not defined

because the vectors are of different dimensions.

ØTwo vectors are perpendicular to each other if and only if their scalar product is equal to 0.

e.g., u = [1 -1] and

úû

ùêë

é=43

v

úû

ùêë

é=11

v

SCALAR MULTIPLE OF A MATRIX

ØGiven any matrix A and any scalar c, the scalar multiple of matrix A, cA, is obtained from the matrix A by multiplying each element of A by c.

úû

ùêë

é-

=úû

ùêë

é-

=0363

3 0121

AA

ADDITION OF TWO MATRICES

ØLet A=[aij] and B=[bij] be two matrices with the same

order (say mxn). Then, the matrix C=A+B is defined

to be the mxn matrix whose ijth element is aij+bij.

úû

ùêë

é=ú

û

ùêë

é-+-+---

=+=

úû

ùêë

é----

=úû

ùêë

é-

=

002000

111120332211

112321

110321

BAC

BA

THE TRANSPOSE OF A MATRIX

ØGiven any mxn matrix

ØThe transpose of A (written AT) is the nxm matrix

úúúú

û

ù

êêêê

ë

é

=

mnmm

n

n

aaa

aaaaaa

A

..........

..

..

21

22221

11211

úúúú

û

ù

êêêê

ë

é

=

nmnn

n

n

T

aaa

aaaaaa

A

..........

..

..

21

22212

12111

EXAMPLE

ØFor any matrix A, (AT)T=A.

( ) úû

ùêë

é=

úúú

û

ù

êêê

ë

é=ú

û

ùêë

é=

654321

635241

654321 T TTAAA

MATRIX MULTIPLICATION

ØGiven two matrices A and B, the matrix product of A and B is defined if and only if the number of columns in A is equal to the number of rows in B. The matrix product C=AB is determined as follows:

cij = scalar product of (row i of A and column j of B)

PROPERTIES OF MATRIX MULTIPLICATION

1. Matrix multiplication is associative, i.e., A(BC)=(AB)C.

2. Matrix multiplication is distributive, i.e., A(B+C)=AB+AC.

EXAMPLE

[ ]

[ ]

[ ]

[ ] 531

12

421

12

431

11

321

11

22

21

12

11

=úû

ùêë

é=

=úû

ùêë

é=

=úû

ùêë

é=

=úû

ùêë

é=

c

c

c

c

ABC

BA

=

úû

ùêë

é=ú

û

ùêë

é=

3211

1211

úû

ùêë

é==

5443

ABC

MATRICES AND SYSTEMS OF LINEAR EQUATIONS

ØConsider a system of linear equations given by

Øx1, x2, …, xn are referred to as variablesØaij’s and bi’s are constantsØA set of equations like above is called a linear system

of m equations in n variables

11 1 12 2 1 1

21 1 22 2 2 2

1 1 2 2

...

...... ... ... ... ... ... ... ... ...

...

n n

n n

m m mn n m

a x a x a x ba x a x a x b

a x a x a x b

+ + + =+ + + =

+ + + =

SOLUTION

ØA solution to a linear system of m equations in nunknowns is a set of values for the unknowns that satisfy each of the systems m equations.

EXAMPLE

x1+ 2x2 = 52x1 - x2= 0

úû

ùêë

é=21

x

úû

ùêë

é=13

x

00 0 ? 2)1(2

55 5 ? )2(21

=-

=+

05 0 ? 1)3(2

55 5 ? )1(23

¹-

=+

Solution

Not a solution

MATRIX REPRESENTATION OF SYSTEMS OF LINEAR EQUATIONS

Ax = b11 12 1 1 1

21 22 2 2 2

1 2

...

..., ,

... ... ... ... ... ......

n

n

m m mn n m

a a a x ba a a x b

A x b

a a a x b

é ù é ù é ùê ú ê ú ê úê ú ê ú ê ú= = =ê ú ê ú ê úê ú ê ú ê úë û ë û ë û

[ ]úúúúú

û

ù

êêêêê

ë

é

=

mmnmm

n

n

b

bb

aaa

aaaaaa

bA...

...............

...

...

2

1

21

22221

11211

Augmented matrix

THE GAUSS-JORDAN METHOD FOR SOLVING SYSTEMS OF LINEAR EQUATIONS

ØGauss-Jordan method is used to find solution(s) to systems of linear equations. A system of linear equations must satisfy one of the following cases:§ Case 1: The system has no solution§ Case 2: The system has a unique solution§ Case 3: The system has an infinite number of solutions

ELEMENTARY ROW OPERATIONS

ØAn elementary row operation (ero) transforms a given matrix A into a new matrix A’ via one of the following operations.§ Type 1 ero: A’ is obtained by multiplying any row of A by a

nonzero scalar.§ Type 2 ero: Begin by multiplying any row of A (say, row i) by a

nonzero scalar c. For some j≠i, let row j of A’ = c(row i of A) + row j of A, and let the other rows of A’ be the same as the rows of A.

§ Type 3 ero: Interchange any two rows of A.

FACTS

ØIf the matrix A’ is obtained from A via an ero, A’ and Aare equivalent.

ØIf the augmented matrix [A’Ιb’] is obtained from [AΙb]via an ero, the systems Ax=b and A’x=b’ are equivalent.

ØAny sequence of ero’s performed on the augmented matrix [AΙb] corresponding to the system Ax=b will yield an equivalent linear system.

GAUSS-JORDAN METHOD

ØThe Gauss-Jordan method solves a linear system of equations by utilizing ero’s in a systematic fashion.

Step 1 To solve Ax=b, write down the augmented matrix [AΙb].

Step 2 At any stage, define a current row, current column, and a current entry. Begin with row 1 as the current row, column 1 as the current column, and a11 as the current entry.

GAUSS-JORDAN METHOD

a. If a11 (the current entry) is nonzero, use ero’s to transform column 1(the current column) to [1 0 … 0]T. Then, obtain the new current row, column, and entry by moving down one row and one column to the right, and go to Step 3.

b. If a11 (the current entry) equals 0, then do a Type 3 ero involving the current row and any row that contains a nonzero entry in the current column. Use ero’s to transform column 1 (the current column) to [1 0 … 0]T. Then, obtain the new current row, column, and entry by moving down one row and one column to the right, and go to Step 3.

c. If there are no nonzero numbers in the first column, obtain a new current column and entry by moving one column to the right. Then go to Step 3.

GAUSS-JORDAN METHOD

Step 3a. If the new current entry is nonzero, use ero’s to transform it to 1

and the rest of the current column’s entries to 0. When finished, obtain a new current row, column, and entry. If this is impossible, stop. Otherwise, repeat Step 3.

b. If the current entry is 0, do a Type 3 ero with the current row and any row that contains a nonzero entry in the current column. Then, use ero’s to transform column entry to 1 and the rest of the current column’s entries to 0. When finished, obtain the new current row, column, and entry. If this is impossible, stop. Otherwise, repeat Step 3.

c. If the current column has no nonzero numbers below the current row, obtain a new current column and entry and repeat Step 3. If it is impossible, stop.

GAUSS-JORDAN METHOD

Step 4 Write down the system of equations A’x=b’ that corresponds to the matrix [A’Ιb’] obtained when Step 3 is completed. Then, A’x=b’ will have the same set of solutions as Ax=b.§The Gauss–Jordan method converts the augmented matrix [AΙb]into [A’ Ιb’] such that

[ ]úúúúú

û

ù

êêêêê

ë

é

=

'

'2

'1

...1...00............0...100...01

''

mb

bb

bA ''22

'11 ..., , , mn bxbxbx ===

EXAMPLE

§ Solve the following system of linear equations using the Gauss-Jordan method.

2x1 + 2x2 + x3 = 92x1 - x2 + 2x3 = 6x1 - x2 + 2x3 = 5

SOLUTION

[ ]úúú

û

ù

êêê

ë

é

--=

569

211212122

bA [ ]úúúú

û

ù

êêêê

ë

é

--=

5629

2112122111

11 bA

[ ]úúúú

û

ù

êêêê

ë

é

---=

5329

2111302111

22 bA[ ]úúúú

û

ù

êêêê

ë

é

---=

21329

23201302111

33 bA

SOLUTION

[ ]úúúú

û

ù

êêêê

ë

é

-

-=

21129

232031102111

44 bA [ ]úúúú

û

ù

êêêê

ë

é

-

-=

21127

232031106501

55 bA

[ ]úúúú

û

ù

êêêê

ë

é

-=

25127

650031106501

66 bA[ ]

úúúú

û

ù

êêêê

ë

é

-=3127

10031106501

77 bA

SOLUTION

[ ]úúú

û

ù

êêê

ë

é

=321

100010001

99 bA[ ]úúú

û

ù

êêê

ë

é-=

311

1003110001

88 bA

x1 = 1 x2 = 2 x3 = 3

ANALYSIS OF THE SOLUTIONS TO SYSTEMS OF LINEAR EQUATIONS

For any linear system, a variable that appears with a coefficient of 1 in a single equation and a coefficient of 0 in all other equations is called a basic variable. Any variable that is not a basic variable is called a nonbasic variable.

x set of all of the variables in the system Ax=bxB set of all of the basic variables in the system Ax=bxN set of all of the nonbasic variables in the system Ax=b

x = xBUxN

ANALYSIS OF THE SOLUTIONS TO SYSTEMS OF LINEAR EQUATIONS

ØThe solution to A’x=b’ can be categorized in one of the three cases:

Case 1: A’x=b’ has at least one row of the form [0 0 … 0Ιc] and c≥0. Then, Ax=b has no solution.

Case 2: When case 1 does not apply and xN=x, then Ax=bhas a unique solution.

Case 3: When case 1 does not apply and xN≠x, then Ax=bhas infinite number of solutions.

EXAMPLE

Ø Case 1 does not apply since there are no rows of the form [0 0 … 0Ιc] and c≠0.

Ø Case 2 does not apply since,xB={x1, x2, x3}xN={x4, x5}

Ø There are infinite number of solutions.

[ ]úúúúú

û

ù

êêêêê

ë

é

=

0123

00000101000201011001

'' bA

EXAMPLE

ØAssign arbitrary values to the variables in xN; x4=c, x5=k.ØWrite down the equations in [A’ Ιb’],

x1 + c + k = 3 è x1 = 3 - c – kx2 + 2c = 2 è x2 = 2 - 2cx3 + k = 1 è x3 = 1 - k

Ø It is easy to see that there are infinite number of values of c and kthat will satisfy this system of equations.

[ ]úúúúú

û

ù

êêêêê

ë

é

=

0123

00000101000201011001

'' bA

LINEAR COMBINATION

ØA linear combination of the vectors in V is any vector of the form c1v1+c2v2+…+ckvk, where c1, c2, …, ck are arbitrary scalars.

Example: V={[1,2], [2,1]}

2v1-v2 = 2([1 2]) – [2 1] = [0 3]0v1-3v2 = 0([1 2]) – 3([2 1]) = [6 3]

LINEAR INDEPENDENCE &LINEAR DEPENDENCE

ØA set V of m-dimensional vectors is linearly independent if the only linear combination of vectors in V that equals 0 is the trivial linear combination.

ØA set V of m-dimensional vectors is linearly dependent if there is a nontrivial linear combination of the vectors in V that adds up to 0.

EXAMPLE 1

ØV={[1,0], [0,1]} Try to find a linear combination of vectors in V that yields 0.

Øc1([1 0]) + c2([0 1]) = [0 0]ØIn order to satisfy this, [c1 c2] = [0 0] è c1=c2=0ØThe only linear combination of vectors in V that yields

0 is the trivial linear combination. Therefore, V is a linearly independent set of vectors.

EXAMPLE 2

ØV={[1,2], [2,4]} Try to find a linear combination of vectors in V that yields 0.

Øc1([1 2]) + c2([2 4]) = [0 0][c1 2c1] + [2c2 4c2] = [0 0]c1 + 2c2 = 0 è c1 = -2c22c1 + 4c2 = 0 è 2c1 = -4c2

ØSo, c1 = 2 c2 = -1 is one of the possible solutions.ØThere exists a nontrivial linear combination of vectors

in V that yields 0. Therefore, V is a linearly dependent set of vectors.

THE RANK OF A MATRIX

ØLet A be any mxn matrix, and denote the rows of Aby r1, r2, …, rm. Also define R={ r1, r2, …, rm}.

ØThe rank of A is the number of vectors in the largest linearly independent subset of R.

ØIf for a matrix A with m rows, rank A=m; then the matrix is a collection of linearly independent set of vectors. If rank A<m; then the matrix contains a linearly dependent set of vectors.

EXAMPLES

úúú

û

ù

êêê

ë

é

3202110001

úúú

û

ù

êêê

ë

é=

320120001

Aúúú

û

ù

êêê

ë

é

1002110001

úúú

û

ù

êêê

ë

é

100010001

rank(A) = 3

1 0 00 1 01 1 0

Bé ùê ú= ê úê úë û

rank(B) = 2

THE INVERSE OF A MATRIX

ØA single linear equation in a single variable can be solved by multiplying both sides of the equation by multiplicative inverse of the variable coefficient.

ØExample:4x=3 è 4-1(4x) = (4-1)3 è x=3/4

ØWe can generalize this approach to square systems of linear equations (i.e., number of equations = number of unknowns).

SQUARE AND IDENTITY MATRIX

Ø A square matrix is any matrix that has an equal number of rows and columns.

Ø The diagonal elements of a square matrix are those elements aij such that i=j.

Ø A square matrix for which all diagonal elements are equal to 1 and all non-diagonal elements are equal to 0 is called an identity matrix.

úúú

û

ù

êêê

ë

é=ú

û

ùêë

é=

100010001

, 1001

32 II

INVERSE OF A

ØFor a given mxm matrix A, the mxm matrix B is the inverse of A if

BA = AB = Im

úúú

û

ù

êêê

ë

é=

úúú

û

ù

êêê

ë

é--

úúú

û

ù

êêê

ë

é

-

-

100010001

201715101

101213102

AA-1=I

FINDING THE INVERSE OF A MATRIX WITH THE GAUSS-JORDAN METHOD

Step 1 Write down the mx2m matrix [AΙIm].

Step 2 Use ero’s to transform [AΙIm] into [ImΙB]. This will only be possible if rank(A)=m; in this case, B=A-1. If rank(A)<m, then A has no inverse.

EXAMPLE

úû

ùêë

é=

3152

A

[ ]úúû

ù

êêë

é=

1001

3152

2IA [ ]úúû

ù

êêë

é=

1002

1

31251

12IA

[ ]úú

û

ù

êê

ë

é

-=12

102

1

210251

22IA[ ]úúû

ù

êêë

é

-=

2102

1

10251

32IA

[ ]úúû

ù

êêë

é

--

=2153

1001

42IA úû

ùêë

é-

-=-

21531A

USING MATRIX INVERSE TO SOLVE LINEAR SYSTEMS OF EQUATIONS

ØGiven a linear system of equations, Ax=bØMultiply both sides by A-1

A-1Ax = A-1b è x = A-1b

EXAMPLE

2x1 + 5x2 = 7x1 + 3x2 = 4 ú

û

ùêë

é=ú

û

ùêë

éúû

ùêë

é47

3152

2

1xx

úû

ùêë

é-

-=Þú

û

ùêë

é=

2153

3152 1-AA

úû

ùêë

é=ú

û

ùêë

éÞú

û

ùêë

éúû

ùêë

é-

-=ú

û

ùêë

é11

47

2153

2

1

2

1xx

xx