linear algebra review - cornell university · linear algebra review orie 4741 september 1, 2017...

37
Linear Algebra Review ORIE 4741 September 1, 2017 Linear Algebra Review September 1, 2017 1 / 33

Upload: vohanh

Post on 21-Aug-2018

225 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Linear Algebra Review - Cornell University · Linear Algebra Review ORIE 4741 September 1, 2017 Linear Algebra Review September 1, 2017 1 / 33. Outline 1 Linear Independence ... That

Linear Algebra ReviewORIE 4741

September 1, 2017

Linear Algebra Review September 1, 2017 1 / 33

Page 2: Linear Algebra Review - Cornell University · Linear Algebra Review ORIE 4741 September 1, 2017 Linear Algebra Review September 1, 2017 1 / 33. Outline 1 Linear Independence ... That

Outline

1 Linear Independence and Dependence

2 Matrix Rank

3 Invertible Matrices

4 Norms

5 Projection Matrix

6 Helpful Matrix Hints

7 Gradients

8 Computational Complexity Notation

Linear Algebra Review September 1, 2017 2 / 33

Page 3: Linear Algebra Review - Cornell University · Linear Algebra Review ORIE 4741 September 1, 2017 Linear Algebra Review September 1, 2017 1 / 33. Outline 1 Linear Independence ... That

Linear Independence and Dependence

Linear Algebra Review September 1, 2017 3 / 33

Page 4: Linear Algebra Review - Cornell University · Linear Algebra Review ORIE 4741 September 1, 2017 Linear Algebra Review September 1, 2017 1 / 33. Outline 1 Linear Independence ... That

Linear IndependenceAlgebraic Definition

Definition

The sequence of vectors v1, v2, ..., vn is linearly independent if the onlycombination that gives the zero vector is 0v1 + 0v2 + ... + 0vn.

Example

The vectors [1, 0, 0], [0, 1, 0], and [0, 0, 1] are linearly independent

Linear Algebra Review September 1, 2017 4 / 33

Page 5: Linear Algebra Review - Cornell University · Linear Algebra Review ORIE 4741 September 1, 2017 Linear Algebra Review September 1, 2017 1 / 33. Outline 1 Linear Independence ... That

Linear DependenceAlgebraic Definition

Definition

The sequence of vectors v1, v2, ..., vn is linearly dependent if there existsa combination that gives the zero vector other than 0v1 + 0v2 + ... + 0vn.

Example

The vectors [1, 2, 0], [2, 4, 0], and [0, 0, 1] are linearly dependent because−2[1, 2, 0] + 1[2, 4, 0] + 0[0, 0, 1] = [0, 0, 0].

Linear Algebra Review September 1, 2017 5 / 33

Page 6: Linear Algebra Review - Cornell University · Linear Algebra Review ORIE 4741 September 1, 2017 Linear Algebra Review September 1, 2017 1 / 33. Outline 1 Linear Independence ... That

Linear DependenceGeometric Definition

Can you find a linear combination of these vectors that equals 0?

[scale=.05]

Linear Algebra Review September 1, 2017 6 / 33

Page 7: Linear Algebra Review - Cornell University · Linear Algebra Review ORIE 4741 September 1, 2017 Linear Algebra Review September 1, 2017 1 / 33. Outline 1 Linear Independence ... That

Linear DependenceGeometric Definition

Any multiple of −1 ∗ Green + 2 ∗ Blue +−2 ∗ Red = 0

scale=.05Linear Algebra Review September 1, 2017 7 / 33

Page 8: Linear Algebra Review - Cornell University · Linear Algebra Review ORIE 4741 September 1, 2017 Linear Algebra Review September 1, 2017 1 / 33. Outline 1 Linear Independence ... That

Matrix Rank

Linear Algebra Review September 1, 2017 8 / 33

Page 9: Linear Algebra Review - Cornell University · Linear Algebra Review ORIE 4741 September 1, 2017 Linear Algebra Review September 1, 2017 1 / 33. Outline 1 Linear Independence ... That

Row Rank

Definition

The row rank of a matrix is the number of linearly independent rows in amatrix.

Example

Each row in matrix A is linearly independent, so row rank(A) = 3. Tworows in matrix B are linearly independent, so row rank(B) = 2.

A =

1 0 00 1 00 0 1

B =

1 0 01 1 0−2 0 0

Linear Algebra Review September 1, 2017 9 / 33

Page 10: Linear Algebra Review - Cornell University · Linear Algebra Review ORIE 4741 September 1, 2017 Linear Algebra Review September 1, 2017 1 / 33. Outline 1 Linear Independence ... That

Column Rank

Definition

The column rank of a matrix is the number of linearly independentcolumns in a matrix.

Example

Each column in matrix A is linearly independent, so row rank(A) = 3. Thefirst two columns in matrix B are linearly independent, so row rank(B) = 2.

A =

1 0 00 1 00 0 1

B =

1 0 01 1 0−2 0 0

Linear Algebra Review September 1, 2017 10 / 33

Page 11: Linear Algebra Review - Cornell University · Linear Algebra Review ORIE 4741 September 1, 2017 Linear Algebra Review September 1, 2017 1 / 33. Outline 1 Linear Independence ... That

Rank

Did you notice the row ranks and the column ranks for the matrices werethe same?

Definition

The rank of a matrix is the number of linearly independent rows(columns) in a matrix.

Linear Algebra Review September 1, 2017 11 / 33

Page 12: Linear Algebra Review - Cornell University · Linear Algebra Review ORIE 4741 September 1, 2017 Linear Algebra Review September 1, 2017 1 / 33. Outline 1 Linear Independence ... That

Full-Rank

Definition

A matrix B is full-rank if:

rank(B) = min{# columns in B, # of rows in B}

Example

Both A and B are full-rank.

A =

1 0 0 00 1 0 00 0 1 0

B =

1 0 00 1 00 0 10 0 0

Linear Algebra Review September 1, 2017 12 / 33

Page 13: Linear Algebra Review - Cornell University · Linear Algebra Review ORIE 4741 September 1, 2017 Linear Algebra Review September 1, 2017 1 / 33. Outline 1 Linear Independence ... That

Invertible Matrices

Linear Algebra Review September 1, 2017 13 / 33

Page 14: Linear Algebra Review - Cornell University · Linear Algebra Review ORIE 4741 September 1, 2017 Linear Algebra Review September 1, 2017 1 / 33. Outline 1 Linear Independence ... That

Matrix Multiplication

Multiply A*B where:

A=

[1 23 4

]B=

[5 67 8

]

AB =

[(1 ∗ 5) + (2 ∗ 7) (1 ∗ 6) + 2 ∗ 8(3 ∗ 5) + (4 ∗ 7) (3 ∗ 6) + 4 ∗ 8

]=

[19 2243 50

]

Linear Algebra Review September 1, 2017 14 / 33

Page 15: Linear Algebra Review - Cornell University · Linear Algebra Review ORIE 4741 September 1, 2017 Linear Algebra Review September 1, 2017 1 / 33. Outline 1 Linear Independence ... That

Matrix Multiplication

Multiply A*B where:

A=

[1 23 4

]B=

[5 67 8

]

AB =

[(1 ∗ 5) + (2 ∗ 7) (1 ∗ 6) + 2 ∗ 8(3 ∗ 5) + (4 ∗ 7) (3 ∗ 6) + 4 ∗ 8

]=

[19 2243 50

]

Linear Algebra Review September 1, 2017 14 / 33

Page 16: Linear Algebra Review - Cornell University · Linear Algebra Review ORIE 4741 September 1, 2017 Linear Algebra Review September 1, 2017 1 / 33. Outline 1 Linear Independence ... That

Identity Matrix

Definition

The identity matrix, I, is an nxn matrix with 1s on the diagonal and 0severywhere else with the property that Ix = xI = x for any vector x(IA = AI = A for any matrix A).

Example

An example of the 3x3 identity matrix is:

I3 =

1 0 00 1 00 0 1

Linear Algebra Review September 1, 2017 15 / 33

Page 17: Linear Algebra Review - Cornell University · Linear Algebra Review ORIE 4741 September 1, 2017 Linear Algebra Review September 1, 2017 1 / 33. Outline 1 Linear Independence ... That

Definition of Invertible Matrix

Definition

The matrix A is invertible if there exists a matrix A−1 such that

A−1A = I

AA−1 = I

Note: An invertible matrix should be square (same number of rows andcolumns) and have full-rank.

Linear Algebra Review September 1, 2017 16 / 33

Page 18: Linear Algebra Review - Cornell University · Linear Algebra Review ORIE 4741 September 1, 2017 Linear Algebra Review September 1, 2017 1 / 33. Outline 1 Linear Independence ... That

Invertible Matrix Example

Let A =

[4 31 1

].

Then, the inverse of A is: A−1 =

[1 −31 4

].

Check that A−1 satisfies the definition of an inverse:

AA−1 =

[4 31 1

] [1 −31 4

]=

[1 00 1

]

A−1A =

[1 −31 4

] [4 31 1

]=

[1 00 1

]Therefore, A−1 is the inverse of A.

Linear Algebra Review September 1, 2017 17 / 33

Page 19: Linear Algebra Review - Cornell University · Linear Algebra Review ORIE 4741 September 1, 2017 Linear Algebra Review September 1, 2017 1 / 33. Outline 1 Linear Independence ... That

Norms

Linear Algebra Review September 1, 2017 18 / 33

Page 20: Linear Algebra Review - Cornell University · Linear Algebra Review ORIE 4741 September 1, 2017 Linear Algebra Review September 1, 2017 1 / 33. Outline 1 Linear Independence ... That

Vector Norm

Definition

The norm of a vector x, denoted ||x || or ||x ||2, is the length of x:

||x || =√x21 + x22 + ... + x2n

Example

Let x =

1234

, then

||x || =√

(12 + 22 + 32 + 42) =√

1 + 4 + 9 + 16 =√

30

Linear Algebra Review September 1, 2017 19 / 33

Page 21: Linear Algebra Review - Cornell University · Linear Algebra Review ORIE 4741 September 1, 2017 Linear Algebra Review September 1, 2017 1 / 33. Outline 1 Linear Independence ... That

Projection Matrix

Linear Algebra Review September 1, 2017 20 / 33

Page 22: Linear Algebra Review - Cornell University · Linear Algebra Review ORIE 4741 September 1, 2017 Linear Algebra Review September 1, 2017 1 / 33. Outline 1 Linear Independence ... That

Span

Definition

Let A be a matrix. The Aspan of the columns of A is the set of all linearcombinations of the columns of A.

Example

The rows of A =

[1 00 1

]spans all of R2.

Linear Algebra Review September 1, 2017 21 / 33

Page 23: Linear Algebra Review - Cornell University · Linear Algebra Review ORIE 4741 September 1, 2017 Linear Algebra Review September 1, 2017 1 / 33. Outline 1 Linear Independence ... That

Projection Matrix

Definition

A square matrix P is the projection matrix onto the span(columns of A) if

P2 = P.

Example

P =

[12 −1

2

−12

12

]is a projection matrix because P2 = P.

Linear Algebra Review September 1, 2017 22 / 33

Page 24: Linear Algebra Review - Cornell University · Linear Algebra Review ORIE 4741 September 1, 2017 Linear Algebra Review September 1, 2017 1 / 33. Outline 1 Linear Independence ... That

Projection Matrix Example Continued

Example

P =

[12 −1

2

−12

12

]is a projection matrix that projects onto

[1−1

].

P

[1−1

]=

[1−1

]= 1

[1−1

]

P

[2

27

]=

[−44

]= −4

[1−1

]

P

[254−1000

]=

[627−627

]= 627

[1−1

]

Linear Algebra Review September 1, 2017 23 / 33

Page 25: Linear Algebra Review - Cornell University · Linear Algebra Review ORIE 4741 September 1, 2017 Linear Algebra Review September 1, 2017 1 / 33. Outline 1 Linear Independence ... That

Helpful Matrix Hints

Linear Algebra Review September 1, 2017 24 / 33

Page 26: Linear Algebra Review - Cornell University · Linear Algebra Review ORIE 4741 September 1, 2017 Linear Algebra Review September 1, 2017 1 / 33. Outline 1 Linear Independence ... That

Helpful Matrix Properties and Identities

AB 6= BA

ATBT = (BA)T

A(B + C ) = AB + AC

(AB)C = A(BC )

IA = AI = A

AA−1 = A−1A = I

Linear Algebra Review September 1, 2017 25 / 33

Page 27: Linear Algebra Review - Cornell University · Linear Algebra Review ORIE 4741 September 1, 2017 Linear Algebra Review September 1, 2017 1 / 33. Outline 1 Linear Independence ... That

Gradients

Linear Algebra Review September 1, 2017 26 / 33

Page 28: Linear Algebra Review - Cornell University · Linear Algebra Review ORIE 4741 September 1, 2017 Linear Algebra Review September 1, 2017 1 / 33. Outline 1 Linear Independence ... That

Multivariate Derivatives

Find ∂f∂x , ∂f

∂y∂f∂z , where

f (x , y , z) = 5x + 2x2 + xy + 3xy2z − 4y3 + 2yz + 4z2 − x4yz2.

∂f∂x = 5 + 4x + y + 3y2z − 4x3yz2

∂f∂y = x + 6xyz − 12y2 + 2z − x4z2

∂f∂z = 3xy2 + 2y + 8z − 2x4yz4

Linear Algebra Review September 1, 2017 27 / 33

Page 29: Linear Algebra Review - Cornell University · Linear Algebra Review ORIE 4741 September 1, 2017 Linear Algebra Review September 1, 2017 1 / 33. Outline 1 Linear Independence ... That

Multivariate Derivatives

Find ∂f∂x , ∂f

∂y∂f∂z , where

f (x , y , z) = 5x + 2x2 + xy + 3xy2z − 4y3 + 2yz + 4z2 − x4yz2.

∂f∂x = 5 + 4x + y + 3y2z − 4x3yz2

∂f∂y = x + 6xyz − 12y2 + 2z − x4z2

∂f∂z = 3xy2 + 2y + 8z − 2x4yz4

Linear Algebra Review September 1, 2017 27 / 33

Page 30: Linear Algebra Review - Cornell University · Linear Algebra Review ORIE 4741 September 1, 2017 Linear Algebra Review September 1, 2017 1 / 33. Outline 1 Linear Independence ... That

Multivariate Derivatives

Find ∂f∂x , ∂f

∂y∂f∂z , where

f (x , y , z) = 5x + 2x2 + xy + 3xy2z − 4y3 + 2yz + 4z2 − x4yz2.

∂f∂x = 5 + 4x + y + 3y2z − 4x3yz2

∂f∂y = x + 6xyz − 12y2 + 2z − x4z2

∂f∂z = 3xy2 + 2y + 8z − 2x4yz4

Linear Algebra Review September 1, 2017 27 / 33

Page 31: Linear Algebra Review - Cornell University · Linear Algebra Review ORIE 4741 September 1, 2017 Linear Algebra Review September 1, 2017 1 / 33. Outline 1 Linear Independence ... That

Multivariate Derivatives

Find ∂f∂x , ∂f

∂y∂f∂z , where

f (x , y , z) = 5x + 2x2 + xy + 3xy2z − 4y3 + 2yz + 4z2 − x4yz2.

∂f∂x = 5 + 4x + y + 3y2z − 4x3yz2

∂f∂y = x + 6xyz − 12y2 + 2z − x4z2

∂f∂z = 3xy2 + 2y + 8z − 2x4yz4

Linear Algebra Review September 1, 2017 27 / 33

Page 32: Linear Algebra Review - Cornell University · Linear Algebra Review ORIE 4741 September 1, 2017 Linear Algebra Review September 1, 2017 1 / 33. Outline 1 Linear Independence ... That

Gradient

Definition

Let f (x1, x2, ..., xn) be a multivariable function. The gradient of f, ∇f , isthe multivariable generalization of the derivative of f. The gradient is avector, where each row corresponds to a partial derivative with respect toa variable of the function.

∇f =

∂f∂x1

∂f∂x2...∂f∂xn

Linear Algebra Review September 1, 2017 28 / 33

Page 33: Linear Algebra Review - Cornell University · Linear Algebra Review ORIE 4741 September 1, 2017 Linear Algebra Review September 1, 2017 1 / 33. Outline 1 Linear Independence ... That

Example Gradient

Let f (x , y , z) = x2 + 3xy + 4xyz + z .

Then ∇f is:

∇f =

∂f∂x

∂f∂y

∂f∂z

=

2x + 3y + 4yz3x + 4xz4xy + 1

Linear Algebra Review September 1, 2017 29 / 33

Page 34: Linear Algebra Review - Cornell University · Linear Algebra Review ORIE 4741 September 1, 2017 Linear Algebra Review September 1, 2017 1 / 33. Outline 1 Linear Independence ... That

Computational Complexity Notation

Linear Algebra Review September 1, 2017 30 / 33

Page 35: Linear Algebra Review - Cornell University · Linear Algebra Review ORIE 4741 September 1, 2017 Linear Algebra Review September 1, 2017 1 / 33. Outline 1 Linear Independence ... That

Big O Notation

Big O notation is used to describe the run-time (computationalcomplexity) of algorithms.

Definition

Let f(n) be the run-time of some algorithm. If f(n) = O(g(n)), then thereexists a constant C and a constant N such that:|f (x)| ≤ C ∗ |g(x)| for all x > N.

Example

When an algorithm runs in time O(n). That means it runs in linear time.So, where n is the amount of data, if an algorithm runs in time 5n,1000n+50000, n, or n-45, the algorithm runs in time O(n).

Linear Algebra Review September 1, 2017 31 / 33

Page 36: Linear Algebra Review - Cornell University · Linear Algebra Review ORIE 4741 September 1, 2017 Linear Algebra Review September 1, 2017 1 / 33. Outline 1 Linear Independence ... That

Big O Notation

http://bigocheatsheet.com

Linear Algebra Review September 1, 2017 32 / 33

Page 37: Linear Algebra Review - Cornell University · Linear Algebra Review ORIE 4741 September 1, 2017 Linear Algebra Review September 1, 2017 1 / 33. Outline 1 Linear Independence ... That

References

Strang, Gilbert. (2009, 4th ed.). Introduction to Linear Algebra.

Ling-Hsiao Ly’s 2012 Lecture 3 Projection and Projection Matrices Noteshttp://www.ss.ncu.edu.tw/ lyu/lecturef ilesen/lyuLANotes/LyuLA2012/LyuLA32012.pdf

Linear Algebra Review September 1, 2017 33 / 33