linear algebra review - cornell university · linear algebra review orie 4741 september 1, 2017...
Post on 21-Aug-2018
225 Views
Preview:
TRANSCRIPT
Linear Algebra ReviewORIE 4741
September 1, 2017
Linear Algebra Review September 1, 2017 1 / 33
Outline
1 Linear Independence and Dependence
2 Matrix Rank
3 Invertible Matrices
4 Norms
5 Projection Matrix
6 Helpful Matrix Hints
7 Gradients
8 Computational Complexity Notation
Linear Algebra Review September 1, 2017 2 / 33
Linear Independence and Dependence
Linear Algebra Review September 1, 2017 3 / 33
Linear IndependenceAlgebraic Definition
Definition
The sequence of vectors v1, v2, ..., vn is linearly independent if the onlycombination that gives the zero vector is 0v1 + 0v2 + ... + 0vn.
Example
The vectors [1, 0, 0], [0, 1, 0], and [0, 0, 1] are linearly independent
Linear Algebra Review September 1, 2017 4 / 33
Linear DependenceAlgebraic Definition
Definition
The sequence of vectors v1, v2, ..., vn is linearly dependent if there existsa combination that gives the zero vector other than 0v1 + 0v2 + ... + 0vn.
Example
The vectors [1, 2, 0], [2, 4, 0], and [0, 0, 1] are linearly dependent because−2[1, 2, 0] + 1[2, 4, 0] + 0[0, 0, 1] = [0, 0, 0].
Linear Algebra Review September 1, 2017 5 / 33
Linear DependenceGeometric Definition
Can you find a linear combination of these vectors that equals 0?
[scale=.05]
Linear Algebra Review September 1, 2017 6 / 33
Linear DependenceGeometric Definition
Any multiple of −1 ∗ Green + 2 ∗ Blue +−2 ∗ Red = 0
scale=.05Linear Algebra Review September 1, 2017 7 / 33
Matrix Rank
Linear Algebra Review September 1, 2017 8 / 33
Row Rank
Definition
The row rank of a matrix is the number of linearly independent rows in amatrix.
Example
Each row in matrix A is linearly independent, so row rank(A) = 3. Tworows in matrix B are linearly independent, so row rank(B) = 2.
A =
1 0 00 1 00 0 1
B =
1 0 01 1 0−2 0 0
Linear Algebra Review September 1, 2017 9 / 33
Column Rank
Definition
The column rank of a matrix is the number of linearly independentcolumns in a matrix.
Example
Each column in matrix A is linearly independent, so row rank(A) = 3. Thefirst two columns in matrix B are linearly independent, so row rank(B) = 2.
A =
1 0 00 1 00 0 1
B =
1 0 01 1 0−2 0 0
Linear Algebra Review September 1, 2017 10 / 33
Rank
Did you notice the row ranks and the column ranks for the matrices werethe same?
Definition
The rank of a matrix is the number of linearly independent rows(columns) in a matrix.
Linear Algebra Review September 1, 2017 11 / 33
Full-Rank
Definition
A matrix B is full-rank if:
rank(B) = min{# columns in B, # of rows in B}
Example
Both A and B are full-rank.
A =
1 0 0 00 1 0 00 0 1 0
B =
1 0 00 1 00 0 10 0 0
Linear Algebra Review September 1, 2017 12 / 33
Invertible Matrices
Linear Algebra Review September 1, 2017 13 / 33
Matrix Multiplication
Multiply A*B where:
A=
[1 23 4
]B=
[5 67 8
]
AB =
[(1 ∗ 5) + (2 ∗ 7) (1 ∗ 6) + 2 ∗ 8(3 ∗ 5) + (4 ∗ 7) (3 ∗ 6) + 4 ∗ 8
]=
[19 2243 50
]
Linear Algebra Review September 1, 2017 14 / 33
Matrix Multiplication
Multiply A*B where:
A=
[1 23 4
]B=
[5 67 8
]
AB =
[(1 ∗ 5) + (2 ∗ 7) (1 ∗ 6) + 2 ∗ 8(3 ∗ 5) + (4 ∗ 7) (3 ∗ 6) + 4 ∗ 8
]=
[19 2243 50
]
Linear Algebra Review September 1, 2017 14 / 33
Identity Matrix
Definition
The identity matrix, I, is an nxn matrix with 1s on the diagonal and 0severywhere else with the property that Ix = xI = x for any vector x(IA = AI = A for any matrix A).
Example
An example of the 3x3 identity matrix is:
I3 =
1 0 00 1 00 0 1
Linear Algebra Review September 1, 2017 15 / 33
Definition of Invertible Matrix
Definition
The matrix A is invertible if there exists a matrix A−1 such that
A−1A = I
AA−1 = I
Note: An invertible matrix should be square (same number of rows andcolumns) and have full-rank.
Linear Algebra Review September 1, 2017 16 / 33
Invertible Matrix Example
Let A =
[4 31 1
].
Then, the inverse of A is: A−1 =
[1 −31 4
].
Check that A−1 satisfies the definition of an inverse:
AA−1 =
[4 31 1
] [1 −31 4
]=
[1 00 1
]
A−1A =
[1 −31 4
] [4 31 1
]=
[1 00 1
]Therefore, A−1 is the inverse of A.
Linear Algebra Review September 1, 2017 17 / 33
Norms
Linear Algebra Review September 1, 2017 18 / 33
Vector Norm
Definition
The norm of a vector x, denoted ||x || or ||x ||2, is the length of x:
||x || =√x21 + x22 + ... + x2n
Example
Let x =
1234
, then
||x || =√
(12 + 22 + 32 + 42) =√
1 + 4 + 9 + 16 =√
30
Linear Algebra Review September 1, 2017 19 / 33
Projection Matrix
Linear Algebra Review September 1, 2017 20 / 33
Span
Definition
Let A be a matrix. The Aspan of the columns of A is the set of all linearcombinations of the columns of A.
Example
The rows of A =
[1 00 1
]spans all of R2.
Linear Algebra Review September 1, 2017 21 / 33
Projection Matrix
Definition
A square matrix P is the projection matrix onto the span(columns of A) if
P2 = P.
Example
P =
[12 −1
2
−12
12
]is a projection matrix because P2 = P.
Linear Algebra Review September 1, 2017 22 / 33
Projection Matrix Example Continued
Example
P =
[12 −1
2
−12
12
]is a projection matrix that projects onto
[1−1
].
P
[1−1
]=
[1−1
]= 1
[1−1
]
P
[2
27
]=
[−44
]= −4
[1−1
]
P
[254−1000
]=
[627−627
]= 627
[1−1
]
Linear Algebra Review September 1, 2017 23 / 33
Helpful Matrix Hints
Linear Algebra Review September 1, 2017 24 / 33
Helpful Matrix Properties and Identities
AB 6= BA
ATBT = (BA)T
A(B + C ) = AB + AC
(AB)C = A(BC )
IA = AI = A
AA−1 = A−1A = I
Linear Algebra Review September 1, 2017 25 / 33
Gradients
Linear Algebra Review September 1, 2017 26 / 33
Multivariate Derivatives
Find ∂f∂x , ∂f
∂y∂f∂z , where
f (x , y , z) = 5x + 2x2 + xy + 3xy2z − 4y3 + 2yz + 4z2 − x4yz2.
∂f∂x = 5 + 4x + y + 3y2z − 4x3yz2
∂f∂y = x + 6xyz − 12y2 + 2z − x4z2
∂f∂z = 3xy2 + 2y + 8z − 2x4yz4
Linear Algebra Review September 1, 2017 27 / 33
Multivariate Derivatives
Find ∂f∂x , ∂f
∂y∂f∂z , where
f (x , y , z) = 5x + 2x2 + xy + 3xy2z − 4y3 + 2yz + 4z2 − x4yz2.
∂f∂x = 5 + 4x + y + 3y2z − 4x3yz2
∂f∂y = x + 6xyz − 12y2 + 2z − x4z2
∂f∂z = 3xy2 + 2y + 8z − 2x4yz4
Linear Algebra Review September 1, 2017 27 / 33
Multivariate Derivatives
Find ∂f∂x , ∂f
∂y∂f∂z , where
f (x , y , z) = 5x + 2x2 + xy + 3xy2z − 4y3 + 2yz + 4z2 − x4yz2.
∂f∂x = 5 + 4x + y + 3y2z − 4x3yz2
∂f∂y = x + 6xyz − 12y2 + 2z − x4z2
∂f∂z = 3xy2 + 2y + 8z − 2x4yz4
Linear Algebra Review September 1, 2017 27 / 33
Multivariate Derivatives
Find ∂f∂x , ∂f
∂y∂f∂z , where
f (x , y , z) = 5x + 2x2 + xy + 3xy2z − 4y3 + 2yz + 4z2 − x4yz2.
∂f∂x = 5 + 4x + y + 3y2z − 4x3yz2
∂f∂y = x + 6xyz − 12y2 + 2z − x4z2
∂f∂z = 3xy2 + 2y + 8z − 2x4yz4
Linear Algebra Review September 1, 2017 27 / 33
Gradient
Definition
Let f (x1, x2, ..., xn) be a multivariable function. The gradient of f, ∇f , isthe multivariable generalization of the derivative of f. The gradient is avector, where each row corresponds to a partial derivative with respect toa variable of the function.
∇f =
∂f∂x1
∂f∂x2...∂f∂xn
Linear Algebra Review September 1, 2017 28 / 33
Example Gradient
Let f (x , y , z) = x2 + 3xy + 4xyz + z .
Then ∇f is:
∇f =
∂f∂x
∂f∂y
∂f∂z
=
2x + 3y + 4yz3x + 4xz4xy + 1
Linear Algebra Review September 1, 2017 29 / 33
Computational Complexity Notation
Linear Algebra Review September 1, 2017 30 / 33
Big O Notation
Big O notation is used to describe the run-time (computationalcomplexity) of algorithms.
Definition
Let f(n) be the run-time of some algorithm. If f(n) = O(g(n)), then thereexists a constant C and a constant N such that:|f (x)| ≤ C ∗ |g(x)| for all x > N.
Example
When an algorithm runs in time O(n). That means it runs in linear time.So, where n is the amount of data, if an algorithm runs in time 5n,1000n+50000, n, or n-45, the algorithm runs in time O(n).
Linear Algebra Review September 1, 2017 31 / 33
Big O Notation
http://bigocheatsheet.com
Linear Algebra Review September 1, 2017 32 / 33
References
Strang, Gilbert. (2009, 4th ed.). Introduction to Linear Algebra.
Ling-Hsiao Ly’s 2012 Lecture 3 Projection and Projection Matrices Noteshttp://www.ss.ncu.edu.tw/ lyu/lecturef ilesen/lyuLANotes/LyuLA2012/LyuLA32012.pdf
Linear Algebra Review September 1, 2017 33 / 33
top related