cs221 linear algebra
TRANSCRIPT
Linear Algebra Review
CS221: Introduction to Arti�cial Intelligence
Carlos Fernandez-Granda
10/11/2011
CS221 Linear Algebra Review 10/11/2011 1 / 37
Index
1 Vectors
2 Matrices
3 Matrix Decompositions
4 Application : Image Compression
CS221 Linear Algebra Review 10/11/2011 2 / 37
Vectors
1 Vectors
2 Matrices
3 Matrix Decompositions
4 Application : Image Compression
CS221 Linear Algebra Review 10/11/2011 3 / 37
Vectors
What is a vector ?
An ordered tuple of numbers :
u =
u1u2. . .un
A quantity that has a magnitude and a direction.
CS221 Linear Algebra Review 10/11/2011 4 / 37
Vectors
What is a vector ?
An ordered tuple of numbers :
u =
u1u2. . .un
A quantity that has a magnitude and a direction.
CS221 Linear Algebra Review 10/11/2011 4 / 37
Vectors
What is a vector ?
An ordered tuple of numbers :
u =
u1u2. . .un
A quantity that has a magnitude and a direction.
CS221 Linear Algebra Review 10/11/2011 4 / 37
Vectors
Vector spaces
More formally a vector is an element of a vector space.
A vector space V is a set that contains all linear combinations of itselements :
If vectors u and v ∈ V, then u + v ∈ V.If vector u ∈ V, then α u ∈ V for any scalar α.There exists 0 ∈ V such that u + 0 = u for any u ∈ V.
A subspace is a subset of a vector space that is also a vector space.
CS221 Linear Algebra Review 10/11/2011 5 / 37
Vectors
Vector spaces
More formally a vector is an element of a vector space.
A vector space V is a set that contains all linear combinations of itselements :
If vectors u and v ∈ V, then u + v ∈ V.If vector u ∈ V, then α u ∈ V for any scalar α.There exists 0 ∈ V such that u + 0 = u for any u ∈ V.
A subspace is a subset of a vector space that is also a vector space.
CS221 Linear Algebra Review 10/11/2011 5 / 37
Vectors
Vector spaces
More formally a vector is an element of a vector space.
A vector space V is a set that contains all linear combinations of itselements :
If vectors u and v ∈ V, then u + v ∈ V.
If vector u ∈ V, then α u ∈ V for any scalar α.There exists 0 ∈ V such that u + 0 = u for any u ∈ V.
A subspace is a subset of a vector space that is also a vector space.
CS221 Linear Algebra Review 10/11/2011 5 / 37
Vectors
Vector spaces
More formally a vector is an element of a vector space.
A vector space V is a set that contains all linear combinations of itselements :
If vectors u and v ∈ V, then u + v ∈ V.If vector u ∈ V, then α u ∈ V for any scalar α.
There exists 0 ∈ V such that u + 0 = u for any u ∈ V.A subspace is a subset of a vector space that is also a vector space.
CS221 Linear Algebra Review 10/11/2011 5 / 37
Vectors
Vector spaces
More formally a vector is an element of a vector space.
A vector space V is a set that contains all linear combinations of itselements :
If vectors u and v ∈ V, then u + v ∈ V.If vector u ∈ V, then α u ∈ V for any scalar α.There exists 0 ∈ V such that u + 0 = u for any u ∈ V.
A subspace is a subset of a vector space that is also a vector space.
CS221 Linear Algebra Review 10/11/2011 5 / 37
Vectors
Vector spaces
More formally a vector is an element of a vector space.
A vector space V is a set that contains all linear combinations of itselements :
If vectors u and v ∈ V, then u + v ∈ V.If vector u ∈ V, then α u ∈ V for any scalar α.There exists 0 ∈ V such that u + 0 = u for any u ∈ V.
A subspace is a subset of a vector space that is also a vector space.
CS221 Linear Algebra Review 10/11/2011 5 / 37
Vectors
Examples
Euclidean space Rn.
The span of any set of vectors {u1,u2, . . . ,un}, de�ned as :
span (u1,u2, . . . ,un) = {α1u1 + α2u2 + · · ·+ αnun | αi ∈ R}
CS221 Linear Algebra Review 10/11/2011 6 / 37
Vectors
Examples
Euclidean space Rn.
The span of any set of vectors {u1,u2, . . . ,un}, de�ned as :
span (u1,u2, . . . ,un) = {α1u1 + α2u2 + · · ·+ αnun | αi ∈ R}
CS221 Linear Algebra Review 10/11/2011 6 / 37
Vectors
Subspaces
A line through the origin in Rn(span of a vector).
A plane in Rn(span of two vectors).
CS221 Linear Algebra Review 10/11/2011 7 / 37
Vectors
Subspaces
A line through the origin in Rn(span of a vector).
A plane in Rn(span of two vectors).
CS221 Linear Algebra Review 10/11/2011 7 / 37
Vectors
Linear Independence and Basis of Vector Spaces
A vector u is linearly independent of a set of vectors {u1,u2, . . . ,un}if it does not lie in their span.
A set of vectors is linearly independent if every vector is linearlyindependent of the rest.
A basis of a vector space V is a linearly independent set of vectorswhose span is equal to V.If a vector space has a basis with d vectors, its dimension is d.
Any other basis of the same space will consist of exactly d vectors.
The dimension of a vector space can be in�nite (function spaces).
CS221 Linear Algebra Review 10/11/2011 8 / 37
Vectors
Linear Independence and Basis of Vector Spaces
A vector u is linearly independent of a set of vectors {u1,u2, . . . ,un}if it does not lie in their span.
A set of vectors is linearly independent if every vector is linearlyindependent of the rest.
A basis of a vector space V is a linearly independent set of vectorswhose span is equal to V.If a vector space has a basis with d vectors, its dimension is d.
Any other basis of the same space will consist of exactly d vectors.
The dimension of a vector space can be in�nite (function spaces).
CS221 Linear Algebra Review 10/11/2011 8 / 37
Vectors
Linear Independence and Basis of Vector Spaces
A vector u is linearly independent of a set of vectors {u1,u2, . . . ,un}if it does not lie in their span.
A set of vectors is linearly independent if every vector is linearlyindependent of the rest.
A basis of a vector space V is a linearly independent set of vectorswhose span is equal to V.
If a vector space has a basis with d vectors, its dimension is d.
Any other basis of the same space will consist of exactly d vectors.
The dimension of a vector space can be in�nite (function spaces).
CS221 Linear Algebra Review 10/11/2011 8 / 37
Vectors
Linear Independence and Basis of Vector Spaces
A vector u is linearly independent of a set of vectors {u1,u2, . . . ,un}if it does not lie in their span.
A set of vectors is linearly independent if every vector is linearlyindependent of the rest.
A basis of a vector space V is a linearly independent set of vectorswhose span is equal to V.If a vector space has a basis with d vectors, its dimension is d.
Any other basis of the same space will consist of exactly d vectors.
The dimension of a vector space can be in�nite (function spaces).
CS221 Linear Algebra Review 10/11/2011 8 / 37
Vectors
Linear Independence and Basis of Vector Spaces
A vector u is linearly independent of a set of vectors {u1,u2, . . . ,un}if it does not lie in their span.
A set of vectors is linearly independent if every vector is linearlyindependent of the rest.
A basis of a vector space V is a linearly independent set of vectorswhose span is equal to V.If a vector space has a basis with d vectors, its dimension is d.
Any other basis of the same space will consist of exactly d vectors.
The dimension of a vector space can be in�nite (function spaces).
CS221 Linear Algebra Review 10/11/2011 8 / 37
Vectors
Linear Independence and Basis of Vector Spaces
A vector u is linearly independent of a set of vectors {u1,u2, . . . ,un}if it does not lie in their span.
A set of vectors is linearly independent if every vector is linearlyindependent of the rest.
A basis of a vector space V is a linearly independent set of vectorswhose span is equal to V.If a vector space has a basis with d vectors, its dimension is d.
Any other basis of the same space will consist of exactly d vectors.
The dimension of a vector space can be in�nite (function spaces).
CS221 Linear Algebra Review 10/11/2011 8 / 37
Vectors
Norm of a Vector
A norm ||u|| measures the magnitude of a vector.
Mathematically, any function from the vector space to R thatsatis�es :
Homogeneity ||α u|| = |α| ||u||Triangle inequality ||u + v|| ≤ ||u||+ ||v||Point separation ||u|| = 0 if and only if u = 0.
Examples :
Manhattan or `1 norm : ||u||1
=∑
i|ui |.
Euclidean or `2 norm : ||u||2
=√∑
iu2i.
CS221 Linear Algebra Review 10/11/2011 9 / 37
Vectors
Norm of a Vector
A norm ||u|| measures the magnitude of a vector.
Mathematically, any function from the vector space to R thatsatis�es :
Homogeneity ||α u|| = |α| ||u||Triangle inequality ||u + v|| ≤ ||u||+ ||v||Point separation ||u|| = 0 if and only if u = 0.
Examples :
Manhattan or `1 norm : ||u||1
=∑
i|ui |.
Euclidean or `2 norm : ||u||2
=√∑
iu2i.
CS221 Linear Algebra Review 10/11/2011 9 / 37
Vectors
Norm of a Vector
A norm ||u|| measures the magnitude of a vector.
Mathematically, any function from the vector space to R thatsatis�es :
Homogeneity ||α u|| = |α| ||u||
Triangle inequality ||u + v|| ≤ ||u||+ ||v||Point separation ||u|| = 0 if and only if u = 0.
Examples :
Manhattan or `1 norm : ||u||1
=∑
i|ui |.
Euclidean or `2 norm : ||u||2
=√∑
iu2i.
CS221 Linear Algebra Review 10/11/2011 9 / 37
Vectors
Norm of a Vector
A norm ||u|| measures the magnitude of a vector.
Mathematically, any function from the vector space to R thatsatis�es :
Homogeneity ||α u|| = |α| ||u||Triangle inequality ||u + v|| ≤ ||u||+ ||v||
Point separation ||u|| = 0 if and only if u = 0.
Examples :
Manhattan or `1 norm : ||u||1
=∑
i|ui |.
Euclidean or `2 norm : ||u||2
=√∑
iu2i.
CS221 Linear Algebra Review 10/11/2011 9 / 37
Vectors
Norm of a Vector
A norm ||u|| measures the magnitude of a vector.
Mathematically, any function from the vector space to R thatsatis�es :
Homogeneity ||α u|| = |α| ||u||Triangle inequality ||u + v|| ≤ ||u||+ ||v||Point separation ||u|| = 0 if and only if u = 0.
Examples :
Manhattan or `1 norm : ||u||1
=∑
i|ui |.
Euclidean or `2 norm : ||u||2
=√∑
iu2i.
CS221 Linear Algebra Review 10/11/2011 9 / 37
Vectors
Norm of a Vector
A norm ||u|| measures the magnitude of a vector.
Mathematically, any function from the vector space to R thatsatis�es :
Homogeneity ||α u|| = |α| ||u||Triangle inequality ||u + v|| ≤ ||u||+ ||v||Point separation ||u|| = 0 if and only if u = 0.
Examples :
Manhattan or `1 norm : ||u||1
=∑
i|ui |.
Euclidean or `2 norm : ||u||2
=√∑
iu2i.
CS221 Linear Algebra Review 10/11/2011 9 / 37
Vectors
Inner Product
Inner product between u and v : 〈u, v〉 =∑n
i=1uivi .
It is the projection of one vector onto the other.
Related to the Euclidean norm : 〈u,u〉 = ||u||22.
CS221 Linear Algebra Review 10/11/2011 10 / 37
Vectors
Inner Product
Inner product between u and v : 〈u, v〉 =∑n
i=1uivi .
It is the projection of one vector onto the other.
Related to the Euclidean norm : 〈u,u〉 = ||u||22.
CS221 Linear Algebra Review 10/11/2011 10 / 37
Vectors
Inner Product
Inner product between u and v : 〈u, v〉 =∑n
i=1uivi .
It is the projection of one vector onto the other.
Related to the Euclidean norm : 〈u,u〉 = ||u||22.
CS221 Linear Algebra Review 10/11/2011 10 / 37
Vectors
Inner Product
The inner product is a measure of correlation between two vectors, scaledby the norms of the vectors :
If 〈u, v〉 > 0, u and v are aligned.
If 〈u, v〉 < 0, u and v are opposed.
If 〈u, v〉 = 0, u and v are orthogonal.
CS221 Linear Algebra Review 10/11/2011 11 / 37
Vectors
Inner Product
The inner product is a measure of correlation between two vectors, scaledby the norms of the vectors :
If 〈u, v〉 > 0, u and v are aligned.
If 〈u, v〉 < 0, u and v are opposed.
If 〈u, v〉 = 0, u and v are orthogonal.
CS221 Linear Algebra Review 10/11/2011 11 / 37
Vectors
Inner Product
The inner product is a measure of correlation between two vectors, scaledby the norms of the vectors :
If 〈u, v〉 > 0, u and v are aligned.
If 〈u, v〉 < 0, u and v are opposed.
If 〈u, v〉 = 0, u and v are orthogonal.
CS221 Linear Algebra Review 10/11/2011 11 / 37
Vectors
Inner Product
The inner product is a measure of correlation between two vectors, scaledby the norms of the vectors :
If 〈u, v〉 > 0, u and v are aligned.
If 〈u, v〉 < 0, u and v are opposed.
If 〈u, v〉 = 0, u and v are orthogonal.
CS221 Linear Algebra Review 10/11/2011 11 / 37
Vectors
Orthonormal Basis
The vectors in an orthonormal basis have unit Euclidean norm andare orthogonal to each other.
To express a vector x in an orthonormal basis, we can just take innerproducts.
Example : x = α1b1 + α2b2.
We compute 〈x,b1〉 :
〈x,b1〉 = 〈α1b1 + α2b2,b1〉= α1 〈b1,b1〉+ α2 〈b2,b1〉 By linearity.
= α1 + 0 By orthonormality.
Likewise, α2 = 〈x,b2〉.
CS221 Linear Algebra Review 10/11/2011 12 / 37
Vectors
Orthonormal Basis
The vectors in an orthonormal basis have unit Euclidean norm andare orthogonal to each other.
To express a vector x in an orthonormal basis, we can just take innerproducts.
Example : x = α1b1 + α2b2.
We compute 〈x,b1〉 :
〈x,b1〉 = 〈α1b1 + α2b2,b1〉= α1 〈b1,b1〉+ α2 〈b2,b1〉 By linearity.
= α1 + 0 By orthonormality.
Likewise, α2 = 〈x,b2〉.
CS221 Linear Algebra Review 10/11/2011 12 / 37
Vectors
Orthonormal Basis
The vectors in an orthonormal basis have unit Euclidean norm andare orthogonal to each other.
To express a vector x in an orthonormal basis, we can just take innerproducts.
Example : x = α1b1 + α2b2.
We compute 〈x,b1〉 :
〈x,b1〉 = 〈α1b1 + α2b2,b1〉= α1 〈b1,b1〉+ α2 〈b2,b1〉 By linearity.
= α1 + 0 By orthonormality.
Likewise, α2 = 〈x,b2〉.
CS221 Linear Algebra Review 10/11/2011 12 / 37
Vectors
Orthonormal Basis
The vectors in an orthonormal basis have unit Euclidean norm andare orthogonal to each other.
To express a vector x in an orthonormal basis, we can just take innerproducts.
Example : x = α1b1 + α2b2.
We compute 〈x,b1〉 :
〈x,b1〉 = 〈α1b1 + α2b2,b1〉= α1 〈b1,b1〉+ α2 〈b2,b1〉 By linearity.
= α1 + 0 By orthonormality.
Likewise, α2 = 〈x,b2〉.
CS221 Linear Algebra Review 10/11/2011 12 / 37
Vectors
Orthonormal Basis
The vectors in an orthonormal basis have unit Euclidean norm andare orthogonal to each other.
To express a vector x in an orthonormal basis, we can just take innerproducts.
Example : x = α1b1 + α2b2.
We compute 〈x,b1〉 :
〈x,b1〉 = 〈α1b1 + α2b2,b1〉= α1 〈b1,b1〉+ α2 〈b2,b1〉 By linearity.
= α1 + 0 By orthonormality.
Likewise, α2 = 〈x,b2〉.
CS221 Linear Algebra Review 10/11/2011 12 / 37
Matrices
1 Vectors
2 Matrices
3 Matrix Decompositions
4 Application : Image Compression
CS221 Linear Algebra Review 10/11/2011 13 / 37
Matrices
Linear Operator
A linear operator L : U → V is a map from a vector space U toanother vector space V that satis�es :
L (u1 + u2) = L (u1) + L (u2).L (α u) = α L (u) for any scalar α.
If the dimension n of U and m of V are �nite, L can be represented byan m × n matrix A.
A =
A11 A12 . . . A1n
A21 A22 . . . A2n
. . .Am1 Am2 . . . Amn
CS221 Linear Algebra Review 10/11/2011 14 / 37
Matrices
Linear Operator
A linear operator L : U → V is a map from a vector space U toanother vector space V that satis�es :
L (u1 + u2) = L (u1) + L (u2).
L (α u) = α L (u) for any scalar α.
If the dimension n of U and m of V are �nite, L can be represented byan m × n matrix A.
A =
A11 A12 . . . A1n
A21 A22 . . . A2n
. . .Am1 Am2 . . . Amn
CS221 Linear Algebra Review 10/11/2011 14 / 37
Matrices
Linear Operator
A linear operator L : U → V is a map from a vector space U toanother vector space V that satis�es :
L (u1 + u2) = L (u1) + L (u2).L (α u) = α L (u) for any scalar α.
If the dimension n of U and m of V are �nite, L can be represented byan m × n matrix A.
A =
A11 A12 . . . A1n
A21 A22 . . . A2n
. . .Am1 Am2 . . . Amn
CS221 Linear Algebra Review 10/11/2011 14 / 37
Matrices
Linear Operator
A linear operator L : U → V is a map from a vector space U toanother vector space V that satis�es :
L (u1 + u2) = L (u1) + L (u2).L (α u) = α L (u) for any scalar α.
If the dimension n of U and m of V are �nite, L can be represented byan m × n matrix A.
A =
A11 A12 . . . A1n
A21 A22 . . . A2n
. . .Am1 Am2 . . . Amn
CS221 Linear Algebra Review 10/11/2011 14 / 37
Matrices
Matrix Vector Multiplication
Au =
A11 A12 . . . A1n
A21 A22 . . . A2n
. . .Am1 Am2 . . . Amn
u1u2. . .un
=
v1v2. . .vm
CS221 Linear Algebra Review 10/11/2011 15 / 37
Matrices
Matrix Vector Multiplication
Au =
A11 A12 . . . A1n
A21 A22 . . . A2n
. . .Am1 Am2 . . . Amn
u1u2. . .un
=
v1v2. . .vm
CS221 Linear Algebra Review 10/11/2011 15 / 37
Matrices
Matrix Vector Multiplication
Au =
A11 A12 . . . A1n
A21 A22 . . . A2n
. . .Am1 Am2 . . . Amn
u1u2. . .un
=
v1v2. . .vm
CS221 Linear Algebra Review 10/11/2011 15 / 37
Matrices
Matrix Vector Multiplication
Au =
A11 A12 . . . A1n
A21 A22 . . . A2n
. . .Am1 Am2 . . . Amn
u1u2. . .un
=
v1v2. . .vm
CS221 Linear Algebra Review 10/11/2011 15 / 37
Matrices
Matrix Product
Applying two linear operators A and B de�nes another linear operator,which can be represented by another matrix AB.
AB =
A11 A12 . . . A1n
A21 A22 . . . A2n
. . .Am1 Am2 . . . Amn
B11 B12 . . . B1p
B21 B22 . . . B2p
. . .Bn1 Bn2 . . . Bnp
=
AB11 AB12 . . . AB1p
AB21 AB22 . . . AB2p
. . .ABm1 ABm2 . . . ABmp
Note that if A is m × n and B is n × p, then AB is m × p.
CS221 Linear Algebra Review 10/11/2011 16 / 37
Matrices
Matrix Product
Applying two linear operators A and B de�nes another linear operator,which can be represented by another matrix AB.
AB =
A11 A12 . . . A1n
A21 A22 . . . A2n
. . .Am1 Am2 . . . Amn
B11 B12 . . . B1p
B21 B22 . . . B2p
. . .Bn1 Bn2 . . . Bnp
=
AB11 AB12 . . . AB1p
AB21 AB22 . . . AB2p
. . .ABm1 ABm2 . . . ABmp
Note that if A is m × n and B is n × p, then AB is m × p.
CS221 Linear Algebra Review 10/11/2011 16 / 37
Matrices
Matrix Product
Applying two linear operators A and B de�nes another linear operator,which can be represented by another matrix AB.
AB =
A11 A12 . . . A1n
A21 A22 . . . A2n
. . .Am1 Am2 . . . Amn
B11 B12 . . . B1p
B21 B22 . . . B2p
. . .Bn1 Bn2 . . . Bnp
=
AB11 AB12 . . . AB1p
AB21 AB22 . . . AB2p
. . .ABm1 ABm2 . . . ABmp
Note that if A is m × n and B is n × p, then AB is m × p.
CS221 Linear Algebra Review 10/11/2011 16 / 37
Matrices
Matrix Product
Applying two linear operators A and B de�nes another linear operator,which can be represented by another matrix AB.
AB =
A11 A12 . . . A1n
A21 A22 . . . A2n
. . .Am1 Am2 . . . Amn
B11 B12 . . . B1p
B21 B22 . . . B2p
. . .Bn1 Bn2 . . . Bnp
=
AB11 AB12 . . . AB1p
AB21 AB22 . . . AB2p
. . .ABm1 ABm2 . . . ABmp
Note that if A is m × n and B is n × p, then AB is m × p.
CS221 Linear Algebra Review 10/11/2011 16 / 37
Matrices
Matrix Product
Applying two linear operators A and B de�nes another linear operator,which can be represented by another matrix AB.
AB =
A11 A12 . . . A1n
A21 A22 . . . A2n
. . .Am1 Am2 . . . Amn
B11 B12 . . . B1p
B21 B22 . . . B2p
. . .Bn1 Bn2 . . . Bnp
=
AB11 AB12 . . . AB1p
AB21 AB22 . . . AB2p
. . .ABm1 ABm2 . . . ABmp
Note that if A is m × n and B is n × p, then AB is m × p.
CS221 Linear Algebra Review 10/11/2011 16 / 37
Matrices
Matrix Product
Applying two linear operators A and B de�nes another linear operator,which can be represented by another matrix AB.
AB =
A11 A12 . . . A1n
A21 A22 . . . A2n
. . .Am1 Am2 . . . Amn
B11 B12 . . . B1p
B21 B22 . . . B2p
. . .Bn1 Bn2 . . . Bnp
=
AB11 AB12 . . . AB1p
AB21 AB22 . . . AB2p
. . .ABm1 ABm2 . . . ABmp
Note that if A is m × n and B is n × p, then AB is m × p.
CS221 Linear Algebra Review 10/11/2011 16 / 37
Matrices
Matrix Product
Applying two linear operators A and B de�nes another linear operator,which can be represented by another matrix AB.
AB =
A11 A12 . . . A1n
A21 A22 . . . A2n
. . .Am1 Am2 . . . Amn
B11 B12 . . . B1p
B21 B22 . . . B2p
. . .Bn1 Bn2 . . . Bnp
=
AB11 AB12 . . . AB1p
AB21 AB22 . . . AB2p
. . .ABm1 ABm2 . . . ABmp
Note that if A is m × n and B is n × p, then AB is m × p.
CS221 Linear Algebra Review 10/11/2011 16 / 37
Matrices
Matrix Product
Applying two linear operators A and B de�nes another linear operator,which can be represented by another matrix AB.
AB =
A11 A12 . . . A1n
A21 A22 . . . A2n
. . .Am1 Am2 . . . Amn
B11 B12 . . . B1p
B21 B22 . . . B2p
. . .Bn1 Bn2 . . . Bnp
=
AB11 AB12 . . . AB1p
AB21 AB22 . . . AB2p
. . .ABm1 ABm2 . . . ABmp
Note that if A is m × n and B is n × p, then AB is m × p.
CS221 Linear Algebra Review 10/11/2011 16 / 37
Matrices
Matrix Product
Applying two linear operators A and B de�nes another linear operator,which can be represented by another matrix AB.
AB =
A11 A12 . . . A1n
A21 A22 . . . A2n
. . .Am1 Am2 . . . Amn
B11 B12 . . . B1p
B21 B22 . . . B2p
. . .Bn1 Bn2 . . . Bnp
=
AB11 AB12 . . . AB1p
AB21 AB22 . . . AB2p
. . .ABm1 ABm2 . . . ABmp
Note that if A is m × n and B is n × p, then AB is m × p.
CS221 Linear Algebra Review 10/11/2011 16 / 37
Matrices
Matrix Product
Applying two linear operators A and B de�nes another linear operator,which can be represented by another matrix AB.
AB =
A11 A12 . . . A1n
A21 A22 . . . A2n
. . .Am1 Am2 . . . Amn
B11 B12 . . . B1p
B21 B22 . . . B2p
. . .Bn1 Bn2 . . . Bnp
=
AB11 AB12 . . . AB1p
AB21 AB22 . . . AB2p
. . .ABm1 ABm2 . . . ABmp
Note that if A is m × n and B is n × p, then AB is m × p.
CS221 Linear Algebra Review 10/11/2011 16 / 37
Matrices
Transpose of a Matrix
The transpose of a matrix is obtained by �ipping the rows andcolumns.
AT =
A11 A21 . . . An1
A12 A22 . . . An2
. . .A1m A2m . . . Anm
Some simple properties :
(AT)T
= A
(A + B)T = AT + BT
(AB)T = BTAT
The inner product for vectors can be represented as 〈u, v〉 = uTv.
CS221 Linear Algebra Review 10/11/2011 17 / 37
Matrices
Transpose of a Matrix
The transpose of a matrix is obtained by �ipping the rows andcolumns.
AT =
A11 A21 . . . An1
A12 A22 . . . An2
. . .A1m A2m . . . Anm
Some simple properties :
(AT)T
= A
(A + B)T = AT + BT
(AB)T = BTAT
The inner product for vectors can be represented as 〈u, v〉 = uTv.
CS221 Linear Algebra Review 10/11/2011 17 / 37
Matrices
Transpose of a Matrix
The transpose of a matrix is obtained by �ipping the rows andcolumns.
AT =
A11 A21 . . . An1
A12 A22 . . . An2
. . .A1m A2m . . . Anm
Some simple properties :(
AT)T
= A
(A + B)T = AT + BT
(AB)T = BTAT
The inner product for vectors can be represented as 〈u, v〉 = uTv.
CS221 Linear Algebra Review 10/11/2011 17 / 37
Matrices
Transpose of a Matrix
The transpose of a matrix is obtained by �ipping the rows andcolumns.
AT =
A11 A21 . . . An1
A12 A22 . . . An2
. . .A1m A2m . . . Anm
Some simple properties :(
AT)T
= A
(A + B)T = AT + BT
(AB)T = BTAT
The inner product for vectors can be represented as 〈u, v〉 = uTv.
CS221 Linear Algebra Review 10/11/2011 17 / 37
Matrices
Transpose of a Matrix
The transpose of a matrix is obtained by �ipping the rows andcolumns.
AT =
A11 A21 . . . An1
A12 A22 . . . An2
. . .A1m A2m . . . Anm
Some simple properties :(
AT)T
= A
(A + B)T = AT + BT
(AB)T = BTAT
The inner product for vectors can be represented as 〈u, v〉 = uTv.
CS221 Linear Algebra Review 10/11/2011 17 / 37
Matrices
Transpose of a Matrix
The transpose of a matrix is obtained by �ipping the rows andcolumns.
AT =
A11 A21 . . . An1
A12 A22 . . . An2
. . .A1m A2m . . . Anm
Some simple properties :(
AT)T
= A
(A + B)T = AT + BT
(AB)T = BTAT
The inner product for vectors can be represented as 〈u, v〉 = uTv.
CS221 Linear Algebra Review 10/11/2011 17 / 37
Matrices
Identity Operator
I =
1 0 . . . 00 1 . . . 0
. . .0 0 . . . 1
For any matrix A, AI = A.
I is the identity operator for the matrix product.
CS221 Linear Algebra Review 10/11/2011 18 / 37
Matrices
Identity Operator
I =
1 0 . . . 00 1 . . . 0
. . .0 0 . . . 1
For any matrix A, AI = A.
I is the identity operator for the matrix product.
CS221 Linear Algebra Review 10/11/2011 18 / 37
Matrices
Identity Operator
I =
1 0 . . . 00 1 . . . 0
. . .0 0 . . . 1
For any matrix A, AI = A.
I is the identity operator for the matrix product.
CS221 Linear Algebra Review 10/11/2011 18 / 37
Matrices
Column Space, Row Space and Rank
Let A be an m × n matrix.
Column space :
Span of the columns of A.Linear subspace of Rm.
Row space :
Span of the rows of A.Linear subspace of Rn.
Important fact :The column and row spaces of any matrix have the same dimension.
This dimension is the rank of the matrix.
CS221 Linear Algebra Review 10/11/2011 19 / 37
Matrices
Column Space, Row Space and Rank
Let A be an m × n matrix.
Column space :
Span of the columns of A.Linear subspace of Rm.
Row space :
Span of the rows of A.Linear subspace of Rn.
Important fact :The column and row spaces of any matrix have the same dimension.
This dimension is the rank of the matrix.
CS221 Linear Algebra Review 10/11/2011 19 / 37
Matrices
Column Space, Row Space and Rank
Let A be an m × n matrix.
Column space :
Span of the columns of A.
Linear subspace of Rm.
Row space :
Span of the rows of A.Linear subspace of Rn.
Important fact :The column and row spaces of any matrix have the same dimension.
This dimension is the rank of the matrix.
CS221 Linear Algebra Review 10/11/2011 19 / 37
Matrices
Column Space, Row Space and Rank
Let A be an m × n matrix.
Column space :
Span of the columns of A.Linear subspace of Rm.
Row space :
Span of the rows of A.Linear subspace of Rn.
Important fact :The column and row spaces of any matrix have the same dimension.
This dimension is the rank of the matrix.
CS221 Linear Algebra Review 10/11/2011 19 / 37
Matrices
Column Space, Row Space and Rank
Let A be an m × n matrix.
Column space :
Span of the columns of A.Linear subspace of Rm.
Row space :
Span of the rows of A.Linear subspace of Rn.
Important fact :The column and row spaces of any matrix have the same dimension.
This dimension is the rank of the matrix.
CS221 Linear Algebra Review 10/11/2011 19 / 37
Matrices
Column Space, Row Space and Rank
Let A be an m × n matrix.
Column space :
Span of the columns of A.Linear subspace of Rm.
Row space :
Span of the rows of A.
Linear subspace of Rn.
Important fact :The column and row spaces of any matrix have the same dimension.
This dimension is the rank of the matrix.
CS221 Linear Algebra Review 10/11/2011 19 / 37
Matrices
Column Space, Row Space and Rank
Let A be an m × n matrix.
Column space :
Span of the columns of A.Linear subspace of Rm.
Row space :
Span of the rows of A.Linear subspace of Rn.
Important fact :The column and row spaces of any matrix have the same dimension.
This dimension is the rank of the matrix.
CS221 Linear Algebra Review 10/11/2011 19 / 37
Matrices
Column Space, Row Space and Rank
Let A be an m × n matrix.
Column space :
Span of the columns of A.Linear subspace of Rm.
Row space :
Span of the rows of A.Linear subspace of Rn.
Important fact :The column and row spaces of any matrix have the same dimension.
This dimension is the rank of the matrix.
CS221 Linear Algebra Review 10/11/2011 19 / 37
Matrices
Column Space, Row Space and Rank
Let A be an m × n matrix.
Column space :
Span of the columns of A.Linear subspace of Rm.
Row space :
Span of the rows of A.Linear subspace of Rn.
Important fact :The column and row spaces of any matrix have the same dimension.
This dimension is the rank of the matrix.
CS221 Linear Algebra Review 10/11/2011 19 / 37
Matrices
Range and Null space
Let A be an m × n matrix.
Range :
Set of vectors equal to Au for some u ∈ Rn.Linear subspace of Rm.
Null space :
u belongs to the null space if Au = 0.Subspace of Rn.Every vector in the null space is orthogonal to the rows of A.The null space and row space of a matrix are orthogonal.
CS221 Linear Algebra Review 10/11/2011 20 / 37
Matrices
Range and Null space
Let A be an m × n matrix.
Range :
Set of vectors equal to Au for some u ∈ Rn.Linear subspace of Rm.
Null space :
u belongs to the null space if Au = 0.Subspace of Rn.Every vector in the null space is orthogonal to the rows of A.The null space and row space of a matrix are orthogonal.
CS221 Linear Algebra Review 10/11/2011 20 / 37
Matrices
Range and Null space
Let A be an m × n matrix.
Range :
Set of vectors equal to Au for some u ∈ Rn.
Linear subspace of Rm.
Null space :
u belongs to the null space if Au = 0.Subspace of Rn.Every vector in the null space is orthogonal to the rows of A.The null space and row space of a matrix are orthogonal.
CS221 Linear Algebra Review 10/11/2011 20 / 37
Matrices
Range and Null space
Let A be an m × n matrix.
Range :
Set of vectors equal to Au for some u ∈ Rn.Linear subspace of Rm.
Null space :
u belongs to the null space if Au = 0.Subspace of Rn.Every vector in the null space is orthogonal to the rows of A.The null space and row space of a matrix are orthogonal.
CS221 Linear Algebra Review 10/11/2011 20 / 37
Matrices
Range and Null space
Let A be an m × n matrix.
Range :
Set of vectors equal to Au for some u ∈ Rn.Linear subspace of Rm.
Null space :
u belongs to the null space if Au = 0.Subspace of Rn.Every vector in the null space is orthogonal to the rows of A.The null space and row space of a matrix are orthogonal.
CS221 Linear Algebra Review 10/11/2011 20 / 37
Matrices
Range and Null space
Let A be an m × n matrix.
Range :
Set of vectors equal to Au for some u ∈ Rn.Linear subspace of Rm.
Null space :
u belongs to the null space if Au = 0.
Subspace of Rn.Every vector in the null space is orthogonal to the rows of A.The null space and row space of a matrix are orthogonal.
CS221 Linear Algebra Review 10/11/2011 20 / 37
Matrices
Range and Null space
Let A be an m × n matrix.
Range :
Set of vectors equal to Au for some u ∈ Rn.Linear subspace of Rm.
Null space :
u belongs to the null space if Au = 0.Subspace of Rn.
Every vector in the null space is orthogonal to the rows of A.The null space and row space of a matrix are orthogonal.
CS221 Linear Algebra Review 10/11/2011 20 / 37
Matrices
Range and Null space
Let A be an m × n matrix.
Range :
Set of vectors equal to Au for some u ∈ Rn.Linear subspace of Rm.
Null space :
u belongs to the null space if Au = 0.Subspace of Rn.Every vector in the null space is orthogonal to the rows of A.The null space and row space of a matrix are orthogonal.
CS221 Linear Algebra Review 10/11/2011 20 / 37
Matrices
Range and Column Space
Another interpretation of the matrix vector product (Ai is column i of A) :
Au =
(A1 A2 . . . An
)u1u2. . .un
= u1A1 + u2A2 + · · ·+ unAn
The result is a linear combination of the columns of A.
For any matrix, the range is equal to the column space.
CS221 Linear Algebra Review 10/11/2011 21 / 37
Matrices
Range and Column Space
Another interpretation of the matrix vector product (Ai is column i of A) :
Au =
(A1 A2 . . . An
)u1u2. . .un
= u1A1 + u2A2 + · · ·+ unAn
The result is a linear combination of the columns of A.
For any matrix, the range is equal to the column space.
CS221 Linear Algebra Review 10/11/2011 21 / 37
Matrices
Range and Column Space
Another interpretation of the matrix vector product (Ai is column i of A) :
Au =
(A1 A2 . . . An
)u1u2. . .un
= u1A1 + u2A2 + · · ·+ unAn
The result is a linear combination of the columns of A.
For any matrix, the range is equal to the column space.
CS221 Linear Algebra Review 10/11/2011 21 / 37
Matrices
Matrix Inverse
For an n × n matrix A : rank + dim(null space) = n.
If dim(null space)=0 then A is full rank.
In this case, the action of the matrix is invertible.
The inversion is also linear and consequently can be represented byanother matrix A−1.
A−1 is the only matrix such that A−1A = AA−1 = I.
CS221 Linear Algebra Review 10/11/2011 22 / 37
Matrices
Matrix Inverse
For an n × n matrix A : rank + dim(null space) = n.
If dim(null space)=0 then A is full rank.
In this case, the action of the matrix is invertible.
The inversion is also linear and consequently can be represented byanother matrix A−1.
A−1 is the only matrix such that A−1A = AA−1 = I.
CS221 Linear Algebra Review 10/11/2011 22 / 37
Matrices
Matrix Inverse
For an n × n matrix A : rank + dim(null space) = n.
If dim(null space)=0 then A is full rank.
In this case, the action of the matrix is invertible.
The inversion is also linear and consequently can be represented byanother matrix A−1.
A−1 is the only matrix such that A−1A = AA−1 = I.
CS221 Linear Algebra Review 10/11/2011 22 / 37
Matrices
Matrix Inverse
For an n × n matrix A : rank + dim(null space) = n.
If dim(null space)=0 then A is full rank.
In this case, the action of the matrix is invertible.
The inversion is also linear and consequently can be represented byanother matrix A−1.
A−1 is the only matrix such that A−1A = AA−1 = I.
CS221 Linear Algebra Review 10/11/2011 22 / 37
Matrices
Matrix Inverse
For an n × n matrix A : rank + dim(null space) = n.
If dim(null space)=0 then A is full rank.
In this case, the action of the matrix is invertible.
The inversion is also linear and consequently can be represented byanother matrix A−1.
A−1 is the only matrix such that A−1A = AA−1 = I.
CS221 Linear Algebra Review 10/11/2011 22 / 37
Matrices
Orthogonal Matrices
An orthogonal matrix U satis�es UTU = I.
Equivalently, U has orthonormal columns.
Applying an orthogonal matrix to two vectors does not change theirinner product :
〈Uu,Uv〉 = (Uu)T Uv
= uTUTUv
= uTv
= 〈u, v〉
Example : Matrices that represent rotations are orthogonal.
CS221 Linear Algebra Review 10/11/2011 23 / 37
Matrices
Orthogonal Matrices
An orthogonal matrix U satis�es UTU = I.
Equivalently, U has orthonormal columns.
Applying an orthogonal matrix to two vectors does not change theirinner product :
〈Uu,Uv〉 = (Uu)T Uv
= uTUTUv
= uTv
= 〈u, v〉
Example : Matrices that represent rotations are orthogonal.
CS221 Linear Algebra Review 10/11/2011 23 / 37
Matrices
Orthogonal Matrices
An orthogonal matrix U satis�es UTU = I.
Equivalently, U has orthonormal columns.
Applying an orthogonal matrix to two vectors does not change theirinner product :
〈Uu,Uv〉 = (Uu)T Uv
= uTUTUv
= uTv
= 〈u, v〉
Example : Matrices that represent rotations are orthogonal.
CS221 Linear Algebra Review 10/11/2011 23 / 37
Matrices
Orthogonal Matrices
An orthogonal matrix U satis�es UTU = I.
Equivalently, U has orthonormal columns.
Applying an orthogonal matrix to two vectors does not change theirinner product :
〈Uu,Uv〉 = (Uu)T Uv
= uTUTUv
= uTv
= 〈u, v〉
Example : Matrices that represent rotations are orthogonal.
CS221 Linear Algebra Review 10/11/2011 23 / 37
Matrices
Trace and Determinant
The trace is the sum of the diagonal elements of a square matrix.
The determinant of a square matrix A is denoted by |A|.For a 2× 2 matrix A =
(a bc d
):
|A| = ad − bc
The absolute value of |A| is the area of the parallelogram given by therows of A.
CS221 Linear Algebra Review 10/11/2011 24 / 37
Matrices
Trace and Determinant
The trace is the sum of the diagonal elements of a square matrix.
The determinant of a square matrix A is denoted by |A|.
For a 2× 2 matrix A =(a bc d
):
|A| = ad − bc
The absolute value of |A| is the area of the parallelogram given by therows of A.
CS221 Linear Algebra Review 10/11/2011 24 / 37
Matrices
Trace and Determinant
The trace is the sum of the diagonal elements of a square matrix.
The determinant of a square matrix A is denoted by |A|.For a 2× 2 matrix A =
(a bc d
):
|A| = ad − bc
The absolute value of |A| is the area of the parallelogram given by therows of A.
CS221 Linear Algebra Review 10/11/2011 24 / 37
Matrices
Trace and Determinant
The trace is the sum of the diagonal elements of a square matrix.
The determinant of a square matrix A is denoted by |A|.For a 2× 2 matrix A =
(a bc d
):
|A| = ad − bc
The absolute value of |A| is the area of the parallelogram given by therows of A.
CS221 Linear Algebra Review 10/11/2011 24 / 37
Matrices
Trace and Determinant
The trace is the sum of the diagonal elements of a square matrix.
The determinant of a square matrix A is denoted by |A|.For a 2× 2 matrix A =
(a bc d
):
|A| = ad − bc
The absolute value of |A| is the area of the parallelogram given by therows of A.
CS221 Linear Algebra Review 10/11/2011 24 / 37
Matrices
Properties of the Determinant
The de�nition can be generalized to larger matrices.
|A| =∣∣AT
∣∣.|AB| = |A| |B||A| = 0 if and only if A is not invertible.
If A is invertible, then∣∣A−1∣∣ = 1
|A| .
CS221 Linear Algebra Review 10/11/2011 25 / 37
Matrices
Properties of the Determinant
The de�nition can be generalized to larger matrices.
|A| =∣∣AT
∣∣.
|AB| = |A| |B||A| = 0 if and only if A is not invertible.
If A is invertible, then∣∣A−1∣∣ = 1
|A| .
CS221 Linear Algebra Review 10/11/2011 25 / 37
Matrices
Properties of the Determinant
The de�nition can be generalized to larger matrices.
|A| =∣∣AT
∣∣.|AB| = |A| |B|
|A| = 0 if and only if A is not invertible.
If A is invertible, then∣∣A−1∣∣ = 1
|A| .
CS221 Linear Algebra Review 10/11/2011 25 / 37
Matrices
Properties of the Determinant
The de�nition can be generalized to larger matrices.
|A| =∣∣AT
∣∣.|AB| = |A| |B||A| = 0 if and only if A is not invertible.
If A is invertible, then∣∣A−1∣∣ = 1
|A| .
CS221 Linear Algebra Review 10/11/2011 25 / 37
Matrices
Properties of the Determinant
The de�nition can be generalized to larger matrices.
|A| =∣∣AT
∣∣.|AB| = |A| |B||A| = 0 if and only if A is not invertible.
If A is invertible, then∣∣A−1∣∣ = 1
|A| .
CS221 Linear Algebra Review 10/11/2011 25 / 37
Matrix Decompositions
1 Vectors
2 Matrices
3 Matrix Decompositions
4 Application : Image Compression
CS221 Linear Algebra Review 10/11/2011 26 / 37
Matrix Decompositions
Eigenvalues and Eigenvectors
An eigenvalue λ of a square matrix A satis�es :
Au = λu
for some vector u, which we call an eigenvector.
Geometrically, the operator A expands (λ > 1) or contracts (λ < 1)eigenvectors but does not rotate them.
If u is an eigenvector of A, it is in the null space of A− λI, which isconsequently not invertible.
The eigenvalues of A are the roots of the equation |A− λI| = 0.
Not the way eigenvalues are calculated in practice.
Eigenvalues and eigenvectors can be complex valued, even if all theentries of A are real.
CS221 Linear Algebra Review 10/11/2011 27 / 37
Matrix Decompositions
Eigenvalues and Eigenvectors
An eigenvalue λ of a square matrix A satis�es :
Au = λu
for some vector u, which we call an eigenvector.
Geometrically, the operator A expands (λ > 1) or contracts (λ < 1)eigenvectors but does not rotate them.
If u is an eigenvector of A, it is in the null space of A− λI, which isconsequently not invertible.
The eigenvalues of A are the roots of the equation |A− λI| = 0.
Not the way eigenvalues are calculated in practice.
Eigenvalues and eigenvectors can be complex valued, even if all theentries of A are real.
CS221 Linear Algebra Review 10/11/2011 27 / 37
Matrix Decompositions
Eigenvalues and Eigenvectors
An eigenvalue λ of a square matrix A satis�es :
Au = λu
for some vector u, which we call an eigenvector.
Geometrically, the operator A expands (λ > 1) or contracts (λ < 1)eigenvectors but does not rotate them.
If u is an eigenvector of A, it is in the null space of A− λI, which isconsequently not invertible.
The eigenvalues of A are the roots of the equation |A− λI| = 0.
Not the way eigenvalues are calculated in practice.
Eigenvalues and eigenvectors can be complex valued, even if all theentries of A are real.
CS221 Linear Algebra Review 10/11/2011 27 / 37
Matrix Decompositions
Eigenvalues and Eigenvectors
An eigenvalue λ of a square matrix A satis�es :
Au = λu
for some vector u, which we call an eigenvector.
Geometrically, the operator A expands (λ > 1) or contracts (λ < 1)eigenvectors but does not rotate them.
If u is an eigenvector of A, it is in the null space of A− λI, which isconsequently not invertible.
The eigenvalues of A are the roots of the equation |A− λI| = 0.
Not the way eigenvalues are calculated in practice.
Eigenvalues and eigenvectors can be complex valued, even if all theentries of A are real.
CS221 Linear Algebra Review 10/11/2011 27 / 37
Matrix Decompositions
Eigenvalues and Eigenvectors
An eigenvalue λ of a square matrix A satis�es :
Au = λu
for some vector u, which we call an eigenvector.
Geometrically, the operator A expands (λ > 1) or contracts (λ < 1)eigenvectors but does not rotate them.
If u is an eigenvector of A, it is in the null space of A− λI, which isconsequently not invertible.
The eigenvalues of A are the roots of the equation |A− λI| = 0.
Not the way eigenvalues are calculated in practice.
Eigenvalues and eigenvectors can be complex valued, even if all theentries of A are real.
CS221 Linear Algebra Review 10/11/2011 27 / 37
Matrix Decompositions
Eigenvalues and Eigenvectors
An eigenvalue λ of a square matrix A satis�es :
Au = λu
for some vector u, which we call an eigenvector.
Geometrically, the operator A expands (λ > 1) or contracts (λ < 1)eigenvectors but does not rotate them.
If u is an eigenvector of A, it is in the null space of A− λI, which isconsequently not invertible.
The eigenvalues of A are the roots of the equation |A− λI| = 0.
Not the way eigenvalues are calculated in practice.
Eigenvalues and eigenvectors can be complex valued, even if all theentries of A are real.
CS221 Linear Algebra Review 10/11/2011 27 / 37
Matrix Decompositions
Eigendecomposition of a Matrix
Let A be an n × n square matrix with n linearly independenteigenvectors p1 . . .pn and eigenvalues λ1 . . . λn.
We de�ne the matrices :
P =
(p1 . . . pn
)
Λ =
λ1 0 . . . 00 λ2 . . . 0
. . .0 0 . . . λn
A satis�es AP = PΛ.
P is full rank. Inverting it yields the eigendecomposition of A :
A = PΛP−1
CS221 Linear Algebra Review 10/11/2011 28 / 37
Matrix Decompositions
Eigendecomposition of a Matrix
Let A be an n × n square matrix with n linearly independenteigenvectors p1 . . .pn and eigenvalues λ1 . . . λn.
We de�ne the matrices :
P =
(p1 . . . pn
)
Λ =
λ1 0 . . . 00 λ2 . . . 0
. . .0 0 . . . λn
A satis�es AP = PΛ.
P is full rank. Inverting it yields the eigendecomposition of A :
A = PΛP−1
CS221 Linear Algebra Review 10/11/2011 28 / 37
Matrix Decompositions
Eigendecomposition of a Matrix
Let A be an n × n square matrix with n linearly independenteigenvectors p1 . . .pn and eigenvalues λ1 . . . λn.
We de�ne the matrices :
P =
(p1 . . . pn
)
Λ =
λ1 0 . . . 00 λ2 . . . 0
. . .0 0 . . . λn
A satis�es AP = PΛ.
P is full rank. Inverting it yields the eigendecomposition of A :
A = PΛP−1
CS221 Linear Algebra Review 10/11/2011 28 / 37
Matrix Decompositions
Eigendecomposition of a Matrix
Let A be an n × n square matrix with n linearly independenteigenvectors p1 . . .pn and eigenvalues λ1 . . . λn.
We de�ne the matrices :
P =
(p1 . . . pn
)
Λ =
λ1 0 . . . 00 λ2 . . . 0
. . .0 0 . . . λn
A satis�es AP = PΛ.
P is full rank. Inverting it yields the eigendecomposition of A :
A = PΛP−1
CS221 Linear Algebra Review 10/11/2011 28 / 37
Matrix Decompositions
Properties of the Eigendecomposition
Not all matrices are diagonalizable (have an eigendecomposition).Example : ( 1 1
0 1).
Trace(A) = Trace(Λ) =∑n
i=1λi .
|A| = |Λ| =∏n
i=1λi .
The rank of A is equal to the number of nonzero eigenvalues.
If λ is a nonzero eigenvalue of A, 1
λ is an eigenvalue of A−1 with thesame eigenvector.
The eigendecomposition makes it possible to compute matrix powerse�ciently :Am =
(PΛP−1
)m= PΛP−1 PΛP−1 . . .PΛP−1 = PΛmP−1.
CS221 Linear Algebra Review 10/11/2011 29 / 37
Matrix Decompositions
Properties of the Eigendecomposition
Not all matrices are diagonalizable (have an eigendecomposition).Example : ( 1 1
0 1).
Trace(A) = Trace(Λ) =∑n
i=1λi .
|A| = |Λ| =∏n
i=1λi .
The rank of A is equal to the number of nonzero eigenvalues.
If λ is a nonzero eigenvalue of A, 1
λ is an eigenvalue of A−1 with thesame eigenvector.
The eigendecomposition makes it possible to compute matrix powerse�ciently :Am =
(PΛP−1
)m= PΛP−1 PΛP−1 . . .PΛP−1 = PΛmP−1.
CS221 Linear Algebra Review 10/11/2011 29 / 37
Matrix Decompositions
Properties of the Eigendecomposition
Not all matrices are diagonalizable (have an eigendecomposition).Example : ( 1 1
0 1).
Trace(A) = Trace(Λ) =∑n
i=1λi .
|A| = |Λ| =∏n
i=1λi .
The rank of A is equal to the number of nonzero eigenvalues.
If λ is a nonzero eigenvalue of A, 1
λ is an eigenvalue of A−1 with thesame eigenvector.
The eigendecomposition makes it possible to compute matrix powerse�ciently :Am =
(PΛP−1
)m= PΛP−1 PΛP−1 . . .PΛP−1 = PΛmP−1.
CS221 Linear Algebra Review 10/11/2011 29 / 37
Matrix Decompositions
Properties of the Eigendecomposition
Not all matrices are diagonalizable (have an eigendecomposition).Example : ( 1 1
0 1).
Trace(A) = Trace(Λ) =∑n
i=1λi .
|A| = |Λ| =∏n
i=1λi .
The rank of A is equal to the number of nonzero eigenvalues.
If λ is a nonzero eigenvalue of A, 1
λ is an eigenvalue of A−1 with thesame eigenvector.
The eigendecomposition makes it possible to compute matrix powerse�ciently :Am =
(PΛP−1
)m= PΛP−1 PΛP−1 . . .PΛP−1 = PΛmP−1.
CS221 Linear Algebra Review 10/11/2011 29 / 37
Matrix Decompositions
Properties of the Eigendecomposition
Not all matrices are diagonalizable (have an eigendecomposition).Example : ( 1 1
0 1).
Trace(A) = Trace(Λ) =∑n
i=1λi .
|A| = |Λ| =∏n
i=1λi .
The rank of A is equal to the number of nonzero eigenvalues.
If λ is a nonzero eigenvalue of A, 1
λ is an eigenvalue of A−1 with thesame eigenvector.
The eigendecomposition makes it possible to compute matrix powerse�ciently :Am =
(PΛP−1
)m= PΛP−1 PΛP−1 . . .PΛP−1 = PΛmP−1.
CS221 Linear Algebra Review 10/11/2011 29 / 37
Matrix Decompositions
Properties of the Eigendecomposition
Not all matrices are diagonalizable (have an eigendecomposition).Example : ( 1 1
0 1).
Trace(A) = Trace(Λ) =∑n
i=1λi .
|A| = |Λ| =∏n
i=1λi .
The rank of A is equal to the number of nonzero eigenvalues.
If λ is a nonzero eigenvalue of A, 1
λ is an eigenvalue of A−1 with thesame eigenvector.
The eigendecomposition makes it possible to compute matrix powerse�ciently :Am =
(PΛP−1
)m= PΛP−1 PΛP−1 . . .PΛP−1 = PΛmP−1.
CS221 Linear Algebra Review 10/11/2011 29 / 37
Matrix Decompositions
Matlab Example
CS221 Linear Algebra Review 10/11/2011 30 / 37
Matrix Decompositions
Eigendecomposition of a Symmetric Matrix
If A = AT then A is symmetric.
The eigenvalues of symmetric matrices are real.
The eigenvectors of symmetric matrices are orthonormal.
Consequently, the eigendecomposition becomes :A = UΛUT for Λ real and U orthogonal.
The eigenvectors of A are an orthonormal basis for the column space(and the row space, since they are equal).
CS221 Linear Algebra Review 10/11/2011 31 / 37
Matrix Decompositions
Eigendecomposition of a Symmetric Matrix
If A = AT then A is symmetric.
The eigenvalues of symmetric matrices are real.
The eigenvectors of symmetric matrices are orthonormal.
Consequently, the eigendecomposition becomes :A = UΛUT for Λ real and U orthogonal.
The eigenvectors of A are an orthonormal basis for the column space(and the row space, since they are equal).
CS221 Linear Algebra Review 10/11/2011 31 / 37
Matrix Decompositions
Eigendecomposition of a Symmetric Matrix
If A = AT then A is symmetric.
The eigenvalues of symmetric matrices are real.
The eigenvectors of symmetric matrices are orthonormal.
Consequently, the eigendecomposition becomes :A = UΛUT for Λ real and U orthogonal.
The eigenvectors of A are an orthonormal basis for the column space(and the row space, since they are equal).
CS221 Linear Algebra Review 10/11/2011 31 / 37
Matrix Decompositions
Eigendecomposition of a Symmetric Matrix
If A = AT then A is symmetric.
The eigenvalues of symmetric matrices are real.
The eigenvectors of symmetric matrices are orthonormal.
Consequently, the eigendecomposition becomes :A = UΛUT for Λ real and U orthogonal.
The eigenvectors of A are an orthonormal basis for the column space(and the row space, since they are equal).
CS221 Linear Algebra Review 10/11/2011 31 / 37
Matrix Decompositions
Eigendecomposition of a Symmetric Matrix
If A = AT then A is symmetric.
The eigenvalues of symmetric matrices are real.
The eigenvectors of symmetric matrices are orthonormal.
Consequently, the eigendecomposition becomes :A = UΛUT for Λ real and U orthogonal.
The eigenvectors of A are an orthonormal basis for the column space(and the row space, since they are equal).
CS221 Linear Algebra Review 10/11/2011 31 / 37
Matrix Decompositions
Eigendecomposition of a Symmetric Matrix
The action of a symmetric matrix on a vector Au = UΛUTu can bedecomposed into :
1 Projection of u onto the column space of A (multiplication by UT ).2 Scaling of each coe�cient 〈Ui ,u〉 by the corresponding eigenvalue
(multiplication by Λ).3 Linear combination of the eigenvectors scaled by the resulting
coe�cient (multiplication by U).
Summarizing :
Au =n∑
i=1
λi 〈Ui ,u〉Ui
It would be great to generalize this to all matrices...
CS221 Linear Algebra Review 10/11/2011 32 / 37
Matrix Decompositions
Eigendecomposition of a Symmetric Matrix
The action of a symmetric matrix on a vector Au = UΛUTu can bedecomposed into :
1 Projection of u onto the column space of A (multiplication by UT ).
2 Scaling of each coe�cient 〈Ui ,u〉 by the corresponding eigenvalue(multiplication by Λ).
3 Linear combination of the eigenvectors scaled by the resultingcoe�cient (multiplication by U).
Summarizing :
Au =n∑
i=1
λi 〈Ui ,u〉Ui
It would be great to generalize this to all matrices...
CS221 Linear Algebra Review 10/11/2011 32 / 37
Matrix Decompositions
Eigendecomposition of a Symmetric Matrix
The action of a symmetric matrix on a vector Au = UΛUTu can bedecomposed into :
1 Projection of u onto the column space of A (multiplication by UT ).2 Scaling of each coe�cient 〈Ui ,u〉 by the corresponding eigenvalue
(multiplication by Λ).
3 Linear combination of the eigenvectors scaled by the resultingcoe�cient (multiplication by U).
Summarizing :
Au =n∑
i=1
λi 〈Ui ,u〉Ui
It would be great to generalize this to all matrices...
CS221 Linear Algebra Review 10/11/2011 32 / 37
Matrix Decompositions
Eigendecomposition of a Symmetric Matrix
The action of a symmetric matrix on a vector Au = UΛUTu can bedecomposed into :
1 Projection of u onto the column space of A (multiplication by UT ).2 Scaling of each coe�cient 〈Ui ,u〉 by the corresponding eigenvalue
(multiplication by Λ).3 Linear combination of the eigenvectors scaled by the resulting
coe�cient (multiplication by U).
Summarizing :
Au =n∑
i=1
λi 〈Ui ,u〉Ui
It would be great to generalize this to all matrices...
CS221 Linear Algebra Review 10/11/2011 32 / 37
Matrix Decompositions
Eigendecomposition of a Symmetric Matrix
The action of a symmetric matrix on a vector Au = UΛUTu can bedecomposed into :
1 Projection of u onto the column space of A (multiplication by UT ).2 Scaling of each coe�cient 〈Ui ,u〉 by the corresponding eigenvalue
(multiplication by Λ).3 Linear combination of the eigenvectors scaled by the resulting
coe�cient (multiplication by U).
Summarizing :
Au =n∑
i=1
λi 〈Ui ,u〉Ui
It would be great to generalize this to all matrices...
CS221 Linear Algebra Review 10/11/2011 32 / 37
Matrix Decompositions
Eigendecomposition of a Symmetric Matrix
The action of a symmetric matrix on a vector Au = UΛUTu can bedecomposed into :
1 Projection of u onto the column space of A (multiplication by UT ).2 Scaling of each coe�cient 〈Ui ,u〉 by the corresponding eigenvalue
(multiplication by Λ).3 Linear combination of the eigenvectors scaled by the resulting
coe�cient (multiplication by U).
Summarizing :
Au =n∑
i=1
λi 〈Ui ,u〉Ui
It would be great to generalize this to all matrices...
CS221 Linear Algebra Review 10/11/2011 32 / 37
Matrix Decompositions
Singular Value Decomposition
Every matrix has a singular value decomposition :
A = UΣVT
The columns of U are an orthonormal basis for the column space ofA.
The columns of V are an orthonormal basis for the row space of A.
Σ is diagonal. The entries on its diagonal σi = Σii are positive realnumbers, called the singular values of A.
The action of A on a vector u can be decomposed into :
Au =n∑
i=1
σi 〈Vi ,u〉Ui
CS221 Linear Algebra Review 10/11/2011 33 / 37
Matrix Decompositions
Singular Value Decomposition
Every matrix has a singular value decomposition :
A = UΣVT
The columns of U are an orthonormal basis for the column space ofA.
The columns of V are an orthonormal basis for the row space of A.
Σ is diagonal. The entries on its diagonal σi = Σii are positive realnumbers, called the singular values of A.
The action of A on a vector u can be decomposed into :
Au =n∑
i=1
σi 〈Vi ,u〉Ui
CS221 Linear Algebra Review 10/11/2011 33 / 37
Matrix Decompositions
Singular Value Decomposition
Every matrix has a singular value decomposition :
A = UΣVT
The columns of U are an orthonormal basis for the column space ofA.
The columns of V are an orthonormal basis for the row space of A.
Σ is diagonal. The entries on its diagonal σi = Σii are positive realnumbers, called the singular values of A.
The action of A on a vector u can be decomposed into :
Au =n∑
i=1
σi 〈Vi ,u〉Ui
CS221 Linear Algebra Review 10/11/2011 33 / 37
Matrix Decompositions
Singular Value Decomposition
Every matrix has a singular value decomposition :
A = UΣVT
The columns of U are an orthonormal basis for the column space ofA.
The columns of V are an orthonormal basis for the row space of A.
Σ is diagonal. The entries on its diagonal σi = Σii are positive realnumbers, called the singular values of A.
The action of A on a vector u can be decomposed into :
Au =n∑
i=1
σi 〈Vi ,u〉Ui
CS221 Linear Algebra Review 10/11/2011 33 / 37
Matrix Decompositions
Singular Value Decomposition
Every matrix has a singular value decomposition :
A = UΣVT
The columns of U are an orthonormal basis for the column space ofA.
The columns of V are an orthonormal basis for the row space of A.
Σ is diagonal. The entries on its diagonal σi = Σii are positive realnumbers, called the singular values of A.
The action of A on a vector u can be decomposed into :
Au =n∑
i=1
σi 〈Vi ,u〉Ui
CS221 Linear Algebra Review 10/11/2011 33 / 37
Matrix Decompositions
Properties of the Singular Value Decomposition
The eigenvalues of the symmetric matrix ATA are equal to thesquare of the singular values of A :
ATA = VΣUTUΣVT = VΣ2VT
The rank of a matrix is equal to the number of nonzero singularvalues.
The largest singular value σ1 is the solution to the optimizationproblem :
σ1 = maxx 6=0
||Ax||2
||x||2
It can be veri�ed that the largest singular value satis�es the propertiesof a norm, it is called the spectral norm of the matrix.
In statistics analyzing data with the singular value decomposition iscalled Principal Component Analysis.
CS221 Linear Algebra Review 10/11/2011 34 / 37
Matrix Decompositions
Properties of the Singular Value Decomposition
The eigenvalues of the symmetric matrix ATA are equal to thesquare of the singular values of A :
ATA = VΣUTUΣVT = VΣ2VT
The rank of a matrix is equal to the number of nonzero singularvalues.
The largest singular value σ1 is the solution to the optimizationproblem :
σ1 = maxx 6=0
||Ax||2
||x||2
It can be veri�ed that the largest singular value satis�es the propertiesof a norm, it is called the spectral norm of the matrix.
In statistics analyzing data with the singular value decomposition iscalled Principal Component Analysis.
CS221 Linear Algebra Review 10/11/2011 34 / 37
Matrix Decompositions
Properties of the Singular Value Decomposition
The eigenvalues of the symmetric matrix ATA are equal to thesquare of the singular values of A :
ATA = VΣUTUΣVT = VΣ2VT
The rank of a matrix is equal to the number of nonzero singularvalues.
The largest singular value σ1 is the solution to the optimizationproblem :
σ1 = maxx 6=0
||Ax||2
||x||2
It can be veri�ed that the largest singular value satis�es the propertiesof a norm, it is called the spectral norm of the matrix.
In statistics analyzing data with the singular value decomposition iscalled Principal Component Analysis.
CS221 Linear Algebra Review 10/11/2011 34 / 37
Matrix Decompositions
Properties of the Singular Value Decomposition
The eigenvalues of the symmetric matrix ATA are equal to thesquare of the singular values of A :
ATA = VΣUTUΣVT = VΣ2VT
The rank of a matrix is equal to the number of nonzero singularvalues.
The largest singular value σ1 is the solution to the optimizationproblem :
σ1 = maxx 6=0
||Ax||2
||x||2
It can be veri�ed that the largest singular value satis�es the propertiesof a norm, it is called the spectral norm of the matrix.
In statistics analyzing data with the singular value decomposition iscalled Principal Component Analysis.
CS221 Linear Algebra Review 10/11/2011 34 / 37
Matrix Decompositions
Properties of the Singular Value Decomposition
The eigenvalues of the symmetric matrix ATA are equal to thesquare of the singular values of A :
ATA = VΣUTUΣVT = VΣ2VT
The rank of a matrix is equal to the number of nonzero singularvalues.
The largest singular value σ1 is the solution to the optimizationproblem :
σ1 = maxx 6=0
||Ax||2
||x||2
It can be veri�ed that the largest singular value satis�es the propertiesof a norm, it is called the spectral norm of the matrix.
In statistics analyzing data with the singular value decomposition iscalled Principal Component Analysis.
CS221 Linear Algebra Review 10/11/2011 34 / 37
Application : Image Compression
1 Vectors
2 Matrices
3 Matrix Decompositions
4 Application : Image Compression
CS221 Linear Algebra Review 10/11/2011 35 / 37
Application : Image Compression
Compression using the Singular Value Decomposition
The singular value decomposition can be viewed as a sum of rank 1matrices :
If some of the singular values are very small, we can discard them.
This is a form of lossy compression.
CS221 Linear Algebra Review 10/11/2011 36 / 37
Application : Image Compression
Compression using the Singular Value Decomposition
The singular value decomposition can be viewed as a sum of rank 1matrices :
If some of the singular values are very small, we can discard them.
This is a form of lossy compression.
CS221 Linear Algebra Review 10/11/2011 36 / 37
Application : Image Compression
Compression using the Singular Value Decomposition
The singular value decomposition can be viewed as a sum of rank 1matrices :
If some of the singular values are very small, we can discard them.
This is a form of lossy compression.
CS221 Linear Algebra Review 10/11/2011 36 / 37
Application : Image Compression
Compression using the Singular Value Decomposition
Truncating the singular value decomposition allows us to represent thematrix with less parameters.
For a 512× 512 matrix :
Full representation : 512× 512 =262 144
Rank 10 approximation : 512× 10 + 10 + 512× 10 =10 250
Rank 40 approximation : 512× 40 + 40 + 512× 40 =41 000
Rank 80 approximation : 512× 80 + 80 + 512× 80 =82 000
CS221 Linear Algebra Review 10/11/2011 37 / 37
Application : Image Compression
Compression using the Singular Value Decomposition
Truncating the singular value decomposition allows us to represent thematrix with less parameters.
For a 512× 512 matrix :
Full representation : 512× 512 =262 144
Rank 10 approximation : 512× 10 + 10 + 512× 10 =10 250
Rank 40 approximation : 512× 40 + 40 + 512× 40 =41 000
Rank 80 approximation : 512× 80 + 80 + 512× 80 =82 000
CS221 Linear Algebra Review 10/11/2011 37 / 37
Application : Image Compression
Compression using the Singular Value Decomposition
Truncating the singular value decomposition allows us to represent thematrix with less parameters.
For a 512× 512 matrix :
Full representation : 512× 512 =262 144
Rank 10 approximation : 512× 10 + 10 + 512× 10 =10 250
Rank 40 approximation : 512× 40 + 40 + 512× 40 =41 000
Rank 80 approximation : 512× 80 + 80 + 512× 80 =82 000
CS221 Linear Algebra Review 10/11/2011 37 / 37
Application : Image Compression
Compression using the Singular Value Decomposition
Truncating the singular value decomposition allows us to represent thematrix with less parameters.
For a 512× 512 matrix :
Full representation : 512× 512 =262 144
Rank 10 approximation : 512× 10 + 10 + 512× 10 =10 250
Rank 40 approximation : 512× 40 + 40 + 512× 40 =41 000
Rank 80 approximation : 512× 80 + 80 + 512× 80 =82 000
CS221 Linear Algebra Review 10/11/2011 37 / 37
Application : Image Compression
Compression using the Singular Value Decomposition
Truncating the singular value decomposition allows us to represent thematrix with less parameters.
For a 512× 512 matrix :
Full representation : 512× 512 =262 144
Rank 10 approximation : 512× 10 + 10 + 512× 10 =10 250
Rank 40 approximation : 512× 40 + 40 + 512× 40 =41 000
Rank 80 approximation : 512× 80 + 80 + 512× 80 =82 000
CS221 Linear Algebra Review 10/11/2011 37 / 37
Application : Image Compression
Compression using the Singular Value Decomposition
Truncating the singular value decomposition allows us to represent thematrix with less parameters.
For a 512× 512 matrix :
Full representation : 512× 512 =262 144
Rank 10 approximation : 512× 10 + 10 + 512× 10 =10 250
Rank 40 approximation : 512× 40 + 40 + 512× 40 =41 000
Rank 80 approximation : 512× 80 + 80 + 512× 80 =82 000
CS221 Linear Algebra Review 10/11/2011 37 / 37