linear algebra iii: vector spaces - princeton...

17
Math Tools for Neuroscience (NEU 314) Fall 2016 Jonathan Pillow Princeton Neuroscience Institute & Psychology. accompanying notes/slides Lecture 4 (Tuesday 9/27) Linear Algebra III: vector spaces

Upload: others

Post on 20-Oct-2019

20 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Linear Algebra III: vector spaces - Princeton Universitypillowlab.princeton.edu/teaching/mathtools16fall/slides/lec04_vectorspaces.pdf · linear combinations some set of vectors •

Math Tools for Neuroscience (NEU 314)Fall 2016

Jonathan Pillow Princeton Neuroscience Institute & Psychology.

accompanying notes/slides

Lecture 4 (Tuesday 9/27)

Linear Algebra III: vector spaces

Page 2: Linear Algebra III: vector spaces - Princeton Universitypillowlab.princeton.edu/teaching/mathtools16fall/slides/lec04_vectorspaces.pdf · linear combinations some set of vectors •

OutlineLast time: • linear combination• linear independence / dependence• matrix operations: transpose, multiplication, inverse

Topics:• matrix equations• vector space, subspace• basis, orthonormal basis• orthogonal matrix• rank• row space / column space• null space• change of basis

Page 3: Linear Algebra III: vector spaces - Princeton Universitypillowlab.princeton.edu/teaching/mathtools16fall/slides/lec04_vectorspaces.pdf · linear combinations some set of vectors •

inverse• If A is a square matrix, its inverse A-1 (if it exists) satisfies:

“the identity”

(eg., for 4 x 4)

Page 4: Linear Algebra III: vector spaces - Princeton Universitypillowlab.princeton.edu/teaching/mathtools16fall/slides/lec04_vectorspaces.pdf · linear combinations some set of vectors •

The identity matrix

“the identity”

(eg., for 4 x 4)

for any vector

Page 5: Linear Algebra III: vector spaces - Princeton Universitypillowlab.princeton.edu/teaching/mathtools16fall/slides/lec04_vectorspaces.pdf · linear combinations some set of vectors •

two weird tricks

• inverse of a product

• transpose of a product

Page 6: Linear Algebra III: vector spaces - Princeton Universitypillowlab.princeton.edu/teaching/mathtools16fall/slides/lec04_vectorspaces.pdf · linear combinations some set of vectors •

(Square) Matrix Equation

assume (for now)square and invertible

left-multiply both sidesby inverse of A:

Page 7: Linear Algebra III: vector spaces - Princeton Universitypillowlab.princeton.edu/teaching/mathtools16fall/slides/lec04_vectorspaces.pdf · linear combinations some set of vectors •

vector space & basis

is clearly a vector space [verify].

Working backwards, a set of vectors is saidto span a vector space if one can write anyvector in the vector space as a linear com-bination of the set. A spanning set can beredundant: For example, if two of the vec-tors are identical, or are scaled copies of eachother. This redundancy is formalized bydefining linear independence. A set of vec-tors {v1, v2, . . . vM} is linearly independent if(and only if) the only solution to the equation

n

αnvn = 0

is αn = 0 (for all n).

1v

2v

3v

A basis for a vector space is a linearly in-dependent spanning set. For example, con-sider the plane of this page. One vector isnot enough to span the plane. Scalar multi-ples of this vector will trace out a line (whichis a subspace), but cannot “get off the line”to cover the rest of the plane. But two vec-tors are sufficient to span the entire plane.Bases are not unique: any two vectors willdo, as long as they don’t lie along the sameline. Three vectors are redundant: one canalways be written as a linear combination ofthe other two. In general, the vector spaceRN requires a basis of size N .

3

e

e

1v

1v

2v 2v

1v

ex

ex xv

eSx )xS v

eSx

S

ooooo

Geometrically, the basis vectors define a setof coordinate axes for the space (althoughthey need not be perpendicular). The stan-dard basis is the set of unit vectors that liealong the axes of the space:

e1 =

100...0

, e2 =

010...0

, . . . eN =

000...1

.

3

2e

1e

v

v

vv

v

ex

ex xv

eSx )xS v

eSx

S

ooooo

5

• vector space - set of all points that can be obtained by linear combinations some set of vectors

• basis - a set of linearly independent vectors that generate (through linear combinations) all points in a vector space

Two different bases for the same 2D vector space(subspace of R2)

1D vector space

Page 8: Linear Algebra III: vector spaces - Princeton Universitypillowlab.princeton.edu/teaching/mathtools16fall/slides/lec04_vectorspaces.pdf · linear combinations some set of vectors •

span - to generate via linear combination

is clearly a vector space [verify].

Working backwards, a set of vectors is saidto span a vector space if one can write anyvector in the vector space as a linear com-bination of the set. A spanning set can beredundant: For example, if two of the vec-tors are identical, or are scaled copies of eachother. This redundancy is formalized bydefining linear independence. A set of vec-tors {v1, v2, . . . vM} is linearly independent if(and only if) the only solution to the equation

n

αnvn = 0

is αn = 0 (for all n).

1v

2v

3v

A basis for a vector space is a linearly in-dependent spanning set. For example, con-sider the plane of this page. One vector isnot enough to span the plane. Scalar multi-ples of this vector will trace out a line (whichis a subspace), but cannot “get off the line”to cover the rest of the plane. But two vec-tors are sufficient to span the entire plane.Bases are not unique: any two vectors willdo, as long as they don’t lie along the sameline. Three vectors are redundant: one canalways be written as a linear combination ofthe other two. In general, the vector spaceRN requires a basis of size N .

3

e

e

1v

1v

2v 2v

1v

ex

ex xv

eSx )xS v

eSx

S

ooooo

Geometrically, the basis vectors define a setof coordinate axes for the space (althoughthey need not be perpendicular). The stan-dard basis is the set of unit vectors that liealong the axes of the space:

e1 =

100...0

, e2 =

010...0

, . . . eN =

000...1

.

3

2e

1e

v

v

vv

v

ex

ex xv

eSx )xS v

eSx

S

ooooo

5

• vector space - set of all points that can be spanned by some set of vectors

• basis - a set of vectors that can span a vector space

Two different bases for the same 2D vector space(subspace of R2)

1D vector space

Page 9: Linear Algebra III: vector spaces - Princeton Universitypillowlab.princeton.edu/teaching/mathtools16fall/slides/lec04_vectorspaces.pdf · linear combinations some set of vectors •

orthonormal basis• basis composed of orthogonal unit vectors

is clearly a vector space [verify].

Working backwards, a set of vectors is saidto span a vector space if one can write anyvector in the vector space as a linear com-bination of the set. A spanning set can beredundant: For example, if two of the vec-tors are identical, or are scaled copies of eachother. This redundancy is formalized bydefining linear independence. A set of vec-tors {v1, v2, . . . vM} is linearly independent if(and only if) the only solution to the equation

n

αnvn = 0

is αn = 0 (for all n).

1v

2v

3v

A basis for a vector space is a linearly in-dependent spanning set. For example, con-sider the plane of this page. One vector isnot enough to span the plane. Scalar multi-ples of this vector will trace out a line (whichis a subspace), but cannot “get off the line”to cover the rest of the plane. But two vec-tors are sufficient to span the entire plane.Bases are not unique: any two vectors willdo, as long as they don’t lie along the sameline. Three vectors are redundant: one canalways be written as a linear combination ofthe other two. In general, the vector spaceRN requires a basis of size N .

3

e

e

1v

1v

2v 2v

1v

ex

ex xv

eSx )xS v

eSx

S

ooooo

Geometrically, the basis vectors define a setof coordinate axes for the space (althoughthey need not be perpendicular). The stan-dard basis is the set of unit vectors that liealong the axes of the space:

e1 =

100...0

, e2 =

010...0

, . . . eN =

000...1

.

3

2e

1e

v

v

vv

v

ex

ex xv

eSx )xS v

eSx

S

ooooo

5

• Two different orthonormal bases for the same vector space

Page 10: Linear Algebra III: vector spaces - Princeton Universitypillowlab.princeton.edu/teaching/mathtools16fall/slides/lec04_vectorspaces.pdf · linear combinations some set of vectors •

Orthogonal matrix• Square matrix whose columns (and rows) form an

orthonormal basis (i.e., are orthogonal unit vectors)

Properties:

length-preserving

Page 11: Linear Algebra III: vector spaces - Princeton Universitypillowlab.princeton.edu/teaching/mathtools16fall/slides/lec04_vectorspaces.pdf · linear combinations some set of vectors •

• 2D example: rotation matrix

nothing. This matrix is called the identity, denoted I .

If an element of the diagonal is zero, thenthe associated axis is annihilated. The setof vectors that are annihilated by the matrixform a vector space [prove], which is calledthe row nullspace, or simply the nullspaceof the matrix.

0002

v2

v1

[ ]

v

2v1

Another implication of a zero diagonal element is that the matrix cannot “reach” the entireoutput space, but only a proper subspace. This space is called the column space of the matrix,since it is spanned by the matrix columns. The rank of a matrix is just the dimensionality ofthe column space. A matrix is said to have full rank if its rank is equal to the smaller of its twodimensions.

An orthogonal matrix is a square matrixwhose columns are pairwise orthogonal unitvectors. Remember that the columns of amatrix describe the response of the system tothe standard basis. Thus an orthogonal ma-trix maps the standard basis onto a new setof N orthogonal axes, which form an alter-native basis for the space. This operation isa generalized rotation, since it corresponds toa physical rotation of the space and possiblynegation of some axes. Thus, the product oftwo orthogonal matrices is also orthogonal.Note that an orthogonal is full rank (it has nonullspace), since a rotation cannot annihilateany non-zero vector.

1e)^( 2eΟ

)^( 1e

..ge

2e= Ο

Ο

Ο =sinθcosθ

cosθsinθ ][

( )

Linear Systems of Equations

The classic motivation for the study of linear algebra is the solution of sets of linear equationssuch as

a11v1 + a12v2 + . . . + a1NvN = b1

a21v1 + a22v2 + . . . + a2NvN = b2

...

aM1v1 + aM2v2 + . . . + aMNvN = bM

8

Orthogonal matrix

Page 12: Linear Algebra III: vector spaces - Princeton Universitypillowlab.princeton.edu/teaching/mathtools16fall/slides/lec04_vectorspaces.pdf · linear combinations some set of vectors •

Rank• the rank of a matrix is equal to

• the rank of a matrix is the dimensionality of the vector space spanned by its rows or its columns

• # of linearly independent columns• # of linearly independent rows(remarkably, these are always the same)

equivalent definition:

for an m x n matrix A: rank(A) ≤ min(m,n)

(can’t be greater than # of rows or # of columns)

Page 13: Linear Algebra III: vector spaces - Princeton Universitypillowlab.princeton.edu/teaching/mathtools16fall/slides/lec04_vectorspaces.pdf · linear combinations some set of vectors •

column space of a matrix W:

n × m matrix

vector space spanned by the columns of W c1 cm…

• these vectors live in an n-dimensional space, so the column space is a subspace of Rn

Page 14: Linear Algebra III: vector spaces - Princeton Universitypillowlab.princeton.edu/teaching/mathtools16fall/slides/lec04_vectorspaces.pdf · linear combinations some set of vectors •

row space of a matrix W:

n × m matrix

vector space spanned by the rows of W

• these vectors live in an m-dimensional space, so the column space is a subspace of Rm

r1

rn

Page 15: Linear Algebra III: vector spaces - Princeton Universitypillowlab.princeton.edu/teaching/mathtools16fall/slides/lec04_vectorspaces.pdf · linear combinations some set of vectors •

null space of a matrix W:

• the vector space consisting of all vectors that are orthogonal to the rows of W

• the null space is therefore entirely orthogonal to the row space of a matrix. Together, they make up all of Rm.

r1

rn

• equivalently: the null space of W is the vector space of all vectors x such that Wx = 0.

n × m matrix

Page 16: Linear Algebra III: vector spaces - Princeton Universitypillowlab.princeton.edu/teaching/mathtools16fall/slides/lec04_vectorspaces.pdf · linear combinations some set of vectors •

null space of a matrix W:

is clearly a vector space [verify].

Working backwards, a set of vectors is saidto span a vector space if one can write anyvector in the vector space as a linear com-bination of the set. A spanning set can beredundant: For example, if two of the vec-tors are identical, or are scaled copies of eachother. This redundancy is formalized bydefining linear independence. A set of vec-tors {v1, v2, . . . vM} is linearly independent if(and only if) the only solution to the equation

n

αnvn = 0

is αn = 0 (for all n).

1v

2v

3v

A basis for a vector space is a linearly in-dependent spanning set. For example, con-sider the plane of this page. One vector isnot enough to span the plane. Scalar multi-ples of this vector will trace out a line (whichis a subspace), but cannot “get off the line”to cover the rest of the plane. But two vec-tors are sufficient to span the entire plane.Bases are not unique: any two vectors willdo, as long as they don’t lie along the sameline. Three vectors are redundant: one canalways be written as a linear combination ofthe other two. In general, the vector spaceRN requires a basis of size N .

3

e

e

1v

1v

2v 2v

1v

ex

ex xv

eSx )xS v

eSx

S

ooooo

Geometrically, the basis vectors define a setof coordinate axes for the space (althoughthey need not be perpendicular). The stan-dard basis is the set of unit vectors that liealong the axes of the space:

e1 =

100...0

, e2 =

010...0

, . . . eN =

000...1

.

3

2e

1e

v

v

vv

v

ex

ex xv

eSx )xS v

eSx

S

ooooo

5

1D ve

ctor s

pace

span

ned b

y v1

W = ( ) v1

null space

basis for null space

Page 17: Linear Algebra III: vector spaces - Princeton Universitypillowlab.princeton.edu/teaching/mathtools16fall/slides/lec04_vectorspaces.pdf · linear combinations some set of vectors •

• Let B denote a matrix whose columns form an orthonormal basis for a vector space W

Vector of projections of v along each basis vector

Change of basis