linear algebra primer - artificial...

Linear Algebra ReviewStanford University

LinearAlgebraPrimer

JuanCarlosNiebles andRanjayKrishnaStanfordVisionandLearningLab

10/2/171

Another,veryin-depthlinearalgebrareviewfromCS229isavailablehere:http://cs229.stanford.edu/section/cs229-linalg.pdfAndavideodiscussionoflinearalgebrafromEE263ishere(lectures3and4):https://see.stanford.edu/Course/EE263


Outline• Vectorsandmatrices– BasicMatrixOperations– Determinants,norms,trace– SpecialMatrices

• TransformationMatrices– Homogeneouscoordinates– Translation

• Matrixinverse• Matrixrank• EigenvaluesandEigenvectors• MatrixCalculus

10/2/172





10/2/173

Vectorsandmatricesarejustcollectionsoforderednumbersthatrepresentsomething:movementsinspace,scalingfactors,pixelbrightness,etc.We’lldefinesomecommonusesandstandardoperationsonthem.


Vector

• Acolumnvectorwhere

• Arowvectorwhere

denotesthetransposeoperation

10/2/174


Vector• We’lldefaulttocolumnvectorsinthisclass

• You’llwanttokeeptrackoftheorientationofyourvectorswhenprogramminginpython

• YoucantransposeavectorVinpythonbywritingV.t.(Butinclassmaterials,wewillalways useVT toindicatetranspose,andwewilluseV’tomean“Vprime”)

10/2/175


Vectorshavetwomainuses

• Vectorscanrepresentanoffsetin2Dor3Dspace

• Pointsarejustvectorsfromtheorigin

10/2/176

• Data(pixels,gradientsatanimagekeypoint,etc)canalsobetreatedasavector

• Suchvectorsdon’thaveageometricinterpretation,butcalculationslike“distance”canstillhavevalue


Matrix

• Amatrixisanarrayofnumberswithsizeby,i.e.mrowsandncolumns.

• If,wesaythatissquare.

10/2/177


Images

10/2/178

• Pythonrepresentsanimageasamatrixofpixelbrightnesses

• Notethattheupperleftcorneris[y,x]=(0,0)

=


ColorImages• Grayscaleimageshaveonenumberperpixel,andarestoredasanm× nmatrix.

• Colorimageshave3numbersperpixel– red,green,andbluebrightnesses (RGB)

• Storedasanm× n× 3matrix

10/2/179

=


BasicMatrixOperations• Wewilldiscuss:– Addition– Scaling– Dotproduct–Multiplication– Transpose– Inverse/pseudoinverse– Determinant/trace

10/2/1710


MatrixOperations• Addition

– Canonlyaddamatrixwithmatchingdimensions,orascalar.

• Scaling

10/2/1711


• Norm• Moreformally,anormisanyfunctionthatsatisfies4properties:

• Non-negativity: Forall• Definiteness: f(x)=0ifandonlyifx=0.• Homogeneity: Forall• Triangleinequality: Forall

10/2/1712

Vectors


• ExampleNorms

• Generalnorms:

10/2/1713

MatrixOperations


MatrixOperations• Innerproduct(dotproduct)ofvectors–Multiplycorrespondingentriesoftwovectorsandadduptheresult

– x·y isalso|x||y|Cos(theanglebetweenxandy)

10/2/1714


MatrixOperations

• Innerproduct(dotproduct)ofvectors– IfBisaunitvector,thenA·BgivesthelengthofAwhichliesinthedirectionofB

10/2/1715


MatrixOperations• Theproductoftwomatrices

10/2/1716


MatrixOperations• Multiplication

• TheproductABis:

• Eachentryintheresultis(thatrowofA)dotproductwith(thatcolumnofB)

• Manyuses,whichwillbecoveredlater

10/2/1717


MatrixOperations• Multiplicationexample:

10/2/1718

– Eachentryofthematrixproductismadebytakingthedotproductofthecorrespondingrowintheleftmatrix,withthecorrespondingcolumnintherightone.


MatrixOperations• Theproductoftwomatrices

10/2/1719


MatrixOperations

• Powers– Byconvention,wecanrefertothematrixproductAAasA2,andAAAasA3,etc.

– Obviouslyonlysquarematricescanbemultipliedthatway

10/2/1720


MatrixOperations• Transpose– flipmatrix,sorow1becomescolumn1

• Ausefulidentity:

10/2/1721


• Determinant– returnsascalar– Representsarea(orvolume)oftheparallelogramdescribedbythevectorsintherowsofthematrix

– For,– Properties:

10/2/1722

MatrixOperations


• Trace

– Invarianttoalotoftransformations,soit’susedsometimesinproofs.(Rarelyinthisclassthough.)

– Properties:

10/2/1723

MatrixOperations


• VectorNorms

• Matrixnorms:Normscanalsobedefinedformatrices,suchastheFrobenius norm:

10/2/1724

MatrixOperations


SpecialMatrices• IdentitymatrixI– Squarematrix,1’salongdiagonal,0’selsewhere

– I · [anothermatrix]=[thatmatrix]

• Diagonalmatrix– Squarematrixwithnumbersalongdiagonal,0’selsewhere

– Adiagonal· [anothermatrix]scalestherowsofthatmatrix

10/2/1725


SpecialMatrices

• Symmetricmatrix

• Skew-symmetricmatrix

10/2/1726

2

40 �2 �52 0 �75 7 0

3

5


LinearAlgebraPrimer

JuanCarlosNiebles andRanjayKrishnaStanfordVisionandLearningLab

10/2/1727

Another,veryin-depthlinearalgebrareviewfromCS229isavailablehere:http://cs229.stanford.edu/section/cs229-linalg.pdfAndavideodiscussionoflinearalgebrafromEE263ishere(lectures3and4):https://see.stanford.edu/Course/EE263


Announcements– part1

• HW0submittedlastnight• HW1isduenextMonday• HW2willbereleasedtonight• ClassnotesfromlastThursdayduebeforeclassinexactly48hours

10/2/1728


Announcements– part2

• Futurehomeworkassignmentswillbereleasedviagithub–WillallowyoutokeeptrackofchangesIFtheyhappen.

• SubmissionsforHW1onwardswillbedoneallthroughgradescope.– NOMORECORNSUBMISSIONS– Youwillhaveseparatesubmissionsfortheipythonpdfandthepythoncode.

10/2/1729


Recap- Vector

• Acolumnvectorwhere

• Arowvectorwhere

denotesthetransposeoperation

10/2/1730


Recap- Matrix

• Amatrixisanarrayofnumberswithsizeby,i.e.mrowsandncolumns.

• If,wesaythatissquare.

10/2/1731


Recap- ColorImages• Grayscaleimageshaveonenumberperpixel,andarestoredasanm× nmatrix.

• Colorimageshave3numbersperpixel– red,green,andbluebrightnesses (RGB)

• Storedasanm× n× 3matrix

10/2/1732

=


• Norm• Moreformally,anormisanyfunctionthatsatisfies4properties:

• Non-negativity: Forall• Definiteness: f(x)=0ifandonlyifx=0.• Homogeneity: Forall• Triangleinequality: Forall

10/2/1733

Recap- Vectors


Recap– projection

• Innerproduct(dotproduct)ofvectors– IfBisaunitvector,thenA·BgivesthelengthofAwhichliesinthedirectionofB

10/2/1734





10/2/1735

Matrixmultiplicationcanbeusedtotransformvectors.Amatrixusedinthiswayiscalledatransformationmatrix.


Transformation

• Matricescanbeusedtotransformvectorsinusefulways,throughmultiplication:x’=Ax

• Simplestisscaling:

(Verifytoyourselfthatthematrixmultiplicationworksoutthisway)

10/2/1736


Rotation• Howcanyouconvertavectorrepresentedinframe“0”toanew,rotatedcoordinateframe“1”?

10/2/1737


Rotation• Howcanyouconvertavectorrepresentedinframe“0”toanew,rotatedcoordinateframe“1”?

• Rememberwhatavectoris:[componentindirectionoftheframe’sxaxis,componentindirectionofyaxis]

10/2/1738


Rotation• Sotorotateitwemustproducethisvector:[componentindirectionofnew xaxis,componentindirectionofnew yaxis]

• Wecandothiseasilywithdotproducts!• Newxcoordinateis[originalvector]dot [thenewxaxis]• Newycoordinateis[originalvector]dot [thenewyaxis]

10/2/1739


Rotation• Insight:thisiswhathappensinamatrix*vectormultiplication– Resultxcoordinateis:[originalvector]dot [matrixrow1]

– Somatrixmultiplicationcanrotateavectorp:

10/2/1740


Rotation• Supposeweexpressapointinthenewcoordinatesystemwhichisrotatedleft

• Ifweplottheresultintheoriginal coordinatesystem,wehaverotatedthepointright

10/2/1741

– Thus,rotationmatricescanbeusedtorotatevectors.We’llusuallythinkoftheminthatsense-- asoperatorstorotatevectors


2DRotationMatrixFormulaCounter-clockwise rotation by an angle q

úû

ùêë

éúû

ùêë

é -=ú

û

ùêë

éyx

yx

qqqq

cossinsincos

''

P

x

y’P’

q

x’

y

PRP'=

yθsinxθcos'x -=xθsinyθcos'y +=

10/2/1742


TransformationMatrices• Multipletransformationmatricescanbeusedtotransformapoint:p’=R2R1Sp

10/2/1743



• Theeffectofthisistoapplytheirtransformationsoneaftertheother,fromrighttoleft.

• Intheexampleabove,theresultis(R2(R1(Sp)))

10/2/1744



• Theeffectofthisistoapplytheirtransformationsoneaftertheother,fromrighttoleft.

• Intheexampleabove,theresultis(R2(R1(Sp)))

• Theresultisexactlythesameifwemultiplythematricesfirst,toformasingletransformationmatrix:p’=(R2R1S)p

10/2/1745


Homogeneoussystem

• Ingeneral,amatrixmultiplicationletsuslinearlycombinecomponentsofavector

– Thisissufficientforscale,rotate,skewtransformations.

– Butnotice,wecan’taddaconstant!L

10/2/1746


Homogeneoussystem

– The(somewhathacky)solution?Sticka“1”attheendofeveryvector:

– Nowwecanrotate,scale,andskewlikebefore,ANDtranslate (notehowthemultiplicationworksout,above)

– Thisiscalled“homogeneouscoordinates”

10/2/1747


Homogeneoussystem– Inhomogeneouscoordinates,themultiplicationworksoutsotherightmostcolumnofthematrixisavectorthatgetsadded.

– Generally,ahomogeneoustransformationmatrixwillhaveabottomrowof[001],sothattheresulthasa“1”atthebottomtoo.

10/2/1748


Homogeneoussystem• Onemorethingwemightwant:todividetheresultbysomething– Forexample,wemaywanttodividebyacoordinate,tomakethingsscaledownastheygetfartherawayinacameraimage

– Matrixmultiplicationcan’tactuallydivide– So,byconvention,inhomogeneouscoordinates,we’lldividetheresultbyitslastcoordinateafterdoingamatrixmultiplication

10/2/1749


2DTranslation

t

P

P’

10/2/1750

Linear Algebra ReviewStanford University 10/2/1751

2DTranslationusingHomogeneousCoordinates

P

x

y

tx

ty

P’t

úúú

û

ù

êêê

ë

é×úúú

û

ù

êêê

ë

é=

úúú

û

ù

êêê

ë

é++

®1100

1001

1' y

xtt

tytx

y

x

y

x

P

)1,,(),()1,,(),(

yxyx ttttyxyx

®=®=

tP

P



P

x

y

tx

ty

P’t

úúú

û

ù

êêê

ë

é×úúú

û

ù

êêê

ë

é=

úúú

û

ù

êêê

ë

é++

®1100

1001

1' y

xtt

tytx

y

x

y

x

P

)1,,(),()1,,(),(

yxyx ttttyxyx

®=®=

tP

P



P

x

y

tx

ty

P’t

úúú

û

ù

êêê

ë

é×úúú

û

ù

êêê

ë

é=

úúú

û

ù

êêê

ë

é++

®1100

1001

1' y

xtt

tytx

y

x

y

x

P

)1,,(),()1,,(),(

yxyx ttttyxyx

®=®=

tP

tP

PTPtI

×=×úû

ùêë

é=

10


Scaling

P

P’

10/2/1756


ScalingEquation

P

x

y

sx x

P’sy y

)1,,(),(')1,,(),(

ysxsysxsyxyx

yxyx ®=®=

PP

)ys,xs(')y,x( yx=®= PP

10/2/1757


ScalingEquation

P

x

y

sx x

P’sy y

úúú

û

ù

êêê

ë

é×úúú

û

ù

êêê

ë

é=

úúú

û

ù

êêê

ë

é®

11000000

1' y

xs

sysxs

y

x

y

x

P

)1,,(),(')1,,(),(

ysxsysxsyxyx

yxyx ®=®=

PP


10/2/1758


ScalingEquation

P

x

y

sx x

P’sy y

úúú

û

ù

êêê

ë

é×úúú

û

ù

êêê

ë

é=

úúú

û

ù

êêê

ë

é®

11000000

1' y

xs

sysxs

y

x

y

x

P

)1,,(),(')1,,(),(

ysxsysxsyxyx

yxyx ®=®=

PP

S

PSP100S

×=×úû

ùêë

é=

'


10/2/1759


P

P’=S∙PP’’=T∙P’

P’’=T·P’=T·(S·P)=T·S·P

Scaling&Translating

P’’

10/2/1760


Scaling&Translating

P '' = T ⋅S ⋅P =1 0 tx0 1 ty0 0 1

"

#

$$$$

%

&

''''

⋅

sx 0 00 sy 0

0 0 1

"

#

$$$$

%

&

''''

xy1

"

#

$$$

%

&

'''=

=

sx 0 tx0 sy ty0 0 1

"

#

$$$$

%

&

''''

xy1

"

#

$$$

%

&

'''=

sxx + txsyy+ ty1

"

#

$$$$

%

&

''''

= S t0 1

"

#$

%

&'

xy1

"

#

$$$

%

&

'''

A10/2/1761


Scaling&Translating

P '' = T ⋅S ⋅P =1 0 tx0 1 ty0 0 1

"

#

$$$$

%

&

''''

⋅

sx 0 00 sy 0

0 0 1

"

#

$$$$

%

&

''''

xy1

"

#

$$$

%

&

'''=

=

sx 0 tx0 sy ty0 0 1

"

#

$$$$

%

&

''''

xy1

"

#

$$$

%

&

'''=

sxx + txsyy+ ty1

"

#

$$$$

%

&

''''

= S t0 1

"

#$

%

&'

xy1

"

#

$$$

%

&

'''

10/2/1762


Translating&Scalingversus Scaling&Translating

úúú

û

ù

êêê

ë

é++

=úúú

û

ù

êêê

ë

é

úúú

û

ù

êêê

ë

é=

úúú

û

ù

êêê

ë

é

úúú

û

ù

êêê

ë

é

úúú

û

ù

êêê

ë

é=××=

1tystxs

1yx

100ts0t0s

1yx

1000s000s

100t10t01

''' yy

xx

yy

xx

y

x

y

x

PSTP

10/2/1763


Translating&Scaling!=Scaling&Translating

úúú

û

ù

êêê

ë

é++

=úúú

û

ù

êêê

ë

é

úúú

û

ù

êêê

ë

é=

=úúú

û

ù

êêê

ë

é

úúú

û

ù

êêê

ë

é

úúú

û

ù

êêê

ë

é=××=

1tsystsxs

1yx

100tss0ts0s

1yx

100t10t01

1000s000s

'''

yyy

xxx

yyy

xxx

y

x

y

x

PTSP

úúú

û

ù

êêê

ë

é++

=úúú

û

ù

êêê

ë

é

úúú

û

ù

êêê

ë

é=

úúú

û

ù

êêê

ë

é

úúú

û

ù

êêê

ë

é

úúú

û

ù

êêê

ë

é=××=

1tystxs

1yx

100ts0t0s

1yx

1000s000s

100t10t01

''' yy

xx

yy

xx

y

x

y

x

PSTP

10/2/1764


Translating&Scaling!=Scaling&Translating

úúú

û

ù

êêê

ë

é++

=úúú

û

ù

êêê

ë

é

úúú

û

ù

êêê

ë

é=

=úúú

û

ù

êêê

ë

é

úúú

û

ù

êêê

ë

é

úúú

û

ù

êêê

ë

é=××=

1tsystsxs

1yx

100tss0ts0s

1yx

100t10t01

1000s000s

'''

yyy

xxx

yyy

xxx

y

x

y

x

PTSP

úúú

û

ù

êêê

ë

é++

=úúú

û

ù

êêê

ë

é

úúú

û

ù

êêê

ë

é=

úúú

û

ù

êêê

ë

é

úúú

û

ù

êêê

ë

é

úúú

û

ù

êêê

ë

é=××=

1tystxs

1yx

100ts0t0s

1yx

1000s000s

100t10t01

''' yy

xx

yy

xx

y

x

y

x

PSTP

10/2/1765


Rotation

P

P’

10/2/1766


RotationEquationsCounter-clockwise rotation by an angle q

úû

ùêë

éúû

ùêë

é -=ú

û

ùêë

éyx

yx

qqqq

cossinsincos

''

P

x

y’P’

q

x’

y

PRP'=

yθsinxθcos'x -=xθsinyθcos'y +=

10/2/1767


RotationMatrixProperties

A2Drotationmatrixis2x2

1)det( ==×=×

RIRRRR TT

úû

ùêë

éúû

ùêë

é -=ú

û

ùêë

éyx

yx

qqqq

cossinsincos

''

Note: R belongs to the category of normal matrices and satisfies many interesting properties:

10/2/1768


RotationMatrixProperties• Transposeofarotationmatrixproducesarotationintheoppositedirection

• Therowsofarotationmatrixarealwaysmutuallyperpendicular(a.k.a.orthogonal)unitvectors– (andsoareitscolumns)

10/2/1769

1)det( ==×=×

RIRRRR TT


Scaling+Rotation+TranslationP’=(TRS)P

=úúú

û

ù

êêê

ë

é

úúú

û

ù

êêê

ë

é

úúú

û

ù

êêê

ë

é -

úúú

û

ù

êêê

ë

é=×××=

1yx

1000s000s

1000θcossinθ0inθsθcos

100t10t01

R' y

x

y

x

PSTP

=úúú

û

ù

êêê

ë

é

úúú

û

ù

êêê

ë

é

úúú

û

ù

êêê

ë

é -=

1yx

1000s000s

100tθcossinθtinθsθcos

y

x

y

x

úúú

û

ù

êêê

ë

é

úû

ùêë

é=

úúú

û

ù

êêê

ë

é

úû

ùêë

éúû

ùêë

é=

110

1100

10yx

tSRyx

StR

10/2/1770

Thisistheformofthegeneral-purposetransformationmatrix




• Matrixinverse• Matrixrank• EigenvaluesandEigenvectors• MatrixCalculate

10/2/1771

Theinverseofatransformationmatrixreversesitseffect


• GivenamatrixA,itsinverseA-1 isamatrixsuchthatAA-1=A-1A=I

• E.g.

• Inversedoesnotalwaysexist.IfA-1 exists,A isinvertible ornon-singular.Otherwise,it’ssingular.

• Usefulidentities,formatricesthatareinvertible:

10/2/1772

Inverse


• Pseudoinverse– SayyouhavethematrixequationAX=B,whereAandBareknown,andyouwanttosolveforX

10/2/1773

MatrixOperations



– Youcouldcalculatetheinverseandpre-multiplybyit:A-1AX=A-1B→X=A-1B

10/2/1774

MatrixOperations



– Youcouldcalculatetheinverseandpre-multiplybyit:A-1AX=A-1B→X=A-1B

– Pythoncommandwouldbenp.linalg.inv(A)*B– Butcalculatingtheinverseforlargematricesoftenbringsproblemswithcomputerfloating-pointresolution(becauseitinvolvesworkingwithverysmallandverylargenumberstogether).

– Or,yourmatrixmightnotevenhaveaninverse.

10/2/1775

MatrixOperations


• Pseudoinverse– Fortunately,thereareworkaroundstosolveAX=Binthesesituations.Andpythoncandothem!

– Insteadoftakinganinverse,directlyaskpythontosolveforXinAX=B,bytypingnp.linalg.solve(A,B)

– Pythonwilltryseveralappropriatenumericalmethods(includingthepseudoinverseiftheinversedoesn’texist)

– PythonwillreturnthevalueofXwhichsolvestheequation• Ifthereisnoexactsolution,itwillreturntheclosestone• Iftherearemanysolutions,itwillreturnthesmallestone

10/2/1776

MatrixOperations


• Pythonexample:

10/2/1777

MatrixOperations

>> import numpy as np>> x = np.linalg.solve(A,B)x =

1.0000-0.5000





10/2/1778

Therankofatransformationmatrixtellsyouhowmanydimensionsittransformsavectorto.


Linearindependence• Supposewehaveasetofvectorsv1, …, vn• Ifwecanexpressv1 asalinearcombinationoftheothervectorsv2…vn,thenv1 islinearlydependent ontheothervectors.– Thedirectionv1 canbeexpressedasacombinationofthedirectionsv2…vn. (E.g.v1 =.7 v2-.7 v4)

10/2/1779


Linearindependence• Supposewehaveasetofvectorsv1, …, vn• Ifwecanexpressv1 asalinearcombinationoftheothervectorsv2…vn,thenv1 islinearlydependent ontheothervectors.– Thedirectionv1 canbeexpressedasacombinationofthedirectionsv2…vn.(E.g.v1 =.7 v2-.7 v4)

• Ifnovectorislinearlydependentontherestoftheset,thesetislinearlyindependent.– Commoncase:asetofvectorsv1, …, vn isalwayslinearlyindependentifeachvectorisperpendiculartoeveryothervector(andnon-zero)

10/2/1780


LinearindependenceNotlinearlyindependent

10/2/1781

Linearlyindependentset


Matrixrank

• Column/rowrank

– Columnrankalwaysequalsrowrank

• Matrixrank

10/2/1782


Matrixrank• Fortransformationmatrices,theranktellsyouthedimensionsoftheoutput

• E.g.ifrankofA is1,thenthetransformation

p’=Apmapspointsontoaline.

• Here’samatrixwithrank1:

10/2/1783

Allpointsgetmappedtotheliney=2x


Matrixrank• Ifanm xmmatrixisrankm,wesayit’s“fullrank”– Mapsanm x1vectoruniquelytoanotherm x1vector– Aninversematrixcanbefound

• Ifrank<m,wesayit’s“singular”– Atleastonedimensionisgettingcollapsed.Nowaytolookattheresultandtellwhattheinputwas

– Inversedoesnotexist

• Inversealsodoesn’texistfornon-squarematrices

10/2/1784




• Matrixinverse• Matrixrank• EigenvaluesandEigenvectors(SVD)• MatrixCalculus

10/2/1785


EigenvectorandEigenvalue

• Aneigenvector x ofalineartransformation A isanon-zerovectorthat,when A isappliedtoit,doesnotchangedirection.

10/2/1786



• Aneigenvector x ofalineartransformation A isanon-zerovectorthat,when A isappliedtoit,doesnotchangedirection.

• Applying A totheeigenvectoronlyscalestheeigenvectorbythescalarvalue λ,calledaneigenvalue.

10/2/1787



• WewanttofindalltheeigenvaluesofA:

• Whichcanwewrittenas:

• Therefore:

10/2/1788



• Wecansolveforeigenvaluesbysolving:

• Sincewearelookingfornon-zerox,wecaninsteadsolvetheaboveequationas:

10/2/1789


Properties

• ThetraceofaAisequaltothesumofitseigenvalues:

10/2/1790


Properties• ThetraceofaAisequaltothesumofitseigenvalues:

• ThedeterminantofAisequaltotheproductofitseigenvalues

10/2/1791




• TherankofAisequaltothenumberofnon-zeroeigenvaluesofA.

10/2/1792




• TherankofAisequaltothenumberofnon-zeroeigenvaluesofA.

• TheeigenvaluesofadiagonalmatrixD=diag(d1,...dn)arejustthediagonalentriesd1,...dn

10/2/1793


Spectraltheory

• Wecallaneigenvalueλ andanassociatedeigenvectoraneigenpair.

• Thespaceofvectorswhere(A−λI)=0isoftencalledtheeigenspace ofAassociatedwiththeeigenvalueλ.

• ThesetofalleigenvaluesofAiscalleditsspectrum:

10/2/1794


Spectraltheory

• Themagnitudeofthelargesteigenvalue(inmagnitude)iscalledthespectralradius

–WhereCisthespaceofalleigenvaluesofA

10/2/1795


Spectraltheory

• Thespectralradiusisboundedbyinfinitynormofamatrix:

• Proof:Turntoapartnerandprovethis!

10/2/1796


Spectraltheory

• Thespectralradiusisboundedbyinfinitynormofamatrix:

• Proof:Letλ andvbeaneigenpair ofA:

10/2/1797


Diagonalization

• Ann× nmatrixAisdiagonalizableifithasnlinearlyindependenteigenvectors.

• Mostsquarematrices(inasensethatcanbemademathematicallyrigorous)arediagonalizable:– Normalmatricesarediagonalizable– Matriceswithndistincteigenvaluesarediagonalizable

Lemma:Eigenvectorsassociatedwithdistincteigenvaluesarelinearlyindependent.

10/2/1798


Diagonalization

• Ann× nmatrixAisdiagonalizableifithasnlinearlyindependenteigenvectors.

• Mostsquarematricesarediagonalizable:– Normalmatricesarediagonalizable–Matriceswithndistincteigenvaluesarediagonalizable

Lemma:Eigenvectorsassociatedwithdistincteigenvaluesarelinearlyindependent.

10/2/1799


Diagonalization

• Eigenvalueequation:

–WhereDisadiagonalmatrixoftheeigenvalues

10/2/17100


Diagonalization

• Eigenvalueequation:

• Assumingallλi’s areunique:

• Rememberthattheinverseofanorthogonalmatrixisjustitstransposeandtheeigenvectorsareorthogonal

10/2/17101


Symmetricmatrices

• Properties:– ForasymmetricmatrixA,alltheeigenvaluesarereal.

– TheeigenvectorsofAareorthonormal.

10/2/17102


Symmetricmatrices

• Therefore:

– where• So,ifwewantedtofindthevectorxthat:

10/2/17103


Symmetricmatrices

• Therefore:

– where• So,ifwewantedtofindthevectorxthat:

– Isthesameasfindingtheeigenvectorthatcorrespondstothelargesteigenvalue.

10/2/17104


SomeapplicationsofEigenvalues

• PageRank• Schrodinger’sequation• PCA

10/2/17105




• Matrixinverse• Matrixrank• EigenvaluesandEigenvectors(SVD)• MatrixCalculus

10/2/17106


MatrixCalculus– TheGradient

• LetafunctiontakeasinputamatrixAofsizem× nandreturnsarealvalue.

• Thenthegradient off:

10/2/17107


MatrixCalculus – TheGradient

• Everyentryinthematrixis:• thesizeof∇Af(A)isalwaysthesameasthesizeofA.SoifAisjustavectorx:

10/2/17108


Exercise

• Example:

• Find:

10/2/17109


Exercise

• Example:

• Fromthiswecanconcludethat:

10/2/17110


MatrixCalculus – TheGradient

• Properties

10/2/17111


MatrixCalculus – TheHessian

• TheHessianmatrixwithrespecttox,writtenorsimplyasHisthen× nmatrixof

partialderivatives

10/2/17112



• Eachentrycanbewrittenas:

• Exercise:WhyistheHessianalwayssymmetric?

10/2/17113



• Eachentrycanbewrittenas:

• TheHessianisalwayssymmetric,because

• ThisisknownasSchwarz'stheorem:Theorderofpartialderivativesdon’tmatteraslongasthesecondderivativeexistsandiscontinuous.

10/2/17114


MatrixCalculus– TheHessian

• Notethatthehessianisnotthegradientofwholegradientofavector(thisisnotdefined).Itisactuallythegradientofeveryentry ofthegradientofthevector.

10/2/17115


MatrixCalculus– TheHessian

• Eg,thefirstcolumnisthegradientof

10/2/17116


Exercise

• Example:

10/2/17117


Exercise

10/2/17118


Exercise

10/2/17119

Dividethesummationinto3partsdependingonwhether:• i ==kor• j==k


Exercise

10/2/17120


Exercise

10/2/17121


Exercise

10/2/17122


Exercise

10/2/17123


Exercise

10/2/17124


Exercise

10/2/17125


Exercise

10/2/17126


Exercise

10/2/17127


Exercise

10/2/17128


Whatwehavelearned• Vectorsandmatrices– BasicMatrixOperations– SpecialMatrices



10/2/17129

linear algebra primer - artificial...

Documents