linear algebra - isical.ac.inacmsc/wmsfmlt2016/linearalgebra.pdf · plan of action part i —...
TRANSCRIPT
LINEAR ALGEBRASourav Sen Gupta
PLAN OF ACTION
Part I — Geometry of Multivariate Data
1. Vector space modelling of multivariate data
2. Understanding Matrices as linear operators
3. Subspaces and Singular Value Decomposition
Part II — Applications in Multivariate Analysis
1. SVD and Principal Component Analysis
2. Eigenvalues and Eigenvectors in PageRank
3. Factor Analysis and SVD in Recommenders
MULTIVARIATE DATA
MULTIVARIATE DATA
p variables/features
n samples/observations
X1 X2 X3 X4S1 5.1 3.5 1.4 0.2S2 4.9 3 1.4 0.2S3 4.7 3.2 1.3 0.2S4 4.6 3.1 1.5 0.2S5 5 3.6 1.4 0.2
S146 6.7 3 5.2 2.3
S147 6.3 2.5 5 1.9
S148 6.5 3 5.2 2
S149 6.2 3.4 5.4 2.3
S150 5.9 3 5.1 1.8
MULTIVARIATE DATA — HISTOGRAMS
MULTIVARIATE DATA — BOXPLOTS
MULTIVARIATE DATA — CORRPLOT
MULTIVARIATE DATA — SCATTERPLOT
MULTIVARIATE DATA — SCATTERPLOT
MULTIVARIATE DATA — SCATTERPLOT
MULTIVARIATE DATA — SCATTERPLOT
MULTIVARIATE DATA — SCATTERPLOTS
MULTIVARIATE DATA — SCATTERPLOTS
MULTIVARIATE DATA — 3D SCATTERPLOT
MULTIVARIATE DATA — 3D SCATTERPLOT
LET’S START WITH 2D
2D GEOMETRY
2D GEOMETRY
(x,y)
x - units
y - units
X
Y
SIZE
X
Y
(x,y)
SIZE
X
Y
(x,y)
size of (x,y) = sqrt(x2 + y2)L2 Norm
SIZE
X
Y
(x,y)
size of (x,y) = abs(x)+ abs(y)L1 Norm
SIZE
X
Y
(x,y)
size of (x,y) = max(abs(x), abs(y))L∞ Norm
UNIT CIRCLE
1
2
∞
ANGLE
X
Y
(x,y)
(u,v)
ANGLE
X
Y
(x,y)
(u,v)
(x,y).(u,v) = |(x,y)| |(u,v)| cos ADot Product
A
ANGLE
X
Y
(x,y)
(u,v)
(x,y).(u,v) = |(x,y)| |(u,v)|Dot Product
ANGLE
X
Y
(x,y)
(u,v)
(x,y).(u,v) = 0Dot Product
SIZE AND ANGLE CHEATSHEET
Notation for dot product : or =
L2 norm of a vector :
Distance between two vectors :
Do you see any connection with Statistics?
v · w vTwnX
i=1
viwi
vT v =nX
i=1
v2i = ||v||22
||v � w||2 =
vuutnX
i=1
(vi � wi)2
ANOTHER LOOK AT 2D
2D GEOMETRY
(x,y)
x - units
y - units
X
Y
2D GEOMETRY
(u,v)
u - units
v - units
U
V
2D GEOMETRY
U
V
X
Y
U = aX + bY
V = cX + dY
2D GEOMETRY
U
V
X
Y
UV
�=
a bc d
� XY
�
TRANSFORMATIONS
TRANSLATIONpq
�
ROTATIONcos ✓ sin ✓� sin ✓ cos ✓
�
SCALING↵ 00 �
�
GENERICa bc d
�
ACTION OF A MATRIX
MATRIX-VECTOR MULTIPLICATION
x =
M v w
LINEAR MAP
Mv w
LINEAR MAP
Mv w
R5 R4
Rn RmMm⇥n
Domain of the function? — Where does the input originate from?
Codomain of the function? — What are possible values of output?
Range of the function? — What are the actual values of output?
Kernel of the function? — Which inputs map to zero in the range?
SUBSPACES
RANGE
+ + + + =
Must be the linear combination of Columns
KERNEL
=
Must be the orthogonal to all Rows
Rn RmMm⇥n
Column Space of the Matrix = Linear span of all Columns = Range
Row Space of the Matrix = Linear span of Rows = Non-zero output
Null Space of the Matrix = Map to zero = Orthogonal to Row Space
Rn RmMm⇥n
ColSpaceRowSpace
Null Space
LINEAR EQUATIONS
SOLVABILITY OF LINEAR EQUATION Mx = b
x b
SOLVABILITY OF LINEAR EQUATION Mx = b
x bSolution always exists.
Solution is unique as Mx = b and My = b implies (x-y) = 0.
SOLVABILITY OF LINEAR EQUATION Mx = b
xb
SOLVABILITY OF LINEAR EQUATION Mx = b
xbu
v
Infinitely many solutions.
If Mu = b, then M(u+v) = b, for any v from Null Space.
SOLVABILITY OF LINEAR EQUATION Mx = b
xb
SOLVABILITY OF LINEAR EQUATION Mx = b
xb
y
e
If Mx = y, then Mx ~ b + e, with irreducible error e.
Solution does not exist!
xb
y
e
Rn RmMm⇥n
T
xb
y
e
Rn RmMm⇥n
T
MTMx = MTb = MTy provides solution x = (MTM)-1MTy
SOLVABILITY OF LINEAR EQUATION Mx = b
Unique Infinite
Infinite None
SINGULAR VALUES
ACTION OF A MATRIX
M
ACTION OF A MATRIX
M
ACTION OF A MATRIX
M
u1v1
M v1 = d1 u1
ACTION OF A MATRIX
M
u1v1
u2
v2
M v1 = d1 u1
M v2 = d2 u2
SINGULAR VECTORS
M v1 u1v2 v3 v4 v5 u2 u3 u4 u5
d1
d2
d3
d4
d5
x x=
SINGULAR VECTORS
M
v1
u1
v2
v3
v4
v5
u2 u3 u4 u5
d1
d2
d3
d4
d5
xx=
SINGULAR VALUE DECOMPOSITION
M
v1
u1
v2
v3
v4
v5
u2 u3 u4
d1
d2 xx=
M = U⌃V T
M
v1
u1
v2
v3
v4
v5
u2 u3 u4
d1
d2 xx=
ColSpaceRowSpace
v1
v2
v3
v4
v5
u1
u2
u3
u4
M = U⌃V T
M
v1
u1
v2
v3
v4
v5
u2 u3 u4
d1
d2 xx=
M
M
v1
u1
v2
v3
v4
v5
u2 u3 u4
d1
d2 xx=
VT
rotation
M
v1
v2
e1
e2
M
v1
u1
v2
v3
v4
v5
u2 u3 u4
d1
d2 xx=
VT ⌃
rotation scaling
M
e1
e2
d1e1
d2e2
M
v1
u1
v2
v3
v4
v5
u2 u3 u4
d1
d2 xx=
VT U⌃
rotation rotationscaling
M
d1e1
d2e2
d1u1
d2u2
M
v1
u1
v2
v3
v4
v5
u2 u3 u4
d1
d2 xx=
VT U⌃
rotation rotationscaling
M
VT U⌃
rotation rotationscaling
M
REFERENCESGilbert Strang — Lectures in Linear Algebra — MIT OCW 18.06 (or YouTube)Gilbert Strang — The Fundamental Theorem of Linear Algebra — AMM 1993Lloyd Trefethen and David Bau — Numerical Linear Algebra — SIAM 1997