uva cs 6316/4501 – fall 2016 machine learning lecture 2 ... · uva cs 6316/4501 – fall 2016...
TRANSCRIPT
![Page 1: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2 ... · UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2: Algebra and Calculus Review Dr. Yanjun Qi University of](https://reader033.vdocuments.site/reader033/viewer/2022060215/5f05d8367e708231d414fed2/html5/thumbnails/1.jpg)
UVACS6316/4501–Fall2016
MachineLearning
Lecture2:AlgebraandCalculusReview
Dr.YanjunQi
UniversityofVirginiaDepartmentof
ComputerScience
9/1/16 Dr.YanjunQi/UVACS6316/f16 1
![Page 2: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2 ... · UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2: Algebra and Calculus Review Dr. Yanjun Qi University of](https://reader033.vdocuments.site/reader033/viewer/2022060215/5f05d8367e708231d414fed2/html5/thumbnails/2.jpg)
9/1/16 2
⎡ ⎤ ⎡ ⎤⎢ ⎥ ⎢ ⎥=⎢ ⎥ ⎢ ⎥⎣ ⎦ ⎣ ⎦
1 2 7 103 4 and = 8 115 6 9 12
A B C=A–B=?
C=A+B=?
⎡ ⎤ ⎡ ⎤⎢ ⎥ ⎢ ⎥=⎢ ⎥ ⎢ ⎥⎣ ⎦ ⎣ ⎦
1 4 7 1 42 5 8 and = 2 53 6 9 3 6
A B C=AB=?
C=BA=?
?
?
?
Dr.YanjunQi/UVACS6316/f16
Minimumrequirement
test
![Page 3: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2 ... · UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2: Algebra and Calculus Review Dr. Yanjun Qi University of](https://reader033.vdocuments.site/reader033/viewer/2022060215/5f05d8367e708231d414fed2/html5/thumbnails/3.jpg)
Today:
q DataRepresentaMonforMLsystems
q ReviewofLinearAlgebraandMatrixCalculus
9/1/16 3Dr.YanjunQi/UVACS6316/f16
![Page 4: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2 ... · UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2: Algebra and Calculus Review Dr. Yanjun Qi University of](https://reader033.vdocuments.site/reader033/viewer/2022060215/5f05d8367e708231d414fed2/html5/thumbnails/4.jpg)
ATypicalMachineLearningPipeline
9/1/16
Low-level sensing
Pre-processing
Feature Extract
Feature Select
Inference, Prediction, Recognition
Label Collection
Dr.YanjunQi/UVACS6316/f16 4
Evaluation
Optimization
e.g. Data Cleaning Task-relevant
![Page 5: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2 ... · UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2: Algebra and Calculus Review Dr. Yanjun Qi University of](https://reader033.vdocuments.site/reader033/viewer/2022060215/5f05d8367e708231d414fed2/html5/thumbnails/5.jpg)
e.g. SUPERVISED LEARNING
• Find function to map input space X to output space Y
• Generalisation:learnfuncNon/hypothesisfrompastdatainorderto“explain”,“predict”,“model”or“control”newdataexamples
9/1/16 5Dr.YanjunQi/UVACS6316/f16KEY
![Page 6: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2 ... · UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2: Algebra and Calculus Review Dr. Yanjun Qi University of](https://reader033.vdocuments.site/reader033/viewer/2022060215/5f05d8367e708231d414fed2/html5/thumbnails/6.jpg)
ADataset
• Data/points/instances/examples/samples/records:[rows]• Features/a0ributes/dimensions/independentvariables/covariates/
predictors/regressors:[columns,exceptthelast]• Target/outcome/response/label/dependentvariable:special
columntobepredicted[lastcolumn]
9/1/16 6Dr.YanjunQi/UVACS6316/f16
![Page 7: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2 ... · UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2: Algebra and Calculus Review Dr. Yanjun Qi University of](https://reader033.vdocuments.site/reader033/viewer/2022060215/5f05d8367e708231d414fed2/html5/thumbnails/7.jpg)
MainTypesofColumns
• Con7nuous:arealnumber,forexample,ageorheight
• Discrete:asymbol,like“Good”or“Bad”
9/1/16 7Dr.YanjunQi/UVACS6316/f16
![Page 8: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2 ... · UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2: Algebra and Calculus Review Dr. Yanjun Qi University of](https://reader033.vdocuments.site/reader033/viewer/2022060215/5f05d8367e708231d414fed2/html5/thumbnails/8.jpg)
e.g. SUPERVISED Classification
• e.g.Here,targetYisadiscretetargetvariable
9/1/16 8f(x?)
Trainingdatasetconsistsofinput-outputpairs
Dr.YanjunQi/UVACS6316/f16
![Page 9: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2 ... · UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2: Algebra and Calculus Review Dr. Yanjun Qi University of](https://reader033.vdocuments.site/reader033/viewer/2022060215/5f05d8367e708231d414fed2/html5/thumbnails/9.jpg)
Today:
q DataRepresentaMonforMLsystems
q ReviewofLinearAlgebraandMatrixCalculus
9/1/16 9Dr.YanjunQi/UVACS6316/f16
![Page 10: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2 ... · UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2: Algebra and Calculus Review Dr. Yanjun Qi University of](https://reader033.vdocuments.site/reader033/viewer/2022060215/5f05d8367e708231d414fed2/html5/thumbnails/10.jpg)
DEFINITIONS - SCALAR
◆ ��a scalar is a number – (denoted with regular type: 1 or 22)
10 9/1/16 Dr.YanjunQi/UVACS6316/f16
![Page 11: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2 ... · UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2: Algebra and Calculus Review Dr. Yanjun Qi University of](https://reader033.vdocuments.site/reader033/viewer/2022060215/5f05d8367e708231d414fed2/html5/thumbnails/11.jpg)
DEFINITIONS - VECTOR
◆ Vector: a single row or column of numbers – denoted with bold small letters – row vector a = – column vector (default) b =
[ ]54321
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
54321
11 9/1/16 Dr.YanjunQi/UVACS6316/f16
![Page 12: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2 ... · UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2: Algebra and Calculus Review Dr. Yanjun Qi University of](https://reader033.vdocuments.site/reader033/viewer/2022060215/5f05d8367e708231d414fed2/html5/thumbnails/12.jpg)
DEFINITIONS - VECTOR
9/1/16 12
• Vector in Rn is an ordered set of n real numbers. – e.g. v = (1,6,3,4) is in R4
– A column vector: – A row vector: ⎟⎟
⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
4361
( )4361
Dr.YanjunQi/UVACS6316/f16
![Page 13: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2 ... · UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2: Algebra and Calculus Review Dr. Yanjun Qi University of](https://reader033.vdocuments.site/reader033/viewer/2022060215/5f05d8367e708231d414fed2/html5/thumbnails/13.jpg)
DEFINITIONS - MATRIX ◆ A matrix is an array of numbers A = ◆ Denoted with a bold Capital letter ◆ All matrices have an order (or dimension): that is, the number of rows * the number of
columns. So, A is 2 by 3 or (2 * 3). u A square matrix is a matrix that has the
same number of rows and columns (n * n)
⎥⎦⎤
⎢⎣⎡
232221
131211
aaaaaa
13 9/1/16 Dr.YanjunQi/UVACS6316/f16
![Page 14: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2 ... · UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2: Algebra and Calculus Review Dr. Yanjun Qi University of](https://reader033.vdocuments.site/reader033/viewer/2022060215/5f05d8367e708231d414fed2/html5/thumbnails/14.jpg)
DEFINITIONS - MATRIX
• m-by-n matrix in Rmxn with m rows and n columns, each entry filled with a (typically) real number:
• e.g. 3*3 matrix
9/1/16 14
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
2396784821
Dr.YanjunQi/UVACS6316/f16
Squarematrix
![Page 15: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2 ... · UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2: Algebra and Calculus Review Dr. Yanjun Qi University of](https://reader033.vdocuments.site/reader033/viewer/2022060215/5f05d8367e708231d414fed2/html5/thumbnails/15.jpg)
Special matrices
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
fedcba
000
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
cb
a
000000
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
jihgf
edcba
000
000
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
fedcb
a000
diagonal upper-triangular
tri-diagonal lower-triangular
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
100010001
I (identity matrix)
9/1/16 15Dr.YanjunQi/UVACS6316/f16
![Page 16: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2 ... · UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2: Algebra and Calculus Review Dr. Yanjun Qi University of](https://reader033.vdocuments.site/reader033/viewer/2022060215/5f05d8367e708231d414fed2/html5/thumbnails/16.jpg)
Special matrices: Symmetric Matrices
e.g.:
9/1/16 16Dr.YanjunQi/UVACS6316/f16
![Page 17: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2 ... · UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2: Algebra and Calculus Review Dr. Yanjun Qi University of](https://reader033.vdocuments.site/reader033/viewer/2022060215/5f05d8367e708231d414fed2/html5/thumbnails/17.jpg)
Review of MATRIX OPERATIONS
1) Transposition 2) Addition and Subtraction 3) Multiplication 4) Norm (of vector) 5) Matrix Inversion 6) Matrix Rank 7) Matrix calculus
17 9/1/16 Dr.YanjunQi/UVACS6316/f16
![Page 18: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2 ... · UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2: Algebra and Calculus Review Dr. Yanjun Qi University of](https://reader033.vdocuments.site/reader033/viewer/2022060215/5f05d8367e708231d414fed2/html5/thumbnails/18.jpg)
(1) Transpose
Transpose: You can think of it as – �flipping� the rows and columns
( )baba T
=⎟⎟⎠
⎞⎜⎜⎝
⎛
⎟⎟⎠
⎞⎜⎜⎝
⎛=⎟⎟⎠
⎞⎜⎜⎝
⎛dbca
dcba T
e.g.
9/1/16 18Dr.YanjunQi/UVACS6316/f16
![Page 19: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2 ... · UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2: Algebra and Calculus Review Dr. Yanjun Qi University of](https://reader033.vdocuments.site/reader033/viewer/2022060215/5f05d8367e708231d414fed2/html5/thumbnails/19.jpg)
(2) Matrix Addition/Subtraction
• Matrix addition/subtraction – Matrices must be of same size.
9/1/16 19Dr.YanjunQi/UVACS6316/f16
![Page 20: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2 ... · UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2: Algebra and Calculus Review Dr. Yanjun Qi University of](https://reader033.vdocuments.site/reader033/viewer/2022060215/5f05d8367e708231d414fed2/html5/thumbnails/20.jpg)
(2) Matrix Addition/SubtractionAnExample
• Ifwehave
⎡ ⎤ ⎡ ⎤ ⎡ ⎤⎢ ⎥ ⎢ ⎥ ⎢ ⎥=⎢ ⎥ ⎢ ⎥ ⎢ ⎥⎣ ⎦ ⎣ ⎦ ⎣ ⎦
1 2 7 10 8 12+ = 3 4 + 8 11 = 11 15
5 6 9 12 14 18C A B
⎡ ⎤ ⎡ ⎤⎢ ⎥ ⎢ ⎥=⎢ ⎥ ⎢ ⎥⎣ ⎦ ⎣ ⎦
1 2 7 103 4 and = 8 115 6 9 12
A B
thenwecancalculateC=A+Bby
9/1/16 20Dr.YanjunQi/UVACS6316/f16
![Page 21: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2 ... · UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2: Algebra and Calculus Review Dr. Yanjun Qi University of](https://reader033.vdocuments.site/reader033/viewer/2022060215/5f05d8367e708231d414fed2/html5/thumbnails/21.jpg)
(2) Matrix Addition/SubtractionAnExample
• Similarly,ifwehave
⎡ ⎤ ⎡ ⎤ ⎡ ⎤⎢ ⎥ ⎢ ⎥ ⎢ ⎥=⎢ ⎥ ⎢ ⎥ ⎢ ⎥⎣ ⎦ ⎣ ⎦ ⎣ ⎦
1 2 7 10 -6 -8- = 3 4 - 8 11 = -5 -7
5 6 9 12 -4 -6C A B
⎡ ⎤ ⎡ ⎤⎢ ⎥ ⎢ ⎥=⎢ ⎥ ⎢ ⎥⎣ ⎦ ⎣ ⎦
1 2 7 103 4 and = 8 115 6 9 12
A B
thenwecancalculateC=A-Bby
9/1/16 21Dr.YanjunQi/UVACS6316/f16
![Page 22: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2 ... · UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2: Algebra and Calculus Review Dr. Yanjun Qi University of](https://reader033.vdocuments.site/reader033/viewer/2022060215/5f05d8367e708231d414fed2/html5/thumbnails/22.jpg)
OPERATION on MATRIX
1) Transposition 2) Addition and Subtraction 3) Multiplication 4) Norm (of vector) 5) Matrix Inversion 6) Matrix Rank 7) Matrix calculus
22 9/1/16 Dr.YanjunQi/UVACS6316/f16
![Page 23: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2 ... · UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2: Algebra and Calculus Review Dr. Yanjun Qi University of](https://reader033.vdocuments.site/reader033/viewer/2022060215/5f05d8367e708231d414fed2/html5/thumbnails/23.jpg)
(3)ProductsofMatrices• WewritethemulNplicaNonoftwomatricesAandBasAB
• Thisisreferredtoeitheras• pre-mulNplyingBbyA or
• post-mulNplyingAbyB• SoformatrixmulNplicaNonAB,Aisreferredtoasthepremul7plierandBisreferredtoasthepostmul7plier
9/1/16 23Dr.YanjunQi/UVACS6316/f16
![Page 24: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2 ... · UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2: Algebra and Calculus Review Dr. Yanjun Qi University of](https://reader033.vdocuments.site/reader033/viewer/2022060215/5f05d8367e708231d414fed2/html5/thumbnails/24.jpg)
(3)ProductsofMatrices
9/1/16 24
CondiMon:n=q
m x n q x p m x p
Dr.YanjunQi/UVACS6316/f16
n
![Page 25: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2 ... · UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2: Algebra and Calculus Review Dr. Yanjun Qi University of](https://reader033.vdocuments.site/reader033/viewer/2022060215/5f05d8367e708231d414fed2/html5/thumbnails/25.jpg)
(3)ProductsofMatrices
• InordertomulNplymatrices,theymustbeconformable(thenumberofcolumnsinthepremulNpliermustequalthenumberofrowsinpostmulNplier)
• Notethat• an(mxn)x(nxp)=(mxp)• an(mxn)x(pxn)=cannotbedone• a(1xn)x(nx1)=ascalar(1x1)
9/1/16 25Dr.YanjunQi/UVACS6316/f16
![Page 26: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2 ... · UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2: Algebra and Calculus Review Dr. Yanjun Qi University of](https://reader033.vdocuments.site/reader033/viewer/2022060215/5f05d8367e708231d414fed2/html5/thumbnails/26.jpg)
ProductsofMatrices
• IfwehaveA(3x3)andB(3x2)then
⎡ ⎤ ⎡ ⎤ ⎡ ⎤⎢ ⎥ ⎢ ⎥ ⎢ ⎥= =⎢ ⎥ ⎢ ⎥ ⎢ ⎥⎣ ⎦ ⎣ ⎦⎣ ⎦
11 12 13 11 12 11 12
21 22 23 21 22 21 22
31 32 31 3231 32 33
a a a b b c ca a a x b b = c c
b b c ca a aAB C
11 11 11 12 21 13 31
12 11 12 12 22 13 32
21 21 11 22 21 23 31
22 21 12 22 22 23 32
31 31 11 32 21 33 31
32 31 12 32 22 33 32
c = a b + a b +a bc = a b +a b +a bc = a b +a b +a bc = a b +a b +a bc = a b +a b +a bc = a b +a b +a b
where test
9/1/16 26Dr.YanjunQi/UVACS6316/f16
![Page 27: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2 ... · UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2: Algebra and Calculus Review Dr. Yanjun Qi University of](https://reader033.vdocuments.site/reader033/viewer/2022060215/5f05d8367e708231d414fed2/html5/thumbnails/27.jpg)
MatrixMulNplicaNonAnExample
• Ifwehave
⎡ ⎤⎡ ⎤ ⎡ ⎤ ⎡ ⎤⎢ ⎥⎢ ⎥ ⎢ ⎥ ⎢ ⎥=⎢ ⎥⎢ ⎥ ⎢ ⎥ ⎢ ⎥⎣ ⎦ ⎣ ⎦ ⎣ ⎦⎣ ⎦
11 12
21 22
31 32
c c1 4 7 1 4 30 662 5 8 x 2 5 = c c = 36 813 6 9 3 6 42 96c c
AB
( ) ( ) ( )( ) ( ) ( )( ) ( ) ( )( ) ( ) ( )( ) ( ) ( )
11 11 11 12 21 13 31
12 11 12 12 22 13 32
21 21 11 22 21 23 31
22 21 12 22 22 23 32
31 31 11 32 21 33 31
32 31 12 32 22 3
c = a b + a b + a b =1 1 + 4 2 +7 3 = 30
c = a b + a b + a b =1 4 + 4 5 +7 6 = 66
c = a b + a b + a b = 2 1 +5 2 +8 3 = 36
c = a b + a b + a b = 2 4 +5 5 +8 6 = 81
c = a b + a b + a b = 3 1 +6 2 +9 3 = 42
c = a b + a b + a ( ) ( ) ( )3 32b = 3 4 +6 5 +9 6 = 96
⎡ ⎤ ⎡ ⎤⎢ ⎥ ⎢ ⎥=⎢ ⎥ ⎢ ⎥⎣ ⎦ ⎣ ⎦
1 4 7 1 42 5 8 and = 2 53 6 9 3 6
A B
then
where
9/1/16 27Dr.YanjunQi/UVACS6316/f16
![Page 28: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2 ... · UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2: Algebra and Calculus Review Dr. Yanjun Qi University of](https://reader033.vdocuments.site/reader033/viewer/2022060215/5f05d8367e708231d414fed2/html5/thumbnails/28.jpg)
SomeProperNesofMatrixMulNplicaNon
• Notethat• Evenifconformable,ABdoesnotnecessarilyequalBA(i.e.,matrixmulNplicaNonisnotcommuta7ve)
• MatrixmulNplicaNoncanbeextendedbeyondtwomatrices
• matrixmulNplicaNonisassocia7ve,i.e.,A(BC)=(AB)C
9/1/16 28Dr.YanjunQi/UVACS6316/f16
![Page 29: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2 ... · UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2: Algebra and Calculus Review Dr. Yanjun Qi University of](https://reader033.vdocuments.site/reader033/viewer/2022060215/5f05d8367e708231d414fed2/html5/thumbnails/29.jpg)
SomeProperNesofMatrixMulNplicaNon
◆ Multiplication and transposition (AB)T = BTAT
u Multiplication with Identity Matrix
29 9/1/16 Dr.YanjunQi/UVACS6316/f16
![Page 30: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2 ... · UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2: Algebra and Calculus Review Dr. Yanjun Qi University of](https://reader033.vdocuments.site/reader033/viewer/2022060215/5f05d8367e708231d414fed2/html5/thumbnails/30.jpg)
SpecialUsesforMatrixMulNplicaNon
• ProductsofScalars&MatricesèExample,Ifwehave
⎡ ⎤ ⎡ ⎤⎢ ⎥ ⎢ ⎥=⎢ ⎥ ⎢ ⎥⎣ ⎦ ⎣ ⎦
1 2 3.5 7.03.5 3 4 = 10.5 14.0
5 6 17.5 21.0bA
⎡ ⎤⎢ ⎥=⎢ ⎥⎣ ⎦
1 23 4 and b = 3.55 6
A
thenwecancalculatebAby
NotethatbA=Abifbisascalar9/1/16 30Dr.YanjunQi/UVACS6316/f16
![Page 31: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2 ... · UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2: Algebra and Calculus Review Dr. Yanjun Qi University of](https://reader033.vdocuments.site/reader033/viewer/2022060215/5f05d8367e708231d414fed2/html5/thumbnails/31.jpg)
SpecialUsesforMatrixMulNplicaNon
• Dot(orInner)ProductoftwoVectors• PremulNplicaNonofacolumnvectorabyconformablerowvectorbyieldsasinglevaluecalledthedotproductorinnerproduct-If
aT = 3 4 6!"
#$ and b =
528
!
"
%%
#
$
&&
aTb = a•b = 3 4 6!"
#$
528
!
"
%%
#
$
&& = 3 5( )+ 4 2( )+ 6 8( )= 71 = bTa
thentheirinnerproductgivesus
whichisthesumofproductsofelementsinsimilarposiNonsforthetwovectors9/1/16 31Dr.YanjunQi/UVACS6316/f16
![Page 32: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2 ... · UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2: Algebra and Calculus Review Dr. Yanjun Qi University of](https://reader033.vdocuments.site/reader033/viewer/2022060215/5f05d8367e708231d414fed2/html5/thumbnails/32.jpg)
SpecialUsesforMatrixMulNplicaNon
• OuterProductoftwoVectors• PostmulNplicaNonofacolumnvectorabyconformablerowvectorbyieldsamatrixcontainingtheproductsofeachpairofelementsfromthetwomatrices(calledtheouterproduct)-If
aT = 3 4 6!"
#$ and b =
528
!
"
%%
#
$
&&
abT =346
!
"
##
$
%
&&
5 2 8!"
$% =
152030
6812
243248
!
"
##
$
%
&&
thenabTgivesus
9/1/16 32Dr.YanjunQi/UVACS6316/f16
![Page 33: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2 ... · UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2: Algebra and Calculus Review Dr. Yanjun Qi University of](https://reader033.vdocuments.site/reader033/viewer/2022060215/5f05d8367e708231d414fed2/html5/thumbnails/33.jpg)
SpecialUsesforMatrixMulNplicaNon
• OuterProductoftwoVectors,e.g.aspecialcase:
9/1/16 Dr.YanjunQi/UVACS6316/f16 33
![Page 34: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2 ... · UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2: Algebra and Calculus Review Dr. Yanjun Qi University of](https://reader033.vdocuments.site/reader033/viewer/2022060215/5f05d8367e708231d414fed2/html5/thumbnails/34.jpg)
SpecialUsesforMatrixMulNplicaNon
• SumtheSquaredElementsofaVector• PremulNplyacolumnvectorabyitstranspose–If
thenpremulNplicaNonbyarowvectoraT
willyieldthesumofthesquaredvaluesofelementsfora,i.e.
⎡ ⎤⎢ ⎥=⎢ ⎥⎣ ⎦
52 8
a
aT = 5 2 8!"
#$
aTa = 5 2 8!"
#$
528
!
"
%%
#
$
&& = 52 +22 + 82 = 93
9/1/16 34Dr.YanjunQi/UVACS6316/f16
![Page 35: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2 ... · UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2: Algebra and Calculus Review Dr. Yanjun Qi University of](https://reader033.vdocuments.site/reader033/viewer/2022060215/5f05d8367e708231d414fed2/html5/thumbnails/35.jpg)
SpecialUsesforMatrixMulNplicaNon
• Matrix-VectorProducts(I)
9/1/16 Dr.YanjunQi/UVACS6316/f16 35
![Page 36: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2 ... · UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2: Algebra and Calculus Review Dr. Yanjun Qi University of](https://reader033.vdocuments.site/reader033/viewer/2022060215/5f05d8367e708231d414fed2/html5/thumbnails/36.jpg)
SpecialUsesforMatrixMulNplicaNon
• Matrix-VectorProducts(II)
9/1/16 Dr.YanjunQi/UVACS6316/f16 36
![Page 37: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2 ... · UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2: Algebra and Calculus Review Dr. Yanjun Qi University of](https://reader033.vdocuments.site/reader033/viewer/2022060215/5f05d8367e708231d414fed2/html5/thumbnails/37.jpg)
SpecialUsesforMatrixMulNplicaNon
• Matrix-VectorProducts(III)
9/1/16 Dr.YanjunQi/UVACS6316/f16 37
![Page 38: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2 ... · UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2: Algebra and Calculus Review Dr. Yanjun Qi University of](https://reader033.vdocuments.site/reader033/viewer/2022060215/5f05d8367e708231d414fed2/html5/thumbnails/38.jpg)
SpecialUsesforMatrixMulNplicaNon
• Matrix-VectorProducts(IV)
9/1/16 Dr.YanjunQi/UVACS6316/f16 38
![Page 39: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2 ... · UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2: Algebra and Calculus Review Dr. Yanjun Qi University of](https://reader033.vdocuments.site/reader033/viewer/2022060215/5f05d8367e708231d414fed2/html5/thumbnails/39.jpg)
MATRIX OPERATIONS
1) Transposition 2) Addition and Subtraction 3) Multiplication 4) Norm (of vector) 5) Matrix Inversion 6) Matrix Rank 7) Matrix calculus
39 9/1/16 Dr.YanjunQi/UVACS6316/f16
![Page 40: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2 ... · UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2: Algebra and Calculus Review Dr. Yanjun Qi University of](https://reader033.vdocuments.site/reader033/viewer/2022060215/5f05d8367e708231d414fed2/html5/thumbnails/40.jpg)
A norm of a vector ||x|| is informally a measure of the �length� of the vector.
– Common norms: L1, L2 (Euclidean)
– Linfinity
(4) Vector norms
9/1/16 40Dr.YanjunQi/UVACS6316/f16
![Page 41: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2 ... · UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2: Algebra and Calculus Review Dr. Yanjun Qi University of](https://reader033.vdocuments.site/reader033/viewer/2022060215/5f05d8367e708231d414fed2/html5/thumbnails/41.jpg)
VectorNorm(L2,whenp=2)
9/1/16 41Dr.YanjunQi/UVACS6316/f16
![Page 42: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2 ... · UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2: Algebra and Calculus Review Dr. Yanjun Qi University of](https://reader033.vdocuments.site/reader033/viewer/2022060215/5f05d8367e708231d414fed2/html5/thumbnails/42.jpg)
VectorNorms(e.g.,)
![Page 43: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2 ... · UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2: Algebra and Calculus Review Dr. Yanjun Qi University of](https://reader033.vdocuments.site/reader033/viewer/2022060215/5f05d8367e708231d414fed2/html5/thumbnails/43.jpg)
MoreGeneral:Norm
• AnormisanyfuncNong()thatmapsvectorstorealnumbersthatsaNsfiesthefollowingcondiNons:
9/1/16 Dr.YanjunQi/UVACS6316/f16 43
![Page 44: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2 ... · UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2: Algebra and Calculus Review Dr. Yanjun Qi University of](https://reader033.vdocuments.site/reader033/viewer/2022060215/5f05d8367e708231d414fed2/html5/thumbnails/44.jpg)
Orthogonal&Orthonormal
Ifu•v=0,||u||2!=0,||v||2!=0àuandvareorthogonal
Ifu•v=0,||u||2=1,||v||2=1àuandvareorthonormal
x•y=
InnerProductdefinedbetweencolumnvectorxandy,asè
9/1/16 44Dr.YanjunQi/UVACS6316/f16
![Page 45: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2 ... · UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2: Algebra and Calculus Review Dr. Yanjun Qi University of](https://reader033.vdocuments.site/reader033/viewer/2022060215/5f05d8367e708231d414fed2/html5/thumbnails/45.jpg)
Orthogonal matrices
• Aisorthogonalif:
• NotaNon:
Example:
9/1/16 45Dr.YanjunQi/UVACS6316/f16
![Page 46: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2 ... · UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2: Algebra and Calculus Review Dr. Yanjun Qi University of](https://reader033.vdocuments.site/reader033/viewer/2022060215/5f05d8367e708231d414fed2/html5/thumbnails/46.jpg)
Orthogonal matrices
• NotethatifAisorthogonal,iteasytofinditsinverse:
Property:
9/1/16 46Dr.YanjunQi/UVACS6316/f16
![Page 47: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2 ... · UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2: Algebra and Calculus Review Dr. Yanjun Qi University of](https://reader033.vdocuments.site/reader033/viewer/2022060215/5f05d8367e708231d414fed2/html5/thumbnails/47.jpg)
• DefiniMon:Givenavectornorm||x||,thematrixnormdefinedbythevectornormisgivenby:
• Whatdoesamatrixnormrepresent?• Itrepresentsthemaximum“stretching”thatAdoestoavectorx->(Ax).
MatrixNorm
xAx
Ax 0max
≠=
![Page 48: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2 ... · UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2: Algebra and Calculus Review Dr. Yanjun Qi University of](https://reader033.vdocuments.site/reader033/viewer/2022060215/5f05d8367e708231d414fed2/html5/thumbnails/48.jpg)
TheoremA:Thematrixnormcorrespondingto1-normismaximumabsolutecolumnsum:
Proof:Frompreviousslide,wecanhaveAlso,whereAjisthej-thcolumnofA.
Matrix1-Norm
∑=
=n
iijjaA
11max
111max AxAx =
=
�
Ax = x1A1 + x2A2 +!+ xnAn = x jA jj=1
n
∑
![Page 49: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2 ... · UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2: Algebra and Calculus Review Dr. Yanjun Qi University of](https://reader033.vdocuments.site/reader033/viewer/2022060215/5f05d8367e708231d414fed2/html5/thumbnails/49.jpg)
MATRIX OPERATIONS
1) Transposition 2) Addition and Subtraction 3) Multiplication 4) Norm (of vector) 5) Matrix Inversion 6) Matrix Rank 7) Matrix calculus
49 9/1/16 Dr.YanjunQi/UVACS6316/f16
![Page 50: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2 ... · UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2: Algebra and Calculus Review Dr. Yanjun Qi University of](https://reader033.vdocuments.site/reader033/viewer/2022060215/5f05d8367e708231d414fed2/html5/thumbnails/50.jpg)
(5)InverseofaMatrix
• TheinverseofamatrixAiscommonlydenotedbyA-1orinvA.
• TheinverseofannxnmatrixAisthematrixA-1suchthatAA-1=I=A-1A
• Thematrixinverseisanalogoustoascalarreciprocal
• Amatrixwhichhasaninverseiscallednonsingular
9/1/16 50Dr.YanjunQi/UVACS6316/f16
![Page 51: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2 ... · UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2: Algebra and Calculus Review Dr. Yanjun Qi University of](https://reader033.vdocuments.site/reader033/viewer/2022060215/5f05d8367e708231d414fed2/html5/thumbnails/51.jpg)
(5)InverseofaMatrix
• ForsomenxnmatrixA,aninversematrixA-1
maynotexist.• Amatrixwhichdoesnothaveaninverseissingular.
• AninverseofnxnmatrixAexistsiff|A|not0
9/1/16 51Dr.YanjunQi/UVACS6316/f16
![Page 52: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2 ... · UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2: Algebra and Calculus Review Dr. Yanjun Qi University of](https://reader033.vdocuments.site/reader033/viewer/2022060215/5f05d8367e708231d414fed2/html5/thumbnails/52.jpg)
THE DETERMINANT OF A MATRIX
◆ The determinant of a matrix A is denoted by |A| (or det(A) or det A).
◆ Determinants exist only for square matrices.
◆ E.g. If A = ⎥⎦
⎤⎢⎣
⎡
2221
1211
aaaa
21122211 aaaaA −=
52 9/1/16 Dr.YanjunQi/UVACS6316/f16
![Page 53: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2 ... · UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2: Algebra and Calculus Review Dr. Yanjun Qi University of](https://reader033.vdocuments.site/reader033/viewer/2022060215/5f05d8367e708231d414fed2/html5/thumbnails/53.jpg)
THE DETERMINANT OF A MATRIX
2 x 2
3 x 3
n x n
9/1/16 53Dr.YanjunQi/UVACS6316/f16
![Page 54: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2 ... · UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2: Algebra and Calculus Review Dr. Yanjun Qi University of](https://reader033.vdocuments.site/reader033/viewer/2022060215/5f05d8367e708231d414fed2/html5/thumbnails/54.jpg)
THE DETERMINANT OF A MATRIX
diagonal matrix:
9/1/16 54Dr.YanjunQi/UVACS6316/f16
![Page 55: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2 ... · UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2: Algebra and Calculus Review Dr. Yanjun Qi University of](https://reader033.vdocuments.site/reader033/viewer/2022060215/5f05d8367e708231d414fed2/html5/thumbnails/55.jpg)
HOW TO FIND INVERSE MATRIXES? An example,
◆ If ◆ and |A| not 0
⎥⎦
⎤⎢⎣
⎡−
−=
acbd-
)det(11
AA
⎥⎦
⎤⎢⎣
⎡=
dcba
A
55 9/1/16 Dr.YanjunQi/UVACS6316/f16
![Page 56: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2 ... · UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2: Algebra and Calculus Review Dr. Yanjun Qi University of](https://reader033.vdocuments.site/reader033/viewer/2022060215/5f05d8367e708231d414fed2/html5/thumbnails/56.jpg)
Matrix Inverse
• The inverse A-1 of a matrix A has the property: AA-1=A-1A=I
• A-1 exists only if
• Terminology – Singular matrix: A-1 does not exist – Ill-conditioned matrix: A is close to being singular
9/1/16 56Dr.YanjunQi/UVACS6316/f16
![Page 57: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2 ... · UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2: Algebra and Calculus Review Dr. Yanjun Qi University of](https://reader033.vdocuments.site/reader033/viewer/2022060215/5f05d8367e708231d414fed2/html5/thumbnails/57.jpg)
PROPERTIES OF INVERSE MATRICES
◆ ��
◆ ��
◆ ��
( ) 111 -- ABAB =−
€
AT( )−1 = A-1( )T
( ) AA =−11-
57 9/1/16 Dr.YanjunQi/UVACS6316/f16
![Page 58: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2 ... · UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2: Algebra and Calculus Review Dr. Yanjun Qi University of](https://reader033.vdocuments.site/reader033/viewer/2022060215/5f05d8367e708231d414fed2/html5/thumbnails/58.jpg)
Inverse of special matrix
• For diagonal matrices
• For orthogonal matrices – a square matrix with real entries whose columns and rows
are orthogonal unit vectors (i.e., orthonormal vectors)
9/1/16 58Dr.YanjunQi/UVACS6316/f16
![Page 59: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2 ... · UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2: Algebra and Calculus Review Dr. Yanjun Qi University of](https://reader033.vdocuments.site/reader033/viewer/2022060215/5f05d8367e708231d414fed2/html5/thumbnails/59.jpg)
Pseudo-inverse
• The pseudo-inverse A+ of a matrix A (could be non-square, e.g., m x n) is given by:
• It can be shown that:
9/1/16 59Dr.YanjunQi/UVACS6316/f16
![Page 60: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2 ... · UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2: Algebra and Calculus Review Dr. Yanjun Qi University of](https://reader033.vdocuments.site/reader033/viewer/2022060215/5f05d8367e708231d414fed2/html5/thumbnails/60.jpg)
MATRIX OPERATIONS
1) Transposition 2) Addition and Subtraction 3) Multiplication 4) Norm (of vector) 5) Matrix Inversion 6) Matrix Rank 7) Matrix calculus
60 9/1/16 Dr.YanjunQi/UVACS6316/f16
![Page 61: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2 ... · UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2: Algebra and Calculus Review Dr. Yanjun Qi University of](https://reader033.vdocuments.site/reader033/viewer/2022060215/5f05d8367e708231d414fed2/html5/thumbnails/61.jpg)
(6) Rank: Linear independence
• A set of vectors is linearly independent if none of them can be written as a linear combination of the others.
x3=−2x1+x2
èNOTlinearlyindependent
9/1/16 61Dr.YanjunQi/UVACS6316/f16
![Page 62: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2 ... · UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2: Algebra and Calculus Review Dr. Yanjun Qi University of](https://reader033.vdocuments.site/reader033/viewer/2022060215/5f05d8367e708231d414fed2/html5/thumbnails/62.jpg)
(6) Rank: Linear independence
• Alternative definition: Vectors v1,…,vk are linearly independent if c1v1+…+ckvk = 0 implies c1=…=ck=0
9/1/16 62
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛=
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
000
|||
|||
3
2
1
321
ccc
vvv
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛=⎟⎟⎠
⎞⎜⎜⎝
⎛
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
000
313201
vu
(u,v)=(0,0),i.e.thecolumnsarelinearlyindependent.
e.g.
Dr.YanjunQi/UVACS6316/f16
![Page 63: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2 ... · UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2: Algebra and Calculus Review Dr. Yanjun Qi University of](https://reader033.vdocuments.site/reader033/viewer/2022060215/5f05d8367e708231d414fed2/html5/thumbnails/63.jpg)
(6) Rank of a Matrix
• rank(A) (the rank of a m-by-n matrix A) is = The maximal number of linearly independent columns =The maximal number of linearly independent rows
• If A is n by m, then – rank(A)<= min(m,n) – If n=rank(A), then A has full row rank – If m=rank(A), then A has full column rank
Rank=? Rank=?
9/1/16 63Dr.YanjunQi/UVACS6316/f16
![Page 64: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2 ... · UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2: Algebra and Calculus Review Dr. Yanjun Qi University of](https://reader033.vdocuments.site/reader033/viewer/2022060215/5f05d8367e708231d414fed2/html5/thumbnails/64.jpg)
(6) Rank of a Matrix
• Equal to the dimension of the largest square sub-matrix of A that has a non-zero determinant.
Example: has rank 3
9/1/16 64Dr.YanjunQi/UVACS6316/f16
![Page 65: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2 ... · UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2: Algebra and Calculus Review Dr. Yanjun Qi University of](https://reader033.vdocuments.site/reader033/viewer/2022060215/5f05d8367e708231d414fed2/html5/thumbnails/65.jpg)
(6) Rank and singular matrices
9/1/16 65Dr.YanjunQi/UVACS6316/f16
![Page 66: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2 ... · UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2: Algebra and Calculus Review Dr. Yanjun Qi University of](https://reader033.vdocuments.site/reader033/viewer/2022060215/5f05d8367e708231d414fed2/html5/thumbnails/66.jpg)
MATRIX OPERATIONS
1) Transposition 2) Addition and Subtraction 3) Multiplication 4) Norm (of vector) 5) Matrix Inversion 6) Matrix Rank 7) Matrix calculus
66 9/1/16 Dr.YanjunQi/UVACS6316/f16
![Page 67: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2 ... · UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2: Algebra and Calculus Review Dr. Yanjun Qi University of](https://reader033.vdocuments.site/reader033/viewer/2022060215/5f05d8367e708231d414fed2/html5/thumbnails/67.jpg)
( ) ( )0
limh
f a h f ah→
+ −iscalledthederivaNveofat.f a
Wewrite: ( ) ( ) ( )0
limh
f x h f xf x
h→
+ −′ =
�ThederivaNveoffwithrespecttoxis…�
TherearemanywaystowritethederivaMveof ( )y f x=
Review:DerivaNveofaFuncNon
èe.g.definetheslopeofthecurvey=f(x)atthepointx9/1/16 67Dr.YanjunQi/UVACS6316/f16
![Page 68: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2 ... · UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2: Algebra and Calculus Review Dr. Yanjun Qi University of](https://reader033.vdocuments.site/reader033/viewer/2022060215/5f05d8367e708231d414fed2/html5/thumbnails/68.jpg)
-3
-2
-10
1
2
3
4
5
6
-3 -2 -1 1 2 3x
2 3y x= −
( ) ( )2 2
0
3 3limh
x h xy
h→
+ − − −′ =
2 2 2
0
2limh
x xh h xy
h→
+ + −′ =
2y x′ =-6-5-4-3-2-10
123456
-3 -2 -1 1 2 3x
0lim2h
y x h→
′ = +0
→
Review:DerivaNveofaQuadraNcFuncNon
9/1/16 68Dr.YanjunQi/UVACS6316/f16
![Page 69: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2 ... · UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2: Algebra and Calculus Review Dr. Yanjun Qi University of](https://reader033.vdocuments.site/reader033/viewer/2022060215/5f05d8367e708231d414fed2/html5/thumbnails/69.jpg)
SomeimportantrulesfortakingderivaNves
9/1/16 Dr.YanjunQi/UVACS6316/f16 69
![Page 70: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2 ... · UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2: Algebra and Calculus Review Dr. Yanjun Qi University of](https://reader033.vdocuments.site/reader033/viewer/2022060215/5f05d8367e708231d414fed2/html5/thumbnails/70.jpg)
Review:DefiniNonsofgradient(Matrix_calculus/Scalar-by-matrix)
9/1/16 70Dr.YanjunQi/UVACS6316/f16
Inprinciple,gradientsareanaturalextensionofparNalderivaNvestofuncNonsofmulNplevariables.
èDenominatorlayout
![Page 71: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2 ... · UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2: Algebra and Calculus Review Dr. Yanjun Qi University of](https://reader033.vdocuments.site/reader033/viewer/2022060215/5f05d8367e708231d414fed2/html5/thumbnails/71.jpg)
Review:DefiniNonsofgradient(Matrix_calculus/Scalar-by-vector)
9/1/16 71
• Sizeofgradientisalwaysthesameasthesizeof
if
Dr.YanjunQi/UVACS6316/f16
èDenominatorlayout
![Page 72: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2 ... · UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2: Algebra and Calculus Review Dr. Yanjun Qi University of](https://reader033.vdocuments.site/reader033/viewer/2022060215/5f05d8367e708231d414fed2/html5/thumbnails/72.jpg)
For Examples
9/1/16 Dr.YanjunQi/UVACS6316/f16 72
![Page 73: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2 ... · UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2: Algebra and Calculus Review Dr. Yanjun Qi University of](https://reader033.vdocuments.site/reader033/viewer/2022060215/5f05d8367e708231d414fed2/html5/thumbnails/73.jpg)
Exercise:asimpleexample
9/1/16 Dr.YanjunQi/UVACS6316/f16 73
!!
f (w)=wTx = w1 ,w2 ,w3!"
#$
123
%
&
'''
(
)
***=w1+2w2+3w3
!!
∂ f∂w
=∂wTx∂w
= x =123
"
#
$$$
%
&
'''
èDenominatorlayout
![Page 74: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2 ... · UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2: Algebra and Calculus Review Dr. Yanjun Qi University of](https://reader033.vdocuments.site/reader033/viewer/2022060215/5f05d8367e708231d414fed2/html5/thumbnails/74.jpg)
EvenmoregeneralMatrixCalculus:TypesofMatrixDerivaMves
ByThomasMinka.OldandNewMatrixAlgebraUsefulforStaNsNcs
9/1/16 74Dr.YanjunQi/UVACS6316/f16
![Page 75: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2 ... · UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2: Algebra and Calculus Review Dr. Yanjun Qi University of](https://reader033.vdocuments.site/reader033/viewer/2022060215/5f05d8367e708231d414fed2/html5/thumbnails/75.jpg)
Review:HessianMatrix/n==2case
• 1stderivaNvetogradient,
• 2ndderivaNvetoHessian
f (x, y)
g =∇f =∂f∂x
∂f∂y
#
$
%%
&
'
((
H =∂2 f∂x2
∂2 f∂x∂y
∂2 f∂x∂y
∂2 f∂y2
#
$
%%%
&
'
(((
759/1/16 Dr.YanjunQi/UVACS6316/f16
SinglevariateàmulNvariate
![Page 76: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2 ... · UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2: Algebra and Calculus Review Dr. Yanjun Qi University of](https://reader033.vdocuments.site/reader033/viewer/2022060215/5f05d8367e708231d414fed2/html5/thumbnails/76.jpg)
Review:HessianMatrix
9/1/16 Dr.YanjunQi/UVACS6316/f16 76
![Page 77: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2 ... · UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2: Algebra and Calculus Review Dr. Yanjun Qi University of](https://reader033.vdocuments.site/reader033/viewer/2022060215/5f05d8367e708231d414fed2/html5/thumbnails/77.jpg)
TodayRecap
q DataRepresentaMonq LinearAlgebraandMatrixCalculusReview1) Transposition 2) Addition and Subtraction 3) Multiplication 4) Norm (of vector) 5) Matrix Inversion 6) Matrix Rank 7) Matrix calculus
9/1/16 77Dr.YanjunQi/UVACS6316/f16
![Page 78: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2 ... · UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2: Algebra and Calculus Review Dr. Yanjun Qi University of](https://reader033.vdocuments.site/reader033/viewer/2022060215/5f05d8367e708231d414fed2/html5/thumbnails/78.jpg)
Extra:
• HW1isreleasedtoday@Collab• HW1isduenextSat@midnight
• HandoutforLecture2hasbeenposted@hxp://www.cs.virginia.edu/yanjun/teach/2016f/schedule.html
9/1/16 Dr.YanjunQi/UVACS6316/f16 78
![Page 79: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2 ... · UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2: Algebra and Calculus Review Dr. Yanjun Qi University of](https://reader033.vdocuments.site/reader033/viewer/2022060215/5f05d8367e708231d414fed2/html5/thumbnails/79.jpg)
Extra
• Thefollowingtopicsarecoveredbyhandout,butnotbythisslide(willbecovered…)– Trace()– Eigenvalue/Eigenvectors– PosiNvedefinitematrix,Grammatrix– QuadraNcform– ProjecNon(vectoronaplane,oronavector)
9/1/16 Dr.YanjunQi/UVACS6316/f16 79
![Page 80: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2 ... · UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 2: Algebra and Calculus Review Dr. Yanjun Qi University of](https://reader033.vdocuments.site/reader033/viewer/2022060215/5f05d8367e708231d414fed2/html5/thumbnails/80.jpg)
References
q hxp://www.cs.cmu.edu/~zkolter/course/linalg/index.html
q Prof.JamesJ.Cochran’stutorialslides“MatrixAlgebraPrimerII”
q hxp://www.cs.cmu.edu/~aarN/Class/10701/recitaNon/LinearAlgebra_Matlab_Review.ppt
q Prof.AlexanderGray’sslidesq Prof. George Bebis’ slides q Prof. Hal Daum ́e III’ notes
9/1/16 80Dr.YanjunQi/UVACS6316/f16