d. van alphen1 ece 455 – lecture 12 orthogonal matrices singular value decomposition (svd) svd for...
TRANSCRIPT
D. van Alphen 1
ECE 455 – Lecture 12
• Orthogonal Matrices
• Singular Value Decomposition (SVD)
• SVD for Image Compression
Recall: If vectors a and b have the same orientation (both row vectors or both column vectors, then a’b = b’a = dot(a,b)and
dot(a, a) = length2(a)
D. van Alphen 2
Orthogonal Matrices
• Definition: A square matrix is orthogonal () if its columns are orthonormal (i.e., orthogonal and of unit-length).
• Example: Q =
• Let Q be an matrix with columns q1, q2, …, qn.
• Then Q’ Q =
• Note (1) qi’ qi = |qi|2 = 1 (since vectors are unit-length);
and (2) qi’ qj = 0 (since vectors are orthogonal)
2
1
2
12
1
2
1
n
n
n
I
qqq
q
q
q
21
'
'2
'1
Diagonal elements are 1 by note (1) below; off-diagonal elements are 0, by note (2) below)
D. van Alphen 3
Orthogonal Matrices, continued
• So far: Q Q = In
• Similarly, Q Q = In
• Thus, Q-1 = Q
• More properties of orthogonal matrices:
– If Q is orthogonal, so is Q.– The rows of any orthogonal matrix are also orthogonal.
• Example 1: Q =
Q-1 = Q =
cossin
sincos
cossin
sincos
Note: This particular Q is called a rotation matrix.
Multiplying a (2, 1) vector x by Q will rotate x by angle .
D. van Alphen 4
Orthogonal Matrices, continuedRotation Matrix Example
• Let x = and let = 90 in the rotation matrix.
• Then
2
1
1
2
2
1
01
10
2
1
)90cos()90sin(
)90sin()90cos(
xQ
xQ x 90 Note: the length of
the vector was not changed.
D. van Alphen 5
Claim: Orthogonal Matrices Preserve Length
• Multiplying x by an orthogonal matrix Q does not change the length:
| Q x | = | x |
• Proof: | Q x | 2 = (Q x) (Q x) = x Q Q x = x x = | x | 2
• Example: Multiply [x y]’ by “plane rotation” matrix Q:
z = Q
| z | 2 = (x cos - y sin )2 + (x sin + y cos )2
= (x2 cos2 + y2 sin2 – 2xy sin cos
(x2 sin2 + y2 cos2 + 2xy sin cos
= (x2 + y2)(sin2 + cos2 x2 + y2 = |[ x y]’ |2
I, since Q-1 = Q
cosysinx
sinycosx
cossin
sincos
y
x
D. van Alphen 6
Orthogonal Matrices, ContinuedPermutation Matrix Example
• Recall permutation matrix Pij was used in Gaussian Elimination to exchange (or swap) 2 rows.
• Example: to find the 3-by-3 permutation matrix that swaps rows 1 and 2 by pre-multiplying a matrix, we started with identity matrix I3 and swapped rows 1 and 2:
I3 =
• Note that the columns of P12 are unit-length, and orthogonal to each other; hence P12 is an orthogonal matrix.
• Claim: All permutation matrices are orthogonal.
12P
100
001
010
100
010
001
R1 R2
D. van Alphen 7
Singular Value Decomposition (SVD)
• A way to factor matrices: A = U V
– where A is any matrix of size (m, n);
– U and V are orthogonal matrices; • U contains the “left singular vectors”• V contains the “right singular vectors”; and
– is a diagonal matrix, containing the “singular values” on the diagonal
• Closely related to Principal Component Analysis
• Mostly used for data compression – particularly for image compression
D. van Alphen 8
SVD Format
• Suppose that A is a matrix of size (m, n) and rank r. Then the SVD
• A = U V = [u1 … ur … um]
'
'
'11
0
0
n
rr
v
v
v
(m, m) (m, n) (n, n)
D. van Alphen 9
SVD Factorization: Finding and V
• Start with: A = U V (1)
• Note that U U = I, since U is orthogonal. Hence left-multiply both sides of equation (1) by A :
A A = (U V) U V
= V U U V = V V
• Note that is a diagonal matrix, with the elements i2 on the diagonal
• The eigenvalues of (A A) are the i2’s, and the eigenvectors are the
columns of matrix V.
I
D. van Alphen 10
SVD Example
• Let A =
• A’ A =
• MATLAB Code to find eigenvalues and eigenvectors of A’A:
>> [v d] = eig([5 -3; -3 5]) v =
-0.7071 -0.7071 -0.7071 0.7071
d = 2 0 0 8
11
22
53
35
11
22
12
12
sqrt
sqrt(
7071.
7071.,
7071.
7071.11 vv
D. van Alphen 11
SVD Factorization: Finding U
• Start with: A = U V (1)
• Right-multiply both sides of equation (1) by A :
A A = U V (U V)
= U V V U = U U
• As before, is a diagonal matrix, with the elements i2 on the
diagonal
• The eigenvectors of (A A) are the columns of matrix U.
I
D. van Alphen 12
SVD Example, continued
• Recall A =
• A A’ =
• MATLAB Code to find eigenvalues and eigenvectors of AA’:
>> [U d] = eig([8 0; 0 2])U =
0 1 1 0
d = 2 0 0 8
11
22
20
08
12
12
11
22
1
0,
0
121 uu
D. van Alphen 13
SVD Example, continued
• Recall A = = U V
– where U =
• Hence, A = U V
11
22
20
022
10
01
7071.7071.
7071.7071.V
10
01
20
022
7071.7071.
7071.7071.
D. van Alphen 14
SVD for Data Compression
• Recall A = U V = [u1 … ur … um]
A = 1 u1 v1 + 2 u2 v2 + … + r ur vr (2)
• If A has rank r, then only r terms are required in the above formula for an exact representation of A.
• To approximate A, we use fewer than r terms in equation (2).
– Since the i’s are sorted from largest to smallest value in , eliminating the last “few” terms in (2) have a small effect on the image.
'n
'r
'1
r
1
v0
0
v
v
D. van Alphen 15
Example: Clown Image
50 100 150 200 250 300
20
40
60
80
100
120
140
160
180
200
50 100 150 200 250 300
20
40
60
80
100
120
140
160
180
200
(200, 320) (200, 320)
Original Image, Uncompressed;
Full rank: r = 200
Compressed Image using only the first 20 singular values
D. van Alphen 16
Clown Image Compression: MATLAB Code
% program clown_svdload clown % uncompressed image, stored in Xfigure(1); colormap(map);size(X), r = rank(X), image(X)[U D V] = svd(X); % svd of the clown imaged = diag(D); % creates vector with singular valuesfigure(2); semilogy(d); % plot singular valuesk = 20; % # of terms to be used in approx.dd = d(1:k); % pick off 20 largest singular valuesUU = U(:,1:k);VV = V(:,1:k);XX = UU*diag(dd)*VV';figure(3); colormap(map)size(XX), image(XX) % compressed image
D. van Alphen 17
Singular Values for Clown Image
0 20 40 60 80 100 120 140 160 180 20010
0
101
102
103
104
Index of the 200 singular values
Singular
values
For this image: 20th singular value is 4% of 1st singular value
A = 1 u1 v1 + … + 20 u20 v20
+ 21 u21 v21 + … + 200 u200 v200
Negligible due to small singular values, i
D. van Alphen 18
Clown Image: How Much Compression?
• Original Image: 200 x 320 = 64,000 pixel values to store
• Compressed Image: A = 1 u1 v1 + 2 u2 v2 + … + 20 u20 v20
– Each ui has m = 200 components;– Each vi has n = 320 components;– Each i is a scalar 1 component
• Fraction of storage required for compressed image – 10,420/64,000 = 16.3%
• General formula for fraction of storage required:k * (m + n + 1)/(m*n)
where k = # of terms included in approximation
Total = 20 * (200+320 + 1)
= 10,420 values to store
D. van Alphen 19
MATLAB Commands for SVD
>> [U S V] = svd(A)
– Returns all the singular values in S, and the corresponding vectors in U and V;
>> [U S V] = svds(A, k)
– Returns the k largest singular values in S, and the corresponding vectors in U and V;
>> [U S V] = svds(A)
– Returns the 6 largest singular values in S, and the corresponding vectors in U and V;
>> Z = U*S*V’
– Regenerates A or A (the approximation to A), depending upon whether or not you used all of the singular values