tutorial 4. least squares
DESCRIPTION
Tutorial 4. Least Squares. 1. Reminder: Least squares solution. Taking the derivative we obtain:. The normal equation:. If then there is a unique global solution:. If A is square and invertible, then the unique solution is. - PowerPoint PPT PresentationTRANSCRIPT
Tutorial 4. Least Squares.
2
1. Reminder: Least squares solution
bxbxminbxminbx T22x
AAAA
bbxbbxxxmin TTTTTT
x AAAA
Taking the derivative we obtain: 0b2x2 TT AAA
bx TT AAA The normal equation:
0T AAIf then there is a unique global solution: b*x T1T AAA
If A is square and invertible, then the unique solution is b*x 1A
3
cfx2xxmin TT
xK
The derivative this time is given by: 0f2x2 K
0KIf then there is a unique global solution: f*x 1K
General Quadratic Equation (K positive semi-definite):
If K is singular, any solution satisfying leads to the minimal value, and there are infinitely many of those.
f*x K
4
2. Derivatives of Matrix-Vector Expressions
)x(
TT
)x(
TT fxe2xxcxb2xx)x(f
βα
DA
DDAA TT ;
Given the function:
With symmetric matrices:
)x(ex2)x(bx2)x(f αβ DA
The minimizer of f(x) should satisfy the cubic equation:
e)x(b)x(x)x()x( αβαβ DA
5
(continue)
2TT cxb2xx)x(f AFor the special case:
We obtain the requirement:
Thus, the possible solutions (zeroing the gradient of the function) are:
0bxx Aα
0bx.2
0x.1
A
α
If , the first solution is meaningless, and second solution is the correct one.
If , the first solution is the correct one, and the second represents a local maximum!!!!!!
0xx α
0xx α
6
A=[4 1; 1 2]; b=[2;1]; c=-50;
for ind1=1:100 x1=ind1/10-5; for ind2=1:100 x2=ind2/10-5; h(ind1,ind2)=[x1,x2]*A*[x1;x2]-b'*[x1;x2]+c; end;end;figure(1); meshc(h);figure(2); meshc(h.*h);axis([0 100 0 100 0 3000]); caxis([0 3000]);
Example: 2TT cxb2xx)x(f A
Note: In this case, the value of (x) can get below zero, as the next figures show
7
8
-3 -2 -1 0 1 2 3-30
-20
-10
0
10
20
30
40
50
3. Perform: Polynomial fitting
x = (-3:.06:3)'; PointNum=size(x,1);C=[-3 1 2 -1];y0=C(1)+C(2)*x+C(3)*x.^2+C(4)*x.^3; y=y0+10*randn(PointNum,1);figure(1); clf; plot(x,y0,'r');hold on; plot(x,y,'.b');
A=[ones(PointNum,1),x,x.^2,x.^3,x.^4,x.^5,x.^6,x.^7,x.^8,x.^9,x.^10];c10=inv(A'*A)*A'*y;c8=inv(A(:,1:8)'*A(:,1:8))*A(:,1:8)'*y;c6=inv(A(:,1:6)'*A(:,1:6))*A(:,1:6)'*y;c4=inv(A(:,1:4)'*A(:,1:4))*A(:,1:4)'*y;c2=inv(A(:,1:2)'*A(:,1:2))*A(:,1:2)'*y;
Lets search by curve fitting the best Polynomial of orders 2-10, and see how they perform (visually and by evaluating the error):
9
(continue)
disp(sqrt(mean((y0-A*c10).^2)));figure(2); clf; plot(x,y0,'r'); hold on; plot(x,y,'.b');plot(x,A*c10,'g');
2.2604
-3 -2 -1 0 1 2 3-30
-20
-10
0
10
20
30
40
50
10
(continue)
disp(sqrt(mean((y0-A*c8).^2)));figure(3); clf; plot(x,y0,'r'); hold on; plot(x,y,'.b');plot(x,A(:,1:8)*c8,'g');
1.9877
-3 -2 -1 0 1 2 3-30
-20
-10
0
10
20
30
40
50
11
(continue)
disp(sqrt(mean((y0-A*c6).^2)));figure(4); clf; plot(x,y0,'r'); hold on; plot(x,y,'.b');plot(x,A(:,1:6)*c6,'g');
1.8918
-3 -2 -1 0 1 2 3-30
-20
-10
0
10
20
30
40
50
12
(continue)
disp(sqrt(mean((y0-A*c4).^2)));figure(5); clf; plot(x,y0,'r'); hold on; plot(x,y,'.b');plot(x,A(:,1:4)*c4,'g');
0.8114
-3 -2 -1 0 1 2 3-30
-20
-10
0
10
20
30
40
50
13
(continue)
disp(sqrt(mean((y0-A*c2).^2)));figure(6); clf; plot(x,y0,'r'); hold on; plot(x,y,'.b');plot(x,A(:,1:2)*c2,'g');
6.9345
-3 -2 -1 0 1 2 3-30
-20
-10
0
10
20
30
40
50