Download - Numerical Linear Algebra Chap. 2: Least Squares …Numerical Linear Algebra Chap. 2: Least Squares Problems Heinrich Voss [email protected] Hamburg University of Technology Institute

Numerical Linear AlgebraChap. 2: Least Squares Problems

Heinrich [email protected]

Hamburg University of TechnologyInstitute of Numerical Simulation

TUHH Heinrich Voss Numerical Linear Algebra Chap. 2: Least Squares Problems 2005 1 / 51

http://www.tu-harburg.de/mat/hp/voss/

Projection

Projection onto a line

Problem: Given a point b ∈ Rm and a line through the origin in the direction ofa ∈ Rm

Find the point p on the line closest to the point b

Observations:p has a representattion αa for some α ∈ RThe line connecting b to p is perpendicular to the vector a

Hence

0 = aT (b − p) = aT (b − αa) = aT b − αaT a ⇒ α =aT baT a

.


Projection

Projection matrix

p = αa = aα = aaT baT a

=aaT

aT ab =: Pb

The projection of some b onto the direction of some a is obtained bymultiplying the vector b by the rank-one matrix

P =aaT

aT a


Projection

Projection onto a subspace

Problem: Given a point b ∈ Rm and n linearly independent vectorsa1, . . . , an ∈ Rm

Find the linear combination p =∑n

j=1 xjaj ∈ span{a1, . . . , an} which is closestto the point b

p is called projection of b onto span{a1, . . . , an}.

p is characterized by the condition that the error b − p is perpendicular to thesubspace, and this is equivalent to: b − p is perpendicular to a1, . . . , an.


Projection

Projection onto a subspace ct.

b − p = b − Ax is perpendicular to a1, . . . , an if and only if

(a1)T (b − Ax) = 0(a2)T (b − Ax) = 0

...(an)T (b − Ax) = 0

⇐⇒ AT (b − Ax) = 0 ⇐⇒ AT Ax = AT b.


Projection

Theorem

AT A is nonsingular if and only if the columns of A are linearly independent

Let x be in the nullspace of A. Then it holds

Ax = 0 ⇒ AT Ax = 0.

Hence, x is in the nullspace of AT A.

If x is in the nullspace of AT A, then it holds

AT Ax = 0 ⇒ 0 = xT AT Ax = (Ax)T (Ax) = ‖Ax‖2 ⇒ Ax = 0.

If the columns of A are linearly independent, then it follows x = 0.


Projection

Projection onto a subspace

If a1, . . . , an are linearly independent, then the projection p of some vector bonto span{a1, . . . , an} is given by p = Ax where x is the unique solution of

AT Ax = AT b ⇐⇒ x = (AT A)−1AT b

The projector is given by P = A(AT A)−1AT

P is symmetric, and it holds P2 = P

The distance from b to the subspace is ‖b − Pb‖


Projection

Least squares problem

−1 0 1 2 3 4 5

−1

−0.5

0

0.5

1

1.5

2


Projection

Least squares problem ct.

Given are measurements (tj , bj), j = 1, . . . , m.Find straight line f (t) = x1 + x2t which matches the measurements best

Try to solve the linear system

x1 + x2tj = bj , j = 1, . . . , m

as good as possible.

Solve the linear system 1 t11 t2...1 tm

(

x1x2

)=

b1b2...

bm

as good as possible.


Projection

Least squares problem ct.

Replace problem Ax = b by: Find x such that the error ‖Ax − b‖ is as smallas possible. The (unique) solution is called least squares solution.

Solution: The nearest point in the space {Ax : x ∈ Rn} to the point b is theprojection of b onto this space.

Hence the solution of the least squares problem is

x = (AT A)−1Ab.


Projection

Fitting a straight line

Measurements

tj −1 0 1 2 3 4 5bj −0.8 −0.4 0.1 0.5 1.1 1.4 1.9

A =

1 −11 01 11 21 31 41 5

, b =

−0.8−0.40.10.51.11.41.9

AT Ax =

(7 14

14 56

)x = AT b =

(3.8

20.3

)⇒ x =

(−0.36430.4536

)


Projection


−1 0 1 2 3 4 5

−1

−0.5

0

0.5

1

1.5

2


Projection

Fitting a straight lineGiven m points (tj , bj), j = 1, dots, m in the plane. Find a straight line x1 + x2t ,t ∈ R fitting the data in the least squares sense.

A =

1 t11 t2...1 tm

, ⇒ AT b =

(1 1 . . . 1t1 t2 . . . tm

)b1b2...

bn

=

( ∑mj=1 bj∑m

j=1 tjbj

)

AT A =

(1 1 . . . 1t1 t2 . . . tm

)1 t11 t2...1 tm

=

(m

∑mj=1 tj∑m

j=1 tj∑m

j=1 t2j

)

⇒ AT Ax =

(m

∑mj=1 tj∑m

j=1 tj∑m

j=1 t2j

)x =

( ∑mj=1 bj∑m

j=1 tjbj

)= AT b


Projection

Trigonometric polynomial

0 5 10 15 20−2

−1.5

−1

−0.5

0

0.5

1

1.5

2

2.5

3


Projection

Trigonometric polynomial ct.

Given measurements (tj , bj), j = 1, . . . , n. Find trigonometric polynomial

f (t) = x1 + x2 sin(t) + x3 cos(t)

such that the measurements are closest to the graph of f

A =

1 sin(t1) cos(t1)1 sin(t2) cos(t2)

...1 sin(tn) cos(tn)

Solve

‖Ax − b‖ = min! ⇐⇒ AT Ax = AT b


Projection

Trigonometric polynomial ct.

0 5 10 15 20−2

−1.5

−1

−0.5

0

0.5

1

1.5

2

2.5

3


Projection

Orthonormal vectorsq1, q2, . . . , qn ∈ Rm are orthonormal if

(q j)T qk =

{0 when j 6= k1 when j = k

Orthonormal vectors are linearly independent:

n∑j=1

αjq j = 0 ⇒ 0 = (qk )Tn∑

j=1

αjq j =n∑

j=1

αj(qk )T q j = αk , k = 1, . . . , n

If n = m then q1, . . . , qn is an orthonormal basis.

If q1, . . . , qn is an orthonormal basis of Rn, then x ∈ Rn has the representation

x =n∑

j=1

αjq j ⇒ (qk )T x =n∑

j=1

αj(qk )T q j = αk ⇒ x =n∑

j=1

xT q j · q j .


Projection

Example

q1 =

(cos θsin θ

), q2 =

(− sin θcos θ

), x =

(cos αsin α

)

x = xT q1 · q1 + xT q2 · q2

= ((cos α cos θ + sin α sin θ)q1 + (− cos α sin θ + sin α cos θ)q2

= cos(α− θ)

(cos θsin θ

)+ sin(α− θ)

(− sin θcos θ

).


Projection

Orthogonal matrix

An n × n matrix with orthonormal columns is called orthogonal matrix. It isusually denoted by Q.

Every orthogonal matrix Q is nonsingular.

QT Q = I ⇒ Q−1 = QT .

If Q is orthogonal then QT is orthogonal:

(QT )T QT = QQT = QQ−1 = I


Projection

Rotation

Q =

(cos θ − sin θsin θ cos θ

)rotates every vector in the plane through the angle θ

QT =

(cos θ sin θ− sin θ cos θ

)=

(cos(−θ) − sin(−θ)sin(−θ) cos(−θ)

)= Q−1

rotates every vector in the plane through the angle −θ


Projection

Permutation

P =

0 1 0 00 0 0 11 0 0 00 0 1 0

obviously is an orthogonal matrix.

Px =

0 1 0 00 0 0 11 0 0 00 0 1 0

x1x2x3x4

=

x2x4x1x3

PT =

0 0 1 01 0 0 00 0 0 10 1 0 0

= P−1 ⇒

0 0 1 01 0 0 00 0 0 10 1 0 0

x2x4x1x3

=

x1x2x3x4

puts the components back into their original order


Projection

Reflection

If u is any unit vector, then Q = I − 2uuT is orthogonal:

QT = I − 2(uuT )T = I − 2(uT )T uT = I − 2uuT = Q,

QT Q = (I − 2uuT )(I − 2uuT ) = I − 4uuT + 4uuT uuT = I

since uT u = 1.

Q defines a refection at the hyperplane u⊥: Let x = uT x · u + v , uT v = 0.Then it follows

Qx = (uT x)(I − 2uuT )u + (I − 2uuT )v= (uT x)(u − 2uuT u) + (v − 2uuT v) = −(uT x)u + v

Now it is obvious, that Q2 = Q: reflecting twice through a mirror brings backthe original.


Projection

Orthogonal matrices

If Q is orthogonal, then it leaves lengths unchanged:

‖Qx‖2 = (Qx)T (Qx) = xT QT Qx = xT x = ‖x‖2.

It also leaves angles unchanged

cos(x , y) =xT y

‖x‖ · ‖y‖=

xT QT Qy‖Qx‖ · ‖Qy‖

=(Qx)T (Qy)

‖Qx‖ · ‖Qy‖.

Orthogonal matrices are excellent for computations: numbers never grow toolarge when lengths are fixed.


Projection

Projection

Let q1, . . . , qn ∈ Rm orthonormal vectors (how do we know that m ≥ n?)

Then with Q := [q1, . . . , qn] ∈ Rm×n the projector onto V := span{q1, . . . , qn}is

P = Q(QT Q)−1QT = QQT

and the projection of x ∈ Rm onto V is

Px = QQT x =n∑

j=1

(xT q j)q j ,

the truncation of the Fourier development with respect to an orthonormalbasis containing q1, . . . , qn.


Projection


Consider the least squares problem ‖Qx − b‖ = min!, where the systemmatrix has orthonormal columns:

Solution: x = QT b


Projection

Example

Q =113

3 −12 44 −3 −12

12 4 3

is an orthogonal matrix

Represent the vector x = (1, 1, 1)T as a linear combination of the columns q j

of Q.

xT q1 =1913

, xT q2 =−1113

, xT q3 =−513

⇒ x =19

169

34

12

− 11169

−12−34

− 513

4−12

3


Projection


Given measurements (tj , bj), j = 1, . . . , m. Assume that the measurementtimes tj add to zero.

Since the scalar product of (tj)j=1,...,m and (1, 1, . . . , 1)T is zero the normalequations

AT Ax =

(m

∑mj=1 tj∑m

j=1 tj∑m

j=1 t2j

)x =

( ∑mj=1 bj∑m

j=1 tjbj

)= AT b

obtain the form (m 00

∑mj=1 t2

j

)x =

( ∑mj=1 bj∑m

j=1 tjbj

)

and the solution is

x =

( ∑mj=1 bj/m∑m

j=1 tjbj/∑m

j=1 t2j

)


Projection


Orthogonal columns are so helpful that it is worth moving the time origing toproduce them.

To this end subtract the average time t = 1m

∑mj=1 tj . Then the shifted

measurement times tj := tj − t add to zero.

The solution of the least squares problem is

x1 + x2 t = x1 + x2(t − t) = (x1 − x2t) + x2t

where

x1 =1m

m∑j=1

bj and x2 =

∑j=1 bj tj∑mj=1 t2

j

.


Projection

Example ct.

Measurements

tj −1 0 1 2 3 4 5bj −0.8 −0.4 0.1 0.5 1.1 1.4 1.9

The average time is t = 2, and t = (−3,−2,−1, 0, 1, 2, 3)T , and

x1 =17

7∑j=1

bj =3.87

, x2 =

∑j=1 bj tj∑mj=1 t2

j

=12.928

Therefore, the solution of the least equares problem is

(x1 − x2t) + x2t = −5.914

+12.928

t = −0.3643 + 0.4536t .


Projection

Gram Schmidt process

Assume that a1, . . . , an ∈ Rm are given linearly independent vectors.Problem: Determine an orthogonal basis q1, . . . , qn of V := span{a1, . . . , an}

AdvantageFor Q := [q1, . . . , qn] it holds that QT Q = diag{‖q1‖2, . . . , ‖qn‖2} is a diagonalmatrix. Hence the orthogonal projection onto V decouples:

Px := Q(QT Q)−1QT x = Qdiag{‖q1‖2, . . . , ‖qn‖2}−1 (xT q1, . . . , xT qn)T

= Q(

xT q1

‖q1‖2 , . . . ,xT qn

‖qn‖2

)T

=n∑

j=1

xT q j

‖q j‖2 q j .


Projection

Example

Project x =

1234

to the subspace spanned by q1 =

123−1

and q2 =

−1101

.

Since q1 and q2 are orthogonal, the projection is

p =xT q1

‖q1‖2 q1 +xT q2

‖q2‖2 q2 =1015

123−1

+53

−1101

=

−1321

Indeed, p − x =

−21−1−3

is orthogonal to q1 and q2.


Projection

Gram–Schmidt process

Assume that a1, . . . , an ∈ Rm are given linearly independent vectors.Problem: Determine an orthogonal basis q1, . . . , qn of V := span{a1, . . . , an}

Idea: Set q1 = a1, and for j = 2, 3, . . . , n subtract the projection of aj to thesubspace span{a1, . . . , aj−1} from aj . Then this difference q j is orthogonal tospan{a1, . . . , aj−1}, and therefore to the previously determined vectors q i .

Since q1, . . . , q j−1 is an orthogonal basis of span{a1, . . . , aj−1}, theprojections are decoupled. Hence,

q j = aj −j−1∑k=1

(aj)T qk

‖qk‖2 qk , j = 2, . . . , n.


Projection

Example

Apply the Gram–Schmidt process to a1 =

−340

, a2 =

−2111

, a3 =

510−24

q1 = a1, q2 = a2 − (a2)T (q1)

‖q1‖2 q1 =

−2111

− 5025

−340

=

431

q3 = a3− (a3)T (q1)

‖q1‖2 q1− (a3)T (q2)

‖q2‖2 q2 =

510−24

−2525

−340

−2626

431

=

43−25


Projection

Factorization

Remember

q j = aj −j−1∑k=1

(aj)T qk

‖qk‖2 qk , j = 2, . . . , n.

It is more convenient to normalize the q–vectors in the course of the algorithmand to get rid of the denominators.

The Gram–Schmidt algorithm then reads

q1 =1

‖a1‖a1

q j =

(aj −

j−1∑k=1

(aj)T qk qk

)/∥∥∥∥∥aj −j−1∑k=1

(aj)T qk qk

∥∥∥∥∥ , j = 2, . . . , n.


Projection

QR Factorization

The following form is more convenient:

a1 = ‖a1‖q1

aj =

j−1∑k=1

(aj)T qk qk +

∥∥∥∥∥aj −j−1∑k=1

(aj)T qk qk

∥∥∥∥∥q j , j = 2, . . . , n.

Since at every step aj is a linear combination of q1, . . . , q j , and later q i :s arenot involved:

A =(a1, . . . , an) =

(q1, . . . , qn)

r11 r12 r13 . . . r1nr22 r23 . . . r2n

. . .. . .

rnn

= QR

where rij = (aj)T q i , i = 1, . . . , j − 1 and rjj = ‖aj −∑j−1

k=1(aj)T qk qk‖.


Projection

Example

Determine the QR-factorization of

A = [a1, a2, a3] =

0 1 00 2 −10 2 13 3 4

q1 =1

‖a1‖a1 =

0001

‖q2‖q2 = a2 − (a2)T q1q1 =

1223

− 3

0001

=

1220

⇒ 13

1220


Projection

Example ct.

‖q3‖q3 = a3 − (a3)T q1q1 − (a3)T q2q2

=

0−114

− 4

0001

− 0

1220

=

0−110

⇒ q3 =1√2

0−110

Hence the QR–factorization of A is

0 1 00 2 −10 2 13 3 4

=

0 1/3 00 2/3 −1/

√2

0 2/3 1/√

21 0 0

3 3 4

0 3 00 0

√2


Projection

Least squares problems

Any matrix A ∈ Rm×n with linearly independent columns can be factored intoQR.Q ∈ Rm×n has orthonormal columns, and R ∈ Rn×n is upper triangular withpositive diagonal.

Given the QR–factorization of A a least squares problem with system matrix Acan be solved easily:

AT Ax = AT b ⇐⇒ (QR)T (QR)x = (QR)T b⇐⇒ RT QT QRx = RT QT b⇐⇒ RT Rx = RT QT b⇐⇒ Rx = QT b.

Instead of solving the normal equations AT Ax = AT b by Gaussian eliminationone just solves Rx = QT b by back substitution.


Projection

Example

Solve the least squares problem Ax ≈ b where0 1 00 2 −10 2 13 3 4

and b =

4321

.

Normal equations:

AT Ax =

9 9 129 18 12

12 12 18

x = AT b =

3173

⇒ x =1

18

−1028−9


Projection

Example ct.

Taking advantage of the known QR–factorization0 1 00 2 −10 2 13 3 4

=

0 1/3 00 2/3 −1/

√2

0 2/3 1/√

21 0 0

3 3 4

0 3 00 0

√2

we have to solve

Rx =

3 3 40 3 00 0

√2

x = QT b =

114/3−1/

√2

⇒ x =118

−1028−9


Projection

Gram–Schmidt algorithm

1: q1 = a1/‖a1‖2: for j=2,. . . ,n do3: q j = aj

4: for k=1,. . . ,j-1 do5: rkj = (aj)T qk ;6: q j = q j − rkjqk ;7: end for8: q j = q j/‖q j‖9: end for

In steps 5 and 6 the projection of aj onto spanqk is subtracted from q j fork = 1, . . . , j − 1. At that time

q j = aj −k−1∑`=1

(aj)T q`q`

and from (q`)T qk = 0 for ` = 1, . . . , k − 1 it follows that we can replace step 5by rkj = (q j)T qk .This version is called modified Gram–Schmidt method. It is numerically morestable than the original Gram–Schmidt factorization.


Projection

Householder transformation

Although the modified Gram–Schmidt method is knwon to be more stablethan the original Gram–Schmidt method, there is a better way to solve leastsquare problems by Householder transformation.

Problem: Given a vector a ∈ Rm; find a refection H = I − 2uuT , ‖u‖ = 1 suchthat Ha is a multiple of the first unit vector e := (1, 0, . . . , 0)T .

From the orthogonality of H it follows that ‖Ha‖ = ‖a‖. Hence,Ha = a− 2uT a · u = ±‖a‖e, and therefore u and a± ‖a‖e are parallel.

If a is not a multiple of e then there are two solutions of our problem:

u =a + ‖a‖e‖a + ‖a‖e‖

and u =a− ‖a‖e‖a− ‖a‖e‖

,

for a = ‖a‖e the first solution is defined, for a = −‖a‖e only the second one.


Projection

Householder transformation ct.In any case the Householder transformation

H = I − 2wwT

wT wwith w = a + sign(a1)‖a‖e

exists. For stability reasons (cancellation!) we always choose this one.

Let a = (2, 2, 1)T . Then ‖a‖ = 3, and w = (5, 2, 1)T , and

H = I − 1‖w‖2 wwT = I − 1

15

521

(5 2 1)⇒ Ha = −3

100

A second solution is defined by w− = (−1, 2, 1)T :

H = I − 1‖w−‖2 w−wT

− = I − 13

−121

(−1 2 1)⇒ Ha = 3

100


Projection

Least equares problem

Consider the least square problem

‖Ax − b‖ = min! where A ∈ Rm×n, b ∈ Rn.

Since orthogonal matrices do not change the length of a vector, it follows

‖Ax − b‖ = ‖Q(Ax − b)‖ for every orthogonal matrix Q ∈ Rm×m.

Assume that QA =

(RO

)where QT Q = I, R ∈ Rn×n is upper triangular and

O ∈ Rm−n,n denotes the nullmatrix. Then we obtain

‖Ax − b‖2 = ‖QAx −Qb‖2 =∥∥∥(R

O

)x −Qb

∥∥∥2= ‖Rx − b1‖2 + ‖b2‖2

where b1 ∈ Rn contains the leading n components of Qb, and b2 theremaining ones.


Projection

Least equares problem ct.

From‖Ax − b‖2 = ‖Rx − b1‖2 + ‖b2‖2

it is obvious that x = R−1b1 is the solution of the least squares problem, andthat ‖b2‖ is the residuum.

Q is constructed step by step using Householder transformations.

Let H1 be a Householder transformation such that Ha1 = ±‖a1‖e, where a1

denotes the first column of A. Then

A1 := H1A =

r11 r12 . . . r1n0...0

A1


Projection

Least equares problem ct.Next we annihilate the subdiagonal elements of the second column.

Let H2 be the Householder matrix wich maps the first column of A1 to amultiple of the first unit vector (in Rm−1), and let

H2 =

1 0 . . . 00...0

H2

Then we obtain

H2H1A = H2A1 =: A2 =

r11 r12 r13 . . . r1n0 r22 r23 . . . r2n0...0

0...0

A2


Projection

Least equares problem ct.Continuing that way we finally arrive at

HnHn−1 · · ·H1A =: An =

r11 r12 . . . r1n0 r22 . . . r2n

0 0. . .

0 0 . . . rnn0 0 . . . 0...

......

0 0 . . . 0

=:

(RO

)

With b := HnHn−1 · · ·H1b :=

(b1

bn

), b1 ∈ Rn the least squares problem obtains

the form‖Ax − b‖ = ‖Anx − b‖

with the unique solutionx = R−1b1.


Projection

Example

A = [a1, a2] =

1 11 21 21 5

, b =

2.95.15.2

11.9

From ‖a1‖ = 2 it follows that the Householder method mapping the firstcolumn a1 of A to a multiple of the first unit vector is

H1 = I − 2wwT

wT wwhere w =

1111

+ 2

1000

=

3111

.

A1 = H1A =

−3 −50 00 00 3

, H1b =

−12.55−0.050.056.75


Projection

Example ct.

Obviously, this problem is equivalent to∥∥∥∥∥∥∥∥−2 −50 30 00 0

x −

−12.55

6.75−0.050.05

∥∥∥∥∥∥∥∥ = min!

which is solved by(−2 −50 3

)x =

(−12.55

6.75

)⇒ x =

(0.652.25

)


Projection

For didactic reasons we solve the transformed problem

‖A1x − H1b‖ =

∥∥∥∥∥∥∥∥−3 −50 00 00 3

x −

−12.55−0.050.056.75

∥∥∥∥∥∥∥∥ = min!

using a Householder transformation.

H2 = I − 2vvT

vT v, v =

003

+ 3

100

=

303

maps the first column of A2 to a multiple of the first unit vector in R3

Hence

H2

003

=

−300

, H2A1 =

(1 00 H2

)A1 =

−2 −50 −30 00 0


Projection

H2

−0.050.056.75

=

−0.050.056.75

− 29

303

(3 0 3)−0.05

0.056.75

=

−6.750.050.05

and we obtain the equivalent problem∥∥∥∥∥∥∥∥−2 −50 −30 00 0

x −

−6.750.050.05

∥∥∥∥∥∥∥∥ = min!

which is solved by(−2 −50 −3

)x =

(−6.75

)⇒ x =

(0.652.25

)


Download - Numerical Linear Algebra Chap. 2: Least Squares …Numerical Linear Algebra Chap. 2: Least Squares Problems Heinrich Voss [email protected] Hamburg University of Technology Institute

Top Related