basic concepts in linear algebra - boise state universitywright/courses/m365/laintro... ·...

Basic Concepts in Linear Algebra

Department of Mathematics

Boise State University

February 14, 2018

Math 365 Linear Algebra Basics February 14, 2018 1 / 39

Numerical Linear Algebra

Linear systems of equations occur in almost every area of theapplied science, engineering, and mathematics.Hence, numerical linear algebra is one of the pillars ofcomputational mathematics.


Linear systems of equations

A linear system of m equations and n unknowns can be expressed inthe following general form:

a11x1 + a12x2 + a13x3 + · · · + a1nxn = b1,a21x1 + a22x2 + a23x3 + · · · + a2nxn = b2,a31x1 + a32x2 + a33x3 + · · · + a3nxn = b3,

......

.... . .

......

am1x1 + am2x2 + am3x3 + · · · + amnxn = bm.

(1)

Here aij are the coefficients of the systems, bi are the right hand sides(RHS), and xj are the unknown values that must be determined. aijand bi will be given by the problem.


Linear systems of equations

Linear systems can be classified into the following three types:1 Square linear system: If the number of equations equals the

number of unknowns (i.e. m = n).2 Overdetermined system: If the number of equations is greater

than the number of unknowns (i.e. m > n).3 Underdetermined system: If the number equations is less than

the number of unknowns (i.e. m < n).


Matrices and vectors

A convenient notation to describe a linear system of equations is interms of matrices and vectors.


Matrices

A matrix is just a table of numbers containing m rows and ncolumns and can be expressed as:

A =

a11 a12 a13 · · · a1na21 a22 a23 · · · a2na31 a32 a33 · · · a3n

......

.... . .

...am1 am2 am3 · · · amn

.

We typically use capital letters to denote matrices.We write A ∈ Rm×n to denote a matrix with m rows and n columns.A common shorthand notation for a matrix is A =

{aij}

, where thevalues for i and j are understood from the problem.


Vectors

If the matrix only has one row or column then it is called a vector.A column vector with n entries can be expressed as

x =

x1x2x3...

xn

.

A row vector and can be expressed as

x =[x1 x2 x3 · · · xn

].

We typically use bold lower-case letters to denote vectors.A column vector with n real entries is denoted by x ∈ Rn, while arow vector is denoted by x ∈ R1×n.


Matrix & vector operations


Matrix & vector operations: Transpose

Let A ∈ Rm×n with entries

A =


......

.... . .

...am1 am2 am3 · · · amn

,then the transpose of A switches the columns of A with the rows, i.e.

AT =

a11 a21 a31 · · · am1a12 a22 a32 · · · am2a13 a23 a33 · · · am3

......

.... . .

...a1n a2n a3n · · · amn

.

Note that AT ∈ Rn×m and that (AT )T = A.Math 365 Linear Algebra Basics February 14, 2018 9 / 39

Matrix & vector operations: Transpose

The transpose can also be applied to vectors. In this case if x is a(column) vector then xT is a row vector:

if x =

x1x2x3...

xn

then xT =[x1 x2 x3 · · · xn

].

Similarly if x is row vector then xT is a column vector.


Matrix & vector operations: Matrix addition

Let A ∈ Rm×n and B ∈ Rm×n then the sum of A and B is given by

A + B ={

aij + bij

}.

This is just the sum of the corresponding entries of the elements of Aand B.

For this sum to make sense A and B must be the same size.


Matrix & vector operations: scalar multiplication

Let α be a real number and A ∈ Rm×n then the product of α and A isgiven by

αA ={α aij

}.

Note that this is just α times each entry of A.


Matrix & vector operations: Vector-vector products

There are two types of vector-vector products that arise quitefrequently. These can be derived from the definition for matrix-matrixproducts (discussed later), but it is worth stating them separately.

Let x,y ∈ Rn then the inner product or dot product of x and y is

xT y =[x1 x2 · · · xn

]

y1y2...

yn

= x1y1 + x2y2 + . . .+ xnyn =n∑

j=1

xjyj .

Note that the inner product is a single number. The inner productis sometimes denoted by x · y.


Matrix & vector operations: Vector-vector products

There are two types of vector-vector products that arise quitefrequently. These can be derived from the definition for matrix-matrixproducts (discussed later), but it is worth stating them separately.

Let x ∈ Rm and y ∈ Rn then the outer product of x with y is

xyT =

x1x2...

xm

[y1 y2 · · · yn]=

x1y1 x1y2 · · · x1ynx2y1 x2y2 · · · x2yn

......

. . ....

xmy1 xmy2 · · · xmyn

Note that the outer product is a matrix of size m-by-n.


Matrix & vector operations: Matrix-vector products

Let A ∈ Rm×n and x ∈ Rn then the product of A and x is given by

Ax =x1

a11a21a31

...am1

+ x2

a12a22a32

...am2

+ x3

a13a23a33

...am3

+ . . .+ xn

a1na2na3n

...amn

(2)

Thus, the product Ax is a linear combination of the columns of A.


Matrix & vector operations: Matrix-vector products

Important observations regarding the matrix-vector product Ax:The only way for this product to make sense is if A has the samenumber of columns as x does rows.Ax ∈ Rm, i.e. the product is a column vector containing m entries.If we let b = Ax then we can alternatively express the i th entry ofb as

bi =n∑

j=1

aijxj , i = 1, . . . ,m.

This illustrates that bi is just the inner product of the i th row of Awith the vector x.In general, computing Ax using the above formulas requires mnmultiplications and m(n − 1) additions.


Matrix & vector operations: Matrix-matrix products

Let A ∈ Rm×n and B ∈ Rn×p, and let B have columns

B =

b1 b2 · · · bp

.The matrix-matrix product C = AB is given as

C =

Ab1 Ab2 · · · Abp

.This shows the k th column of the product AB is a linear combination ofthe columns of A with the coefficients in the linear combinations beingdetermined by entries in the k th column of B.



Important observations regarding the matrix-matrix product AB,A ∈ Rm×n, B ∈ Rn×p:

Number columns of A must equal number rows B.AB ∈ Rm×p, i.e. the product is a matrix containing m rows and pcolumns.In general, AB 6= BA , i.e. the product does not commute.



Important observations regarding the matrix-matrix product AB,A ∈ Rm×n, B ∈ Rn×p:

We can express each entry of C as

cik =n∑

j=1

aijbjk , i = 1, . . . ,m, k = 1, . . . ,p.

So cik is just the inner product of the i th row of A with the k thcolumn of B.Computing AB using the above formulas requires mnpmultiplications and m(n − 1)p additions.The transpose of the product AB satisfies: (AB)T = BT AT .


Linear systems in matrix-vector notation

Recall that we can express a linear system of equations with mequations and n unknowns as

a11x1 + a12x2 + a13x3 + · · · + a1nxn = b1,a21x1 + a22x2 + a23x3 + · · · + a2nxn = b2,a31x1 + a32x2 + a33x3 + · · · + a3nxn = b3,

......

.... . .

......

am1x1 + am2x2 + am3x3 + · · · + amnxn = bm.

(3)

We can express this linear system in matrix-vector notation using theprevious definitions.


Linear systems in matrix-vector notation

Let x ∈ Rn, b ∈ Rm, and A ∈ Rm×n, then the linear system is given asAx = b, or


......

.... . .

...am1 am2 am3 · · · amn

︸︷︷︸

A

x1x2x3...

xn

︸︷︷︸

x

=

b1b2b3...

bm

︸︷︷︸

b

.


Linear systems: solvability

Recall that Ax is a linear combination of the columns of A:

Ax =x1

a11a21a31

...am1

+ x2

a12a22a32

...am2

+ x3

a13a23a33

...am3

+ . . .+ xn

a1na2na3n

...amn

.

Thus, the only way there will be a solution to Ax = b is if b can bewritten as a linear combination of the columns of A.


Linear systems: solvability

There are three possibilities for the linear system Ax = b:1 There are an infinite number of solutions that satisfy Ax = b.

An infinite number of ways to linearly combine the columns of A to equal b.

2 There is one unique solution to the linear system.Only one way to linearly combine the columns of A to equal b.

3 There is no solution to the linear system.There is no way to linearly combine the columns of A to equal b.


Special types of matrices: Diagonal matrix

A diagonal matrix is an n-by-n square matrix with zeros in every entryexcept possibly the main diagonal:

D =

d1 0 0 · · · 00 d2 0 · · · 00 0 d3 · · · 0...

......

. . ....

0 0 0 · · · dn

,

where d1,d2, . . . ,dn are some real numbers.


Special types of matrices: Identity matrix

The identity matrix is a diagonal matrix with every diagonal entry equalto 1:

I =

1 0 0 · · · 00 1 0 · · · 00 0 1 · · · 0...

......

. . ....

0 0 0 · · · 1

It has the property that for any matrix A ∈ Rn×n, IA = AI = A.


Special types of matrices: Lower triangular matrix

A matrix L ∈ Rm×n is lower triangular if all the entries above its maindiagonal are zero. Square n-by-n lower triangular matrices take theform

L =

`11 0 0 · · · 0`21 `22 0 · · · 0`31 `32 `33 · · · 0...

......

. . ....

`n1 `n2 `n3 · · · `nn

,

where `i,j , i = 1, . . . ,n, j = i , . . . ,n, are some real numbers.


Special types of matrices: Upper triangular matrix

A matrix U ∈ Rm×n is upper triangular if all the entries below its maindiagonal are zero. Square n-by-n upper triangular matrices take theform

U =

u11 u12 u13 · · · u1n0 u22 u23 · · · u2n0 0 u33 · · · u3n...

......

. . ....

0 0 0 · · · unn

,

where ui,j , i = 1, . . . ,n, j = i , . . . ,n, are some real numbers.


Special types of matrices: Symmetric matrix

A matrix A is symmetric if A = AT . Note that only square matrices canbe symmetric.


Inverse of a matrix

Let A be an n-by-n square matrix (i.e. A ∈ Rn×n). If there exists asquare matrix B ∈ Rn×n such that

BA = AB = I,

where I is the n-by-n identity matrix, then B is called the inverse of A.

The inverse of A is denoted by A−1.If A−1 exists then A is called nonsingular, otherwise it is singular.


Matrix inverses and linear systems

If A is a square, nonsingular matrix, then the solution to the linearsystem Ax = b is given formally as

x = A−1b.

Important: When solving a linear system, one should never firstcompute A−1 and then compute the product A−1b. There are muchbetter ways to solve the system (for example using Gaussianelimination when n is not too large).


Properties of matrix inverses

Suppose A is nonsingular then the following statements are trueA−1 is uniqueA−1 is nonsingular and its inverse is AAT is nonsingularIf B ∈ Rn×n is nonsingular then AB is nonsingular and(AB)−1 = B−1A−1

The linear system Ax = b has a unique solution.


Vector norms

A vector norm is a scalar quantity that reflects the “size” of avector x.The norm of a vector x is denoted as ‖x‖.There are many ways to define the size of a vector. If x ∈ Rn, thethree most popular are

one-norm : ‖x‖1 =n∑

k=1

|xk |,

two-norm : ‖x‖2 =

√√√√ n∑k=1

|xk |2,

∞-norm : ‖x‖∞ = max1≤k≤n

|xk |


Vector norms

However a vector norm is defined, it must satisfy the following threeproperties to be called a norm:

1 ‖x‖ ≥ 0 and ‖x‖ = 0 if and only if x = 0 (i.e. x contains all zerosas its entries).

2 ‖αx‖ = |α|‖x‖, for any constant α.3 ‖x + y‖ ≤ ‖x‖+ ‖y‖, where y ∈ Rn. This is called the triangle

inequality.


Unit vectors

A vector x is called a unit vector if its norm is one, i.e. ‖x‖ = 1.Unit vectors will be different depending on the norm applied.Below are several unit vectors in the one, two, and∞ norms forx ∈ R2.

−1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1−1

−0.8

−0.6

−0.4

−0.2

0

0.2

0.4

0.6

0.8

1Un it ve c tors w ith re sp e c t to the one norm

x1

x 2

−1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1−1

−0.8

−0.6

−0.4

−0.2

0

0.2

0.4

0.6

0.8

1Un it ve c tors w ith re sp e c t to the two norm

x1

x 2

−1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1−1

−0.8

−0.6

−0.4

−0.2

0

0.2

0.4

0.6

0.8

1

Unit ve c tors w ith re sp e c t to the ∞ norm

x1

x 2

(a) One-norm (b) Two-norm (c)∞-norm


Matrix norms

A matrix norm is a scalar quantity that reflects the “size” of amatrix A ∈ Rm×n.The norm of A is denoted as ‖A‖.Any matrix norm must satisfy the following four properties:

1 ‖A‖ ≥ 0 and ‖A‖ = 0 if and only if A = 0 (i.e. A contains all zerosas its entries).

2 ‖αA‖ = |α|‖A‖, for any constant α.3 ‖A + B‖ ≤ ‖A‖+ ‖B‖, where B ∈ Rm×n.4 ‖AB‖ ≤ ‖A‖‖B‖, where B ∈ Rn×p. This is called the

submultiplicative inequality.


Matrix norms

Each vector norm induces a matrix norm according to the followingdefinition:

‖A‖p = max‖x‖p 6=0

‖Ax‖p‖x‖p

= max‖x‖p=1

‖Ax‖p,

where x ∈ Rn and p = 1,2, . . ..

Induced norms describe how the matrix stretches unit vectors withrespect to that norm.


Induced matrix norms

Two popular and easy to define induced matrix norms are

One-norm : ‖A‖1 = max1≤k≤n

m∑j=1

|ajk |,

∞-norm : ‖A‖∞ = max1≤j≤m

n∑k=1

|ajk |.

The one-norm corresponds to the maximum of the one norm ofevery column.The∞-norm corresponds to the maximum of the one norm ofevery row.

The two-norm of A is defined as the largest eigenvalue of the matrixAT A. This is computationally expensive to compute.


Non-induced matrix norms

The most popular matrix norm that is not an induced norm is theFrobenius norm:

‖A‖F =

√√√√ m∑j=1

n∑k=1

|ajk |2.


Important results on matrix norms

The following are some useful inequalities involving matrix norms.Here A ∈ Rm×n:‖Ax‖ ≤ ‖A‖‖x‖

1√m‖A‖1 ≤ ‖A‖2 ≤

√n‖A‖1

1√n‖A‖∞ ≤ ‖A‖2 ≤

√m‖A‖∞

‖A‖2 ≤√‖A‖1‖A‖∞


basic concepts in linear algebra - boise state universitywright/courses/m365/laintro... ·...

Documents