ams526: numerical analysis i (numerical linear...

AMS526: Numerical Analysis I(Numerical Linear Algebra)

Lecture 1: Course Overview;Matrix Multiplication

Xiangmin Jiao

Stony Brook University

Xiangmin Jiao Numerical Analysis I 1 / 22

Outline

1 Course Overview

2 The Language of Matrix Computations

3 Algorithms and Efficiency


Course Description

What is numerical linear algebra?I Solving linear algebra problems using efficient algorithms on computers

Topics: Direct and iterative methods for solving simultaneous linearequations. Matrix factorization, conditioning, stability, sparsity, andefficiency. Computation of eigenvalues and eigenvectors. Singularvalue decompositionRequired textbook (also an excellent reference book)

I G. H. Golub and C. F. Van Loan, Matrix Computations, 4th Edition,Johns Hopkins University Press, 2012. ISBN 978-1421407944.

Supplementary textbook (a very insightful book)I L. N. Trefethen and D. Bau III, Numerical Linear Algebra, SIAM, 1997.


Prerequisite

Prerequisite/Co-requisite:I AMS 510 (linear algebra portion) or equivalent undergraduate-level

linear algebra course. Familiarity with following concepts is assumed:Vector spaces, Gaussian elimination, Gram-Schmidt orthogonalization,and eigenvalues/eigenvectors

I AMS 595 (co-requisite for students without programming experience)I This MUST NOT be your first course in linear algebra, or you will get

lost

To review fundamental concepts of linear algebra, see textbook such asI Gilbert Strang, Linear Algebra and Its Applications, 4th Edition, Brooks

Cole, 2006.


Why Learn Numerical Linear Algebra?Numerical linear algebra is foundation of scientific computationsMany problems ultimately reduce to linear algebra concepts oralgorithms, either analytical or computationalExamples: Finite-element analysis, data fitting, PageRank (Google)

Focus of this course: Fundamental concepts, efficiency and stability ofalgorithms, and programming


Course Outline

Basic linear algebra concepts and algorithms (1.5 weeks)Gaussian elimination and its variants (2 weeks)Sensitivity and stability (1.5 weeks)Linear least squares problem (1.5 weeks)Eigenvalues, eigenvectors, and SVD (4 weeks)Iterative methods for linear systems (2 weeks)Course webpage:http://www.ams.sunysb.edu/~jiao/teaching/ams526_fall14

Note: Course schedule online is tentative and is subject to change.


http://www.ams.sunysb.edu/~jiao/teaching/ams526_fall14

Course Policy

Assignments (written or programming)I Assignments are due in class one to two weeks after assignedI You can discuss course materials and homework problems with others,

but you must write your answers completely independentlyI Do NOT copy solutions from any source. Do NOT share your solutions

with others

Exams and testsI All exams are closed-bookI However, one-page cheat sheet is allowed

GradingI Assignments: 30%I Two midterm exams: 40%I Final exam: 30%


Outline

1 Course Overview




Matrices and Vectors

Denote vector space of all m-by-n real matrices by Rm⇥n.

A 2 Rm⇥n , A = (aij) =

2

6664

a11

a12

. . . a1n

a21

a22

. . . a2n

......

. . ....

am1

am2

. . . amn

3

7775

Denote vector space of all real n-vectors by Rn, or Rn⇥1

x 2 Rn , x =

2

6664

x1

x2

...xn

3

7775

Transposition (Rm⇥n ! Rn⇥m): C = AT ) cij = aji

Row vectors are transpose of column vectors and are in R1⇥n


Matrix OperationsAddition and subtraction (Rm⇥n ⇥ Rm⇥n ! Rm⇥n):C = A ± B ) cij = aij ± bij

Scalar-matrix multiplication or scaling (R⇥ Rm⇥n ! Rm⇥n):C = ↵A) cij = ↵aij

Matrix-matrix multiplication/product (Rm⇥p ⇥ Rp⇥n ! Rm⇥n):C = A B ) cij =

Ppk=1

aikbkj (denoted by A ⇤ B in MATLAB)Each operation also applies to vectors. In particular,

I Inner product is row vector times column vector, i.e., c = xT y(it is called dot product in vector calculus and denoted as x · y)

I Outer product is column vector times row vector, i.e., C = xyT

(it is a special case of Kronecker product and denoted as x ⌦ y)

Elementwise multiplication and division (Rm⇥n ⇥ Rm⇥n ! Rm⇥n)I C = A. ⇤ B ) cij = aijbijI C = A./B ) cij = aij/bij , where bij 6= 0

Matrix inversion (A�1) and division (A/B and A\B) to be defined later


Notation of Matrices and VectorsMatrix notation

I Capital letters (e.g., A, B, �, etc.) for matricesI Corresponding lower case with subscript ij (e.g., aij , bij , �ij) for (i , j)

entry; sometimes with notation [A]ij or A(i , j)

Vector notationI Lowercase letters (e.g., x , y , etc.) for vectorsI Corresponding lower case with subscript i for ith entry (e.g., xi , yi )

Lower-case letters for scalars (e.g., c , s, ↵, �, etc.)Some suggest using boldface lowercase (e.g., x) for vectors, regularlowercase (e.g. c) for scalars, and boldface uppercase for matricesA matrix is a collection of column vectors or row vectors

A 2 Rm⇥n , A = [c1

|c2

| . . . |cn] , ck 2 Rm

A 2 Rm⇥n , A =

2

64r

T1

...r

Tm

3

75 , rk 2 Rn


Complex Matrices

Occasionally, complex matrices are involvedVector space of m-by-n complex matrices is designated by Cm⇥n

I Scaling, addition, multiplication of complex matrices correspondexactly to real case

I If A = B + iC 2 Cm⇥n, then Re(A) = B, Im(A) = C , and conjugate ofA is A = (aij)

I Conjugate transpose is defined as AH = BT � iCT , or

G = AH ) gij = aji

(also called adjoint, and denoted by A⇤; (AB)⇤ = B⇤A⇤)

Vector space of complex n-vectors is designated by Cn

I Inner product of complex n-vectors x and y is s = xHyI Outer product of complex n-vectors x and y is S = xyH

We will primarily focus on real matrices


Matrix-Vector Product

Matrix-vector product b = Ax is special case of matrix-matrix product

bi =nX

j=1

aijxj

For A 2 Rm⇥n, Ax is a mapping x 7�! Ax from Rn to Rm

This map is linear, which means that for any x , y 2 Rn and any ↵ 2 R

A(x + y) = Ax + AyA(↵x) = ↵Ax


Linear Combination

Let aj denote jth column of matrix AI Alternative notation is colon notation: A(:, j) or a:,jI Use A(i , :) or ai,: to denote ith row of A

b is a linear combination of column vectors of A, i.e.,

b = Ax =nX

j=1

xjaj = x1

2

6664

a11

a21

...am1

3

7775+ x

2

2

6664

a12

a22

...am2

3

7775+ · · ·+ xn

2

6664

a1m

a2n...

amn

3

7775

In summary, two different views of matrix-vector products:1 Scalar operations: bi =

Pnj=1 aijxj : A acts on x to produce b

2 Vector operations: b =Pn

j=1 xjaj : x acts on A to produce b


Matrix-Matrix Multiplication

Computes C = AB, where A 2 Rm⇥r , B 2 Rr⇥n, and C 2 Rm⇥n

Element-wise, each entry of C is

cij =rX

k=1

aikbkj

Column-wise, each column of C is

cj = Abj =rX

k=1

bkjak ;

in other words, jth column of C is A times jth column of B


Outline

1 Course Overview




Algorithms for Matrix-Vector Multiplication

Suppose A 2 Rm⇥n and x 2 Rn

MATLAB-style code for b = Ax :

Row oriented

for i = 1 : mb(i) = 0for j = 1 : n

b(i) = b(i)+A(i , j)⇤x(j)end

end

Column oriented

b(:) = 0for j = 1 : n

for i = 1 : mb(i) = b(i)+A(i , j)⇤x(j)

end

end

Number of operations is O(mn), but big-Oh notation is insufficientNumber of operations is 2mn: coefficient of leading-order term isimportant for comparison


Flop Count

It is important to assess efficiency of algorithms. But how?I We could implement different algorithms and do direct comparison, but

implementation details can affect true performanceI We could estimate cost of all operations, but it is very tediousI Relatively simple and effective approach is to estimate amount of

floating-point operations, or “flops”, and focus on asymptotic analysisas sizes of matrices approach infinity

IdealizationI Count each operation +,�, ⇤, /, and p as one flopI This estimation is crude, as it omits data movement in memory, which

is non-negligible on modern computer architectures (e.g., different looporders can affect cache performance)

Matrix-vector product requires about 2mn flopsSuppose m = n, it takes quadratic time in n, or O(n2)


Algorithms for Saxpy: Scalar a x plus y

Saxpy computes ax + y and updates y

y = ax + y ) yi = axi + yi

Suppose x , y 2 Rn and a 2 R

MATLAB-style code

for i = 1 : ny(i) = y(i) + a ⇤ x(i)

end

Pseudo-code

for i = 1 : nyi yi + a xi

Number of flops is 2nPseudo-code cannot run on any computer, but are human readableand straightforward to convert into real codes in any programminglanguage (e.g., C, FORTRAN, MATLAB, etc.)We use pseudo-code on slides for conciseness


Gaxpy: Generalized saxpy

Computes y = y + Ax , where A 2 Rm⇥n and x 2 Rn

Row oriented

for i = 1 : mfor j = 1 : n

yi = yi + aijxj

Column oriented

for j = 1 : nfor i = 1 : m

yi = yi + aijxj

Inner loop of column-oriented algorithm can be converted toy = y + xja:,jNumber of flops is 2mn


Matrix Multiplication Update

Computes C = C + AB, where A 2 Rm⇥r , B 2 Rr⇥n, and C 2 Rm⇥n

cij = cij +rX

k=1

aikbkj

for i = 1 : mfor j = 1 : n

for k = 1 : rcij = cij + aikbkj

Number of flops is 2mnrIn BLAS (Basic Linear Algebra Subroutines), functions are groupedinto level-1, 2, and 3, depending on whether complexity is linear,quadratic, or cubic


Six Variants of AlgorithmsThere are six variants depending on permutation of i , j , and k :

ijk, jik , ikj , jki , kij , kji

Inner product: cij = cij + ai ,:bj

for i = 1 : mfor j = 1 : n

cij = cij + ai ,:bj

Saxpy: computes as cj = cj + Abj

for j = 1 : ncj = cj + Abj

Outer product: computes as C = C +Pr

k=1

akb:,k

for k = 1 : rC = C + akb:,k


ams526: numerical analysis i (numerical linear...

Documents