i

Sophomore Level Self-Teaching Webbook for

Computational and Algorithmic LinearAlgebra and

n-Dimensional Geometry

Katta G. MurtyDept. Industrial & Operations Engineering

University of Michigan, Ann ArborMi-48109-2117, USA

Phone: 734-763-3513, Fax: 734-764-3451e-Mail: katta murty@umich.edu

URL: http://www-personal.engin.umich.edu/ murty/Book URL:

http://ioe.engin.umich.edu/books/murty/algorithmic linear algebra/

c©2001 by Katta G. Murty. All rights reserved.

ii

Contents

Preface vii

Glossary of Symb ols and A bbreviat ions xix

A Po em on Linear A lgebra xx

1 Systems of Simultaneous Linear Equations 1

1.1 What Are Algorithms? . . . . . . . . . . . . . . . . . . 11.2 Some Example Applications Involving Linear Equations 81.3 Solving a System of Linear Equations Is a Fundamental

Computational Tool . . . . . . . . . . . . . . . . . . . 611.4 The Detached Coefficient Tableau Form . . . . . . . . 631.5 Operations on Vectors, the Zero and the Unit Vectors,

Subspaces and Coordinate Subspaces of Rn . . . . . . . 68

1.6 Notation to Denote the General System of Linear Equa-tions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76

1.7 Constraint Functions, Linearity Assumptions, Linear andAffine Functions . . . . . . . . . . . . . . . . . . . . . . 78

1.8 Solutions to a System, General Solution; Nonbasic (In-dependent), Basic (Dependent) Variables in ParametricRepresentation . . . . . . . . . . . . . . . . . . . . . . 86

1.9 Linear Dependence Relation Among Constraints, Re-dundant Constraints, Linearly Independent Systems . . 92

1.10 The Fundamental Inconsistent Equation “0 = 1” . . . . 981.11 Elementary Row Operations on a Tableau Leading to an

Equivalent System . . . . . . . . . . . . . . . . . . . . 99

1.12 Memory Matrix to Keep Track of Row Operations . . 1011.13 What to Do With “0=0”, Or “0=1” Rows Encountered

After Row Operations ? . . . . . . . . . . . . . . . . . 104

1.14 The Elimination Approach to Solve Linear Equations . 1091.15 The Pivot Step, a Convenient Tool For Carrying Out

Elimination Steps . . . . . . . . . . . . . . . . . . . . . 113

iii

1.16 The Gauss-Jordan (GJ) Pivotal Method for Solving Lin-ear Equations . . . . . . . . . . . . . . . . . . . . . . . 120

1.17 A Theorem of Alternatives for Systems of Linear Equa-tions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145

1.18 Special Triangular Systems of Linear Equations . . . . 149

1.19 The Gaussian Pivot Step . . . . . . . . . . . . . . . . . 154

1.20 The Gaussian (G) Elimination Method for Solving Lin-ear Equations . . . . . . . . . . . . . . . . . . . . . . . 157

1.21 Comparison of the GJ and the G Elimination Methods 165

1.22 Infeasibility Analysis . . . . . . . . . . . . . . . . . . . 165

1.23 Homogeneous Systems of Linear Equations . . . . . . . 171

1.24 Solving Systems of Linear Equations And/Or Inequalities177

1.25 Can We Obtain a Nonnegative Solution to a System ofLinear Equations by the GJ or the G Methods Directly? 178

1.26 Additional Exercises . . . . . . . . . . . . . . . . . . . 178

1.27 References . . . . . . . . . . . . . . . . . . . . . . . . . 183

2 Matrices, Matrix Arithmetic, Determinants 189

2.1 A Matrix as a Rectangular Array of Numbers, Its Rowsand Columns . . . . . . . . . . . . . . . . . . . . . . . 189

2.2 Matrix−Vector Multiplication . . . . . . . . . . . . . . 193

2.3 Three Ways of Representing the General System of Lin-ear Equations Through Matrix notation . . . . . . . . 197

2.4 Scalar Multiplication and Addition of Matrices, Trans-position . . . . . . . . . . . . . . . . . . . . . . . . . . 201

2.5 Matrix Multiplication & Its Various Interpretations . . 204

2.6 Some Special Matrices . . . . . . . . . . . . . . . . . . 216

2.7 Row Operations and Pivot Steps on a Matrix . . . . . 222

2.8 Determinants of Square Matrices . . . . . . . . . . . . 224

2.9 Additional Exercises . . . . . . . . . . . . . . . . . . . 248

2.10 References . . . . . . . . . . . . . . . . . . . . . . . . . 260

3 n-Dimensional Geometry 265

3.1 Viewing n-tuple Vectors As Points of n-Dimensional SpaceRn By Setting Up Cartesian Coordinate Axes . . . . . 265

iv

3.2 Translating the Origin to the Point a, Translation of aSet of Points . . . . . . . . . . . . . . . . . . . . . . . . 272

3.3 Parametric Representation of Subsets of Rn . . . . . . 276

3.4 Straight Lines in Rn . . . . . . . . . . . . . . . . . . . 277

3.5 Half-Lines and Rays in Rn . . . . . . . . . . . . . . . . 283

3.6 Line Segments in Rn . . . . . . . . . . . . . . . . . . . 286

3.7 Straight Lines Through the Origin . . . . . . . . . . . . 290

3.8 Geometric Interpretation of Addition of Two Vectors,Parallelogram Law . . . . . . . . . . . . . . . . . . . . 292

3.9 Linear Combinations, Linear Hulls or Spans, Subspaces 293

3.10 Column Space, Row Space, and Null Space of a Matrix 307

3.11 Affine Combinations, Affine Hulls or Spans, Affinespaces 308

3.12 Convex Combinations, Convex Hulls . . . . . . . . . . 324

3.13 Nonnegative Combinations, Pos Cones . . . . . . . . . 332

3.14 Hyperplanes . . . . . . . . . . . . . . . . . . . . . . . 335

3.15 Halfspaces . . . . . . . . . . . . . . . . . . . . . . . . 338

3.16 Convex Polyhedra . . . . . . . . . . . . . . . . . . . . 342

3.17 Convex and Nonconvex Sets . . . . . . . . . . . . . . . 346

3.18 Euclidean Distance, and Angles . . . . . . . . . . . . . 347

3.19 The Determinant Interpreted As A Volume Function . 353

3.20 Orthogonal Projections (Nearest Points) . . . . . . . . 358

3.21 Procedures for Some Geometric Problems . . . . . . . . 365

3.22 Additional Exercises . . . . . . . . . . . . . . . . . . . 373

3.23 References . . . . . . . . . . . . . . . . . . . . . . . . 374

4 Numerical Linear Algebra 379

4.1 Linear Dependence, Independence of Sets of Vectors . . 379

4.2 Rank and Bases For a Set of Vectors . . . . . . . . . . 382

4.3 Two Algorithms to Check Linear Dependence, Find Rankand a Basis for a Given Set of Vectors . . . . . . . . . 386

4.4 How to Get an Alternate Basis by Introducing a Non-basic Vector Into a Basis ? . . . . . . . . . . . . . . . . 394

4.5 The Rank of a Matrix . . . . . . . . . . . . . . . . . . 397

4.6 Computing the Inverse of a Nonsingular Square MatrixUsing GJ Pivot Steps . . . . . . . . . . . . . . . . . . . 402

v

4.7 Pivot Matrices, Elementary Matrices, Matrix Factoriza-tions, and the Product Form of the Inverse . . . . . . . 409

4.8 How to Update the Inverse When a Column of the Ma-trix Changes . . . . . . . . . . . . . . . . . . . . . . . . 423

4.9 Results on the General System of Linear Equations Ax =b . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 427

4.10 Optimization, Another Important Distinction BetweenLA, LP, and IP; and the Duality Theorem for LinearOptimization Subject to Linear Equations . . . . . . . 432

4.11 Memory Matrix Versions of GJ, G Elimination Methods 444

4.12 Orthogonal and Orthonormal Sets of Vectors, Gram-Schmidt Orthogonalization, QR-Factorization, Orthog-onal Matrices, Orthonormal Bases . . . . . . . . . . . . 461

4.13 Additional Exercises . . . . . . . . . . . . . . . . . . . 470

4.14 References . . . . . . . . . . . . . . . . . . . . . . . . . 472

5 Quadratic Forms, Positive, Negative (Semi) Definite-ness 477

5.1 Expressing a Quadratic Function in n Variables UsingMatrix Notation . . . . . . . . . . . . . . . . . . . . . . 477

5.2 Convex, Concave Functions; Positive (Negative) (Semi)Definiteness; Indefiniteness . . . . . . . . . . . . . . . . 480

5.3 Algorithm to Check Positive (Semi) Definiteness UsingG Pivot Steps . . . . . . . . . . . . . . . . . . . . . . . 484

5.4 Diagonalization of Quadratic Forms and Square Matrices 497

6 Eigen Values, Eigen Vectors, and Matrix Diagonaliza-tion 503

6.1 Definitions, Some Applications . . . . . . . . . . . . . . 503

6.2 Relationship of Eigen Values and Eigen Vectors to theDiagonalizability of a Matrix . . . . . . . . . . . . . . . 515

6.3 An Illustrative Application of Eigen Values, Eigen Vec-tors, and Diagonalization in Differential Equations . . . 523

6.4 How to Compute Eigen Values ? . . . . . . . . . . . . . 526

6.5 References . . . . . . . . . . . . . . . . . . . . . . . . . 527

vi

7 Software Systems for Linear Algebra Problems 531

7.1 Difference Between Solving a Small Problem By Handand Scientific Computing Using a Computer . . . . . . 531

7.2 How to Enter Vectors and Matrices in MATLAB . . . . 533

7.3 Illustrative Numerical Examples . . . . . . . . . . . . . 538

7.4 References . . . . . . . . . . . . . . . . . . . . . . . . . 552

vii

PREFAC E

Cont ents

The Importance of the Subject . . . . . . . . . . . . . viiRecent Changes in the Way Linear Algebra is Taught . viiiThe Goals of This Book . . . . . . . . . . . . . . . . . viiiImportance of Learning the Details of the Methods . . xThe Screen Version With Weblinks . . . . . . . . . . . xiThe Poetic Beauty of the Subject . . . . . . . . . . . . xiDistinct Features of this Book . . . . . . . . . . . . . . xiiAcknowledgements . . . . . . . . . . . . . . . . . . . . xviContributions Requested . . . . . . . . . . . . . . . . . xviAbout the Results and Excercises . . . . . . . . . . . . xviAbout the Numerical Exercises . . . . . . . . . . . . . xviEach Chapter in a Separate file . . . . . . . . . . . . . xviiInvitation for Feedback . . . . . . . . . . . . . . . . . . xviiGlossary of Symbols and Abbreviations . . . . . . . . . xixA Poem on Linear Algebra . . . . . . . . . . . . . . . . xx

The Imp ortance of the Sub ject

The subject of linear algebra originated more than 2000 yearsago in an effort to develop tools for modeling real world problems inthe form of systems of linear equations, and to solve and analyze theseproblems using those models. Linear algebra and its twentieth cen-tury extensions, linear and integer programming, are the mostuseful and most heavily used branches of mathematics. A thoroughknowledge of the most fundamental parts of linear algebra is an essen-tial requirement for anyone in any technical job these days in order tocarry out job duties at an adequate level. In this internet age withcomputers playing a vital and increasing role in every job, making themost effective use of computers requires a solid background in at leastthe basic parts of linear algebra.

viii

Recent C hanges in the Way Linear Algebra i sTa u g h t

Until about the 1970’s there used to be a full semester course oncomputational linear algebra at the sophomore level in all undergrad-uate engineering curricula. That course used to serve the purpose ofproviding a good background in the basics of linear algebra very well.However, due to increasing pressure to introduce more and more mate-rial in the undergraduate mathematics sequence of engineering curric-ula, beginning in the 1970’s many universities started combining thisfirst level linear algebra course with the course on differential equationsand shaping it into a combined course on differential equations and lin-ear algebra. On top of this, linear algebra is being squeezed out of thiscombined course gradually, with the result that now-a-days most en-gineering undergraduates get only about four weeks of instruction onlinear algebra. The net effect of all this is that many undergraduatesof science and engineering are not developing the linear algebra skillsthat they need.

The Goals of This Book

There are many outstanding books on linear algebra, but most ofthem have their focus on undergraduates in mathematics departments.They concentrate on developing theorem-proving skills in the readers,and not so much on helping them develop mathematical modeling,computational, and algorithmic skills. Students who miss a few classes,particularly those already not doing well in the class, often find it verydifficult to catch up with the help of such text books. The net resultafter about half the term is an unhealthy situation in which only thetop half of the class is comfortably following the teachers lectures, withthe rest falling more and more behind.

Also, with mounting pressure to pack more and more material intoundergraduate mathematics courses, it is highly unlikely that moreclass time can be found to allocate to linear algebra. So, an importantcomponent of a solution strategy to this problem is to see whetherstudents can be helped to learn these concepts to a large extent from

ix

their own efforts.

An incident providing evidence of self-learning: As evidenceof self-learning of higher mathematics, I will mention the strangest ofmathematical talks ever given, by Cole, in October 1903 at an AMSmeeting, on Mersenne’s claim that 267 − 1 is a prime number. Thisclaim fascinated mathematicians for over 250 years. Not a single wordwas spoken at the seminar by anyone, but everyone there understoodthe result obtained by Cole. Cole wrote on the blackboard: 267 −1 = 147573952589676412927 = a. Then he wrote 761838257287, andunderneath it 193707721. Without uttering a word, he multiplied thetwo numbers together to get a, and sat down to the wild applause fromthe audience. ��

In the same way, I strongly believe that students can learn theconcepts of numerical and algorithmic matrix algebra mostly on theirown using a carefully designed webbook with illustrative examples,exercises, figures and well placed summaries. My primary goal is tomake this such a book.

We have all seen prices of paper copies of textbooks going up everyyear. In the USA, the average undergraduate paper textbook costs atleast $70 these days, and those that do not enjoy high sales volumescost even more. To make paper for printing these books, vast areas ofbeautiful forests are being cleared, creating an unreasonable demandon the small part of nature not yet destroyed by us. So, I have decidedto make this a webbook (also called an electronic or e-textbook).The main intent in preparing this book is to use information technol-ogy and web-based tools to create an e-text that will enable studentsto comfortably operate in the happy mode of self-teaching and self-learning. Our focus is to help the students develop the mathematicalmodeling, computational, and algorithmic skills that they need to bringlinear algebra tools and facts to solving real world problems. Proofsof simple results are given without being specially labeled as proofs.Proofs of more complex facts are left out, for the reader to look upin any of the traditional textbooks on the subject, some of which arelisted in the bibliography.

x

This webbook can also be used to supplement students’ classes andtraditional textbooks and to reinforce concepts with more examplesand exercises, in order to assist them in fully understanding the mate-rial presented in classes. The self-teaching environment in the e-textmakes it possible for those students who feel that certain material wascovered too quickly during class, to go back and review on the screenthe carefully developed version in the e-text at their own pace. In thesame way those students who want to move forward with the course-work at a faster pace than the class will be able to do so on their ownin the webbook learning environment.

Imp ortance of Learning t he Details of the Metho ds

Students often ask me why it is important for them to learn thedetails of the methods for solving systems of linear equations thesedays when computers and high quality software for solving them are sowidely available everywhere, and so easy to use. I keep telling them thatmany complex real world applications demand deeper knowledge thanthat of pressing a button to solve a given system of linear equations.First, someone has to construct the system of equations from informa-tion available on the real world problem: this part of the job is calledmathematical modeling. If the model constructed yields a strange andunrealistic solution, or if it turns out to have no solution, one has toanalyze the model and decide what changes should be made in it tomake sure that it represents the real world problem better. This type ofanalysis can only be carried out efficiently by someone who has a goodunderstanding of the methods of linear algebra. We have all seen thetotal helplessness of some grocery store checkout persons who cannotadd two simple numbers when their computerized cash register fails.A person without the proper knowledge of linear algebra methods willbe in the same position when he/she has to analyze and fix an invalidmodel giving strange results to the real world problem. Fortunatelyfor grocery store checkout persons, the occurrence of cash register ter-minal failures is rare. However, for technical people trying to modeltheir problems, the need to analyze and fix an invalid model is a dailyevent. Typically, models for real world problems are revised several

xi

times before they produce an acceptable solution to the problem.

For these reasons, I believe that it is very important for all under-graduates to learn the basics of linear algebra well.

The Screen Vers ion With Weblink s

We are posting this book on the web with size 12 letters for com-fortable reading on the screen. The book will always be available onthe web at this address for anyone to access, so it is very convenientto read most of the book on the screen itself. Much of the explanatorymaterial and the examples would probably be read only a few times, sothere is no reason to print them on paper. I hope the readers willonly print the most essential portions of the book that theyfeel they need to reference frequently on paper.

With weblinks it becomes very easy to search in each chapter. Click-ing on the light shiny blue portion in the table of contents, index, orthe list of important topics at the end of each chapter, takes the screento the appropriate page. I hope the readers will find it very convenientto read the book on the computer screen itself with these links, withouthaving to print it on paper.

The Po etic B eauty o f t he Sub j ect

In classical Indian literature in Sanskrit, Telugu, etc., whenever theywanted to praise something or someone, they would compare them tothe mighty ocean. In an effort to create an even higher pedestal ofexcellence, they developed the concept of “kshiira saagar” (literally“milk ocean”, the mightiest ocean of pure milk goodness), and if theywanted to say that something is very good, they would say that it islike kshiira saagar. Actually, in terms of its applicability, sheer eleganceand depth, linear algebra is very much like kshiira saagar. In thiswebbook you will get acquainted with the most basic tools and facts oflinear algebra that are called for most frequently in applications. Thoseof you who have developed a taste for the beauty of the subject candelve deeper into the subject by looking up some of the other booksmentioned in the bibliography.

xii

Distinct Features o f t his B o o k

Chapter 1 explains the main difference between the classical sub-ject of linear algebra (study of linear equations without any linear in-equalities), and the modern 20th century subjects of linear program-ming (study of linear constraints including linear inequalities) and in-teger programming and combinatorial optimization (study of linearconstraints and integer restrictions on some variables). This importantdistinction is rarely explained in other linear algebra books. When try-ing to handle a system of linear constraints including linear inequalitiesand integer requirements on some variables by classical linear algebratechniques only, we essentially have to ignore the linear inequalitiesand integer requirements. Chapter 1 deals with constructing mathe-matical models of systems of linear equations for real world problems,and the Gauss-Jordan (GJ) and Gaussian (G) elimination methods forsolving these systems and analyzing the solution sets for such systems,including practical infeasibility analysis techniques for modifying aninfeasible system into a feasible one. To keep the new terminology in-troduced to a minimum, the concept of a “matrix” is not even used inChapter 1. All the algorithmic steps are stated using the fundamen-tal tool of row operations on the detached coefficient tableau for thesystem with the variables entered in a top row in every tableau. Thismakes it easier for the readers to see that the essence of these methodsis to take linear combinations of constraints in the original system toget an equivalent but simpler system from which a solution can be readout. Chapter 1 shows many instances of what sets this book apart fromother mathematics books on linear algebra. In the descriptions of GJ,G methods in most of these books, the variables are usually left out.Also, they state the termination condition to be that of reaching theRREF (reduced row echelon form) or the REF (row echelon form). Atableau is defined to be in RREF [REF] if it contain a full set of unitvectors [vectors of an upper triangular matrix] in proper order at theleft end.

OR people have realized that it is not important that all unit vectors(or vectors of the upper triangular matrix) be at the left end of thetableau (they can be anywhere and can be scattered all over); also it

xiii

is not important that they be in proper order from left to right. Theyuse the very simple data structure (this phrase means a strategyfor storing information generated during the algorithm, and using itto improve the efficiency of that algorithm) of associating the variablecorresponding to the rth unit vector (or the rth vector of the uppertriangular matrix) in the final tableau as the rth basic variable orbasic variable in the rth row; and they store these basic variables in acolumn on the tableau as the algorithm progresses. This data structuremakes it easier to read the solution directly from the final tableau ofthe GJ method by making all nonbasic variables = 0; and the rthbasic variable = the rth RHS constant, for all r. We present the GJ, Gmethods using this data structure. Students find it easier to understandthe main ideas behind the elimination methods using it. It also opensthe possibility of pivot column selection strategies instead of alwaysselecting the leftmost eligible column in these methods.

I believe it is George Dantzig who introduced the data structure ofstoring basic variables in his pioneering paper, the 1947 paper on thesimplex method. He called the final tableau the canonical tableau todistinguish it from the mathematical concepts RREF, REF.

Another important difference that appears in Chapter 1, betweenthis book and all the other books on linear algebra, is worth mentioning.A system of linear equations is infeasible (i.e., has no solution) iff thefundamental inconsistent equation “0 = 1” can be expressed as a linearcombination of equations in the system. Thus all the linear algebrabooks state that if the equation “0 = a” where a is a nonzero number,shows up in one of the tableaus during the application of the GJ or Gmethods on the system, then the system is infeasible. As an example,consider the system

x1 + x2 + x3 = 2

−x1 − x2 − x3 = −1

2x1 + 4x2 + 6x3 = 12

The linear combination of equations in this system with coefficients(1, 1, 0) is the equation 0x1 +0x2 +0x3 = 1 or 0 = 1, hence this systemis infeasible.

xiv

It is possible for the equation “0 = 1” to appear in the GJ or Gmethods due to computational errors. In hand computation these maybe human errors, in digital computation on a computer these may bedue to accumulation of roundoff errors. So, whenever the equation “0 =a” for some a �= 0 appears in the GJ or G methods, it is also helpfulfor the methods to produce the row vector of coefficients in a linearcombination of constraints in the original system that yields this “0 =a” equation. In the above example that vector of coefficients is (1, 1, 0).This vector has been called by the obscure name supervisor principlefor infeasibility (the idea behind this name is that if your supervisoris suspicious about your claim of infeasibility of the system, he/shecan personally verify the claim by obtaining the linear combination ofconstraints in the system with coefficients in this vector and verify thatit is “0 = a” where a �= 0). But we will call this vector of coefficients bythe simpler name evidence (or certificate) of infeasibility. Giventhis evidence one can easily check whether it leads to the inconsistentequation “0 = a” where a �= 0.

In the same way when an equation “0 = 0” appears during the GJor G methods, we know that the corresponding original constraint isa redundant constraint that can be deleted from the system withoutchanging its set of solutions. This constraint is redundant becuse itcan be obtained as a linear combination of other constraints on whichpivot steps have been carried out so far. The vector of coefficientsin this linear combination is called the evidence (or certificate) ofredundancy for this redundant constraint.

Other books do not discuss how these evidences can be obtained.These evidences are obtained easily as byproducts of the GJ, G meth-ods as explained in Section 1.12, if the memory matrix is added to thesystem at the beginning of these methods. Also, in the computation-ally efficient versions of the GJ and G methods discussed in Section4.11, we show that these evidences are automatically obtained withoutany additional expense.

Chapter 2 then introduces the concept of a matrix, the evolution ofmatrix arithmetic, and the study of determinants of square matrices.

Chapter 3 introduces the fundamentals of n-dimensional geometrystarting with the representation of vectors as points using a coordinate

xv

frame of reference and the connections between algebra and geometrythat this representation makes possible. We include a thorough treat-ment of important geometric objects such as subspaces, affine spaces,convex sets, hyperplanes, half-spaces, straight lines, half-lines, direc-tions, rays, cones, and orthogonal projections that appear most fre-quently in courses that the student is likely to take following this study,and in real world applications.

Chapter 4 on numerical linear algebra discusses the fundamentalconcepts of linear dependence and independence, rank, inverse, factor-izations; and efficient algorithms to check or compute these. Major dif-ferences in the mathematical properties of linear constraints involvingequations only, and those involving some inequalities; come to the sur-face in the study of linear optimization problems involving them. Thistopic, not discussed in other books, is the subject of Section 4.10. Inthis section we also discuss the duality theorem for linear optimizationsubject to linear equality constraints (a specialization of the dualitytheorem of linear programming) and its relationship to the theorem ofalternatives for linear equation systems. In Section 4.11 we discuss theefficient memory matrix versions of the GJ, G methods (based on theideas introduced by George Dantzig in his revised simplex method forlinear programming), these versions have the additional advantage ofproducing evidences of redundancy or infeasibility automatically with-out any additional work.

Chapter 5 deals with representing quadratic functions using matrixnotation, and efficient algorithms to check whether a given quadraticfunction is convex, concave, or neither. The treatment of positive (neg-ative) (semi)definiteness and indefiniteness of square matrices, and di-agonalization of quadratic forms is included.

Chapter 6 includes the definitions and review of the main propertiesof eigen values and eigen vectors of square matrices, and their role inmatrix diagonalizations.

Finally in the brief Chapter 7, we review the features of commercialsoftware resources available these days to solve many of the problemsdiscussed in earlier chapters.

xvi

Ack now ledgements

MATLAB computations in Chapter 7: Jeanne Whalen helpedme with the MATLAB computations in Chapter 7 on a Marian SarahParker Scholarship during Summer 2001. My thanks to her and tothe University of Michigan Marian Sarah Parker Scholarship Admin-istration (Women in Engineering Office at the University of MichiganCollege of Engineering).

Weblinks: On a Marian Sarah Parker Scholarship during Summer2003, Shital Thekdi helped me to link and make the PDF file of eachchapter completely searchable. My thanks to her and to the Universityof Michigan Marian Sarah Parker Scholarship Administration.

Suggestions, corrections, and many other kinds of help have beenreceived from several others too numerous to mention by name, and Iexpress my heartfelt thanks to all of them.

Contributions Req ues ted

Everyone using this book is requested to send a contribution ofabout US$10 to $15 to the author at the address given on the frontpage. If you are an instructor using this webbook in one of your classes,please remind your students in the first week of classes to send in theircontributions. These funds will be used to keep the webbook updated,and to improve it to be more user-friendly.

Ab out t he Res ults a nd Ex cercis es

Exercises are numbered serially beginning with number 1 in eachsection. Exercise i.j.k refers to the kth exercise in Section i.j. In thesame way Result i.j.k refers to the kth result in Section i.j.

Ab out t he Numerical Exercises

At this level most students cannot really appreciate the details of

xvii

an algorithm until they actually solve several numerical problems bythemselves by hand. That’s why I included many carefully constructednumerical exercises classified by the final outcome (infeasible system,system with unique solution, system with many solutions, redundantconstraints, etc.) so that they know what to expect when they solveeach exercise.

Given proper instructions, a digital computer can execute an algo-rithm on any data without complaining. But hand computation be-comes very tedious if complicated fractions show up. That’s why thenumerical exercises are carefully constructed so that a pivot elementof ±1 is available for most of the steps of the algorithms. I hope thatthis makes it more enjoyable to understand the algorithms by workingout the numerical exercises without encountering the tedium broughton by ugly fractions.

Each Chapter i n a Separate file

In paperbooks all chapters are always put together between twocovers and bound. But for a webbook we feel that it is convenientto have each chapter in a separate file to make it easier for users todownload. That’s why we are putting each chapter as a separate file,beginning with its individual table of contents, and ending with its ownindex of terms defined in it.

Invitation for Feedback

The web affords the wonderful opportunity of frequent revisionsof the text, subject only to the limitation of the authors time andjudgement. Feedback from users on possible improvements of the textor suggestions for interesting exercises to include to make the webbookmore useful will be greatly appreciated, these can be sent by e-mail.

Katta G. MurtyAnna Arbor, MI, December 2001, July 2003.

xviii

xix

Glossary of Symb ols and Abbreviations

R n The n-dimensional real Euclidean vector space. The

space of all vectors of the form x = (x1, . . . , xn)T (writ-ten either as a row vector as here, or as a column vector)where each xj is a real number.

=,≥,≤ Symbols for equality, greater than or equal to, less thanor equal to, which must hold for each component.

xT , AT Transpose of vector x, matrix A.(aij) Matrix with aij as the general element in it.\ Set difference symbol. If D, E are two sets, D\E is the

set of all elements of D which are not in E.A−1 Inverse of the nonsingular square matrix A.< u, v > . Dot (inner) product of vectors u, v.diag(a1, . . . , an) The n × n square matrix whose diagonal elements are

a1, . . . , an in that order, and all off-diagonal entries are0.

Ai., A.j The ith row vector, jth column vector of matrix A.det(A) Determinant of square matrix A||x|| Euclidean norm of vector x = (x1, . . . , xn), it is√

x21 + . . . + x2

n. Euclidean distance between two vectorsx, y is ||x − y||.

rank(A) Rank of a matrix A, same as the rank of its set of rowvectors, or its set of column vectors.

WRT With respect to.RHS Right hand side.PC, PR Pivot column, pivot row.RC , IE Redundant constraint, inconsistent equation identified.RI, CI Row interchange, column interchange.BV Basic variable selected for that row.GJ method The Gauss-Jordan elimination method for solving sys-

tems of linear equations.GJ pivot The Gauss-Jordan pivot step on a tableau or a matrix.G pivot The Gaussian pivot step on a tableau or a matrix.

xx

G method The Gaussian elimination method for solving systems oflinear equations.

PSD, NSD Positive semidefinite, negative semidefinite.PD, ND Positive definite, negative definite.Scalar A real number. “Scalar multiplication” means multipli-

cation by a real number.�� This symbol indicates the end of the present portion of

text (i.e., example, exercise, comment etc.). Next lineeither resumes what was being discussed before this por-tion started, or begins the next portion.

A Po em on Linear Algebra

To put you in a receptive mood as you embark on the study of thiswonderful subject, I thought it fitting to begin this book with a lovelyTelugu poem on the virtues of linear algebra composed by DhulipallaV. Rao with help from my brother Katta Sai, daughter Madhusri Kattaand myself. The original poem in roman transliteration first, followedby the English translation.

- - - - - - - - - - - - - - - - - - - - - - - - - - - -OhO lIniyar AljIbrA nIkivE mAjOhArlu!

sAgaramagAdhamani caduvukunnaMnIlOtiMkAminnani telusukunnAM

aMdAniki marOpEru araviMdaMgaNitulaku nIiMpu marimari aMdaM

jIvanAniki maMchinIru eMtO muKyaMmAnavALi pragatiki nuvvaMtakannA muKyaM

maMciruciki baMginipalli mAmiDi maharAjugaNitameriginavAriki nuvvE rArAju

xxi

paMTalaku tEneTIgalostAyi tODunuvulEkuMTE mAnavajIvitamavunu bIDu

medaDuku padunu nI adhyayanaMpragatiki sOpAnaM nI paTanaM.- - - - - - - - - - - - - - - - - - - - - - - -

Oh, linear algebra! You areDeep as the mighty oceanAs exciting as the pink lotus is beautifulAs satisfying to my brain as the sweet banginapalli∗ mango isto my tongueAs essential to science as water is to lifeAs useful in applications as bees are for the cultivation of crops+

I cannot describe in words how much richer human life has be-come because we discovered you!.

∗ A variety of mango grown in Andhra Pradesh, India, with crispsweet flesh that has no fibrous threads in it, that is consideredby some to be the most delicious among fruits.+ Bees carry out the essential task of pollinating the flowers infruit, vebetable, and some other crops. Without bees, we willnot have the abundance of fruits and vegetables we enjoy today.

Contents

1 Systems of Simultaneous Linear Equations 11.1 What Are Algorithms? . . . . . . . . . . . . . . . . . . 11.2 Some Example Applications Involving Linear Equations 81.3 Solving a System of Linear Equations Is a Fundamental

Computational Tool . . . . . . . . . . . . . . . . . . . 611.4 The Detached Coefficient Tableau Form . . . . . . . . 631.5 Operations on Vectors, the Zero and the Unit Vectors,

Subspaces and Coordinate Subspaces of Rn . . . . . . . 681.6 Notation to Denote the General System of Linear Equa-

tions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 761.7 Constraint Functions, Linearity Assumptions, Linear and

Affine Functions . . . . . . . . . . . . . . . . . . . . . . 781.8 Solutions to a System, General Solution; Nonbasic (In-

dependent), Basic (Dependent) Variables in ParametricRepresentation . . . . . . . . . . . . . . . . . . . . . . 86

1.9 Linear Dependence Relation Among Constraints, Re-dundant Constraints, Linearly Independent Systems . . 92

1.10 The Fundamental Inconsistent Equation “0 = 1” . . . . 981.11 Elementary Row Operations on a Tableau Leading to an

Equivalent System . . . . . . . . . . . . . . . . . . . . 991.12 Memory Matrix to Keep Track of Row Operations . . 1011.13 What to Do With “0=0”, Or “0=1” Rows Encountered

After Row Operations ? . . . . . . . . . . . . . . . . . 1041.14 The Elimination Approach to Solve Linear Equations . 1091.15 The Pivot Step, a Convenient Tool For Carrying Out

Elimination Steps . . . . . . . . . . . . . . . . . . . . . 113

i

ii

1.16 The Gauss-Jordan (GJ) Pivotal Method for Solving Lin-ear Equations . . . . . . . . . . . . . . . . . . . . . . . 120

1.17 A Theorem of Alternatives for Systems of Linear Equa-tions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145

1.18 Special Triangular Systems of Linear Equations . . . . 1491.19 The Gaussian Pivot Step . . . . . . . . . . . . . . . . . 1541.20 The Gaussian (G) Elimination Method for Solving Lin-

ear Equations . . . . . . . . . . . . . . . . . . . . . . . 1571.21 Comparison of the GJ and the G Elimination Methods 1651.22 Infeasibility Analysis . . . . . . . . . . . . . . . . . . . 1651.23 Homogeneous Systems of Linear Equations . . . . . . . 1711.24 Solving Systems of Linear Equations And/Or Inequalities1771.25 Can We Obtain a Nonnegative Solution to a System of

Linear Equations by the GJ or the G Methods Directly? 1781.26 Additional Exercises . . . . . . . . . . . . . . . . . . . 1781.27 References . . . . . . . . . . . . . . . . . . . . . . . . . 183

Chapter 1

Systems of SimultaneousLinear Equations

This is Chapter 1 of “Sophomore Level Self-Teaching Web-book for Computational and Algorithmic Linear Algebra andn-Dimensional Geometry” by Katta G. Murty.

1.1 What Are Algorithms?

Algorithms are procedures for solving computational problems. Some-times algorithms are also called methods.

An algorithm for a computational problem is a systematic step-by-step procedure for solving it that is so clearly stated without anyambiguities that it can be programmed for execution by a machine (acomputer).

Historical note on algorithms: The word “algorithm” origi-nated from the title of the Latin translation of a book

Algoritmi de Numero Indorum

meaning Al-Khwarizmi Concerning the Hindu Art of Reckoning. It waswritten in 825 AD by the Arabic scholar “Muhammad Ibn Musa Al-Khwarizmi” who lived in Iraq, Iran (mostly Baghdad); and was based—

1

2 Ch. 1. Linear eqs.

on earlier Indian and Arabic treatizes. This book survives only in itsLatin translation, as all the copies of the original Arabic version havebeen lost or destroyed.

Algorithms seem to have originated in the work of ancient Indianmathematicians on rules for solving linear and quadratic equations.

As an illustrative example, we provide the following problem anddiscuss an algorithm for solving it:

Problem 1: Input: Given positive integer n. Required output:Find all prime numbers ≤ n.

A prime number is a positive integer > 1 that is divisible only byitself and by 1.

We provide an algorithm for this problem dating back to the 3rdcentury BC, known as the Eratosthenes Sieve.

Algorithm: Eratosthenes Sieve for Finding All Prime Num-bers ≤ n

BEGIN

Initial Step: Write all integers 1 to n in increasing order from leftto right. Strike off 1 and put a pointer at 2, the first prime.

General Step: Suppose pointer is now on r. Counting from r,strike off every rth number to the right.

If all numbers to the right of r are now struck off, collect all thenumbers in the list that are not struck off. These are all the primes≤ n, terminate.

Otherwise move the pointer right to the next number not struckoff, and repeat this general step with the pointer in its new position.

END

Example 1: Illustration of Eratosthenes Sieve f orn = 10

1.1. Algorithms 3

We will now apply this algorithm on the instance of the problemwith n = 10. At the end of Step 1, the sequence of numbers writtenlooks like this:

� 1, 2↑ , 3, 4, 5, 6, 7, 8, 9, 10

After the General step is applied once, the sequence of numberslooks like this:

� 1, 2, 3↑ , � 4, 5, � 6, 7, � 8, 9, � 10

Applying the General step again, we get this:

� 1, 2, 3, � 4, 5↑ , � 6, 7, � 8, � 9, � 10

Applying the General step again, we get this:

� 1, 2, 3, � 4, 5, � 6, 7↑ , � 8, � 9, � 10

Now all the numbers to the right of the pointer are struck off, sowe can terminate. The set of numbers not struck off is {2, 3, 5, 7},this is the set of all primes ≤ 10. ��1

There may be several different algorithms for solving a problem.The mathematical characterization of the most efficient algorithm for aproblem is quite complicated. Informally, we will say that an algorithmfor solving a problem is more efficient than another if it is faster orrequires less computational effort.

It usually requires a deep study of a problem to uncover details ofits structure and find good characterizations of its solution, in orderto develop efficient algorithms for it. We will illustrate this statementwith an example of a problem in networks and an efficient algorithmfor it that came out of a deep study of the problem.

A network consists of points called nodes, and edges each ofwhich is a line joining a pair of nodes. Such networks are used tomodel and represent the system of streets in a city, in transportation,

1This symbol indicates the end of this portion of text, here Example 1.

4 Ch. 1. Linear eqs.

1 2

34

5

Figure 1.1:

routing, traffic flow, and distribution studies. In this representationstreet intersections or traffic centers are represented by nodes in thenetwork and two-way street segments joining adjacent intersections arerepresented by edges.

A network is said to be connected if it is possible to travel fromany node to any other node in the network using only the edges inthe network. Figure 1.1 is a network in which nodes are numberedand drawn as little circles, and each edge joins a pair of nodes. Thisnetwork is clearly connected.

An Euler circuit (named after the 18th century Swiss Mathemati-cian Leonhard Euler) in a connected network is a route which begins ata node, say node 1, goes through each edge of the network exactly once,and returns to the starting node, node 1, at the end. Euler circuits playan important role in problems such as those of finding least distanceroutes for postmen’s beats, good routes for school buses and garbagecollection vehicles, etc.

As an example, in the network with 5 nodes and 7 edges in Figure1.2, traveling along the edges in the order and direction indicated in

1.1. Algorithms 5

Figure 1.2 (the edge joining nodes 1 and 2 is traveled first in the direc-tion from 1 to 2, then the edge joining nodes 2 and 3 from node 2 tonode 3, etc.) leads to one Euler circuit in this network.

Now we describe the problem.

Problem 2: Given a connected network G with nodes 1, . . . , n,check whether there is an Euler circuit in it.

Notice that Problem 2 only poses the question of the existence ofan Euler circuit, it does not ask for an Euler circuit when one exists.

Someone who has read Problem 2 for the first time may try todevelop an algorithm for it by beginning at node 1, and then tracingthe edges in some order without repeating any edge in an effort toconstruct an Euler circuit. But the most efficient algorithm for it isan indirect algorithm developed by L. Euler in the year 1736. Thealgorithm described below, only answers the existence question, anddoes not output an Euler circuit.

1

2

34

5

1

2

3

4

5

67

Figure 1.2:

6 Ch. 1. Linear eqs.

Euler’s algorithm for Problem 2

BEGIN

Define the degree of a node in G to be the number of edges con-taining it. Find the degree of each node in G.

If the degree of every node in G is an even number, there is an Eulercircuit in G. If at least one node in G has odd degree, there is no Eulercircuit in G. Terminate.

END

Example 2: Illustration of Euler’s Algorithm tocheck t he ex is tence o f a n Euler circuit

Let di denote the degree of node i. For the network in Figure 1.2,we have (d1, d2, d3, d4, d5) = (2, 4, 2, 2, 4). Since all these node degreesare even, this network has an Euler circuit. An Euler circuit in it wasgiven above.

For the network in Figure 1.1, we have (d1, d2, d3, d4, d5) = (5, 5, 5, 5, 4).Since there are some odd numbers among these node degrees, this net-work has no Euler circuit by the above algorithm.

The basis for this algorithm is a very beautiful theorem proved byL. Euler in the year 1736 that a connected network has an Euler circuitiff the degree of every node in it is even.

The branch of science dealing with the development of efficient algo-rithms for a variety of computational problems is the most challengingarea of research today.

Exercises

1.1.1: Apply the Eratostenes Sieve to find all prime numbers ≤n = 22.

1.1. Algorithms 7

1

2

34

5

6

7

89

10

Figure 1.3: Network for Exercise 1.1.2

1.1.2: Consider the network in Figure 1.3. Check whether this net-work has an Euler circuit using the algorithm discussed above. Verifythe correctness of the answer by actually trying to construct an Eulercircuit beginning with node 1 and explain your conclusion carefully.��

In this book we will discuss algorithms for solving systems of linearequations, and other related linear algebra problems.

Systems of simultaneous linear equations pervade all technical ar-eas. Knowledge about models involving simultaneous linear equations,and the fundamental concepts behind algorithms for solving and an-alyzing them, is extremely important for everyone with aspirations ofgetting any technical job. Also, systems of simultaneous linear equa-tions lie at the heart of computational methods for solving most math-ematical models, as many of these methods include steps that requirethe solution of such systems. Hence the material discussed in this chap-ter is very fundamental, and extremely important for all students ofengineering and the applied sciences.

8 Ch. 1. Linear eqs.

We begin with some example applications that lead to models in-volving simultaneous linear equations.

1.2 Some Example Applications Involv-

ing Linear Equations

Example 1: Scrap Metal Blending Problem

Consider the following problem. A steel company has four differenttypes of scrap metal (called SM-1 to SM-4) with the following compo-sitions:

Type % in type, by weight, of elementAl Si C Fe

SM-1 5 3 4 88SM-2 7 6 5 82SM-3 2 1 3 94SM-4 1 2 1 96

They need to blend these four scrap metals into a mixture for whichthe composition by weight is:

Al - 4.43%, Si - 3.22%, C - 3.89%, Fe - 88.46%.

How should they prepare this mixture ? To answer this question,we need to determine the proportions of the 4 scrap meatals SM-1,SM-2, SM-3, SM-4 in the blend to be prepared.

Historical note on algebra, linear algebra: The most funda-mental idea in linear algebra which was discovered more than 5000years ago by the Chinese, Indians, Iranians, and Babylonians, is torepresent the quantities that we like to determine by symbols; usuallyletters of the alphabet like x, y, z; and then express the relationshipsbetween the quantities represented by these symbols in the form of—

1.2 Example applications 9

equations, and finally use these equations as tools to find out the truevalues represented by the symbols. The symbols representing the un-known quantities to be determined are nowadays called unknowns orvariables or decision variables.

The process of representing the relationships between the variablesthrough equations or other functional relationships is called modelingor mathematical modeling.

Linear algebra gradually evolved into algebra, one of the chief branchesof mathematics. Even though the subject originated more than 5000years ago, the name algebra itself came much later, it is derived fromthe title of an Arabic book Hisab al-jabr w’almuqabalah written by thesame mathematician Al-Khowarizmi discussed in Section 1.1, aroundthe same year 825 AD. The term “al-jabr” in Arabic means “restoring”in the sense of solving an equation. In Latin translation the title of thisbook became Ludus Algebrae, the second word in this title survivingas the modern word algebra for the subject.

The word linear in “linear angebra” refers to the “linear combina-tions” (see Sections 1.5, 3.5) in the spaces studied, and the linearity of“linear functions” (see Section 1.7) and “linear equations” studied inthe subject.

Model for scrap metal blending problem: Returning to ourscrap metal blending problem, in order to model it, we first define thedecision variables, denoted by x1, x2, x3, x4, where for j = 1 to 4

xj = proportion of SM-j by weight in the mixture

Then the percentage by weight, of the element Al in the mixturewill be, 5x1 + 7x2 + 2x3 + x4, which is required to be 4.43. Arguingthe same way for the elements Si, C, and Fe, we find that the decisionvariables x1 to x4 must satisfy each equation in the following systemof linear equations in order to lead to the desired mixture:

5x1 + 7x2 + 2x3 + x4 = 4.43

3x1 + 6x2 + x3 + 2x4 = 3.22

10 Ch. 1. Linear eqs.

4x1 + 5x2 + 3x3 + x4 = 3.89

88x1 + 82x2 + 94x3 + 96x4 = 88.46

x1 + x2 + x3 + x4 = 1

The last equation in the system stems from the fact that the sum ofthe proportions of various ingradients in a blend must always be equalto 1. This system of equations is the mathematical model for our scrapmetal blending problem. By solving it, we will find the values of thedecision variables x1 to x4 that define the desired blend.

Discussion

RHS Constants, Coefficients

It is customary to write the equations as above with the termsinvolving the variables on the left hand side (LHS) of the “=”sign, and the constant terms on the right hand side (RHS). Theconstant term appearing on the RHS of each equation is known asthe RHS constant in that equation. The RHS constant in the firstequation is 4.43, etc.

The number “5” appearing on the LHS of the first equation inthe system is known as the coefficient of the variable x1 in thatequation, etc.

In this system of equations, every variable appears with a nonzerocoefficient in every equation. If a variable does not appear in an equa-tion, we will say that its coefficient in that equation is zero, and viceversa.

A variable is considered to be in the system iff it has a nonzerocoefficient in at least one of the constraints. Therefore, every variablein the system must have a nonzero coefficient in at least one equation.

Moving Terms from Left to Right & Vice Versa

Terms can be moved from left to right (or vice versa) across the“=” symbol by multiplying them by a “−” sign. The justification forthis comes from the principle that when you add equal amounts to equal

1.2 Example applications 11

quantities, the resulting quantities will be equal. In other words, whenyou add the same quantity to both sides of an equation, the equationcontinues to be valid. For example, the first equation in the abovesystem is:

5x1 + 7x2 + 2x3 + x4 = 4.43

Suppose we want to move the 7x2 term in this equation fromthe left side to the right side of the “=” symbol. Add −7x2 to bothsides of this equation. By the above principle this leads to:

5x1 + 7x2 + 2x3 + x4 − 7x2 = 4.43 − 7x2

On the left hand side the +7x2 and the −7x2 terms cancel witheach other, leading to:

5x1 + 2x3 + x4 = 4.43 − 7x2

Notice that the sign of this term has changed as it moved fromthe left to the right of the equality symbol in this equation. In thesame way, we can write the first constraint in an equivalent manner bymoving the constant term from the right side of “=” to the left sideas:

5x1 + 7x2 + 2x3 + x4 − 4.43 = 0

Another equivalent way of writing the first constraint is:

5x1 = 4.43 − 7x2 − 2x3 − x4

Nonnegativity Restrictions on the Variables

The system in this example consists of 5 equations in 4 variables.From the definition of the variables given above, it is clear that a so-lution to this system of equations makes sense for the blending appli-cation under consideration, only if all the variables in the system havenonnegative values in it. Each nonnegative solution to this system of

12 Ch. 1. Linear eqs.

equations leads to a desirable mixture. The nonnegativity restrictionson the variables are linear inequality constraints. They cannot beexpressed in the form of linear equations, and since the focus of thisbook is linear equations, for the moment we will ignore them.

System of Equations, Solutions, Vectors

In the system of equations obtained above, each equation is asso-ciated with a specific requirement in the problem. For example, equa-tions 1 to 4 are associated with the requirements of the elements Al,Si, C, Fe respectively in the mixture; and equation 5 must be satisfiedby any blend, hence, we will call it the blending requirement.

In the same way, each decision variable in the system is associatedwith a unique scrap metal that can be included in the blend.

A system like this is called a system of linear equations, or moreprecisely, a system of simultaneous linear equations because asolution is required to satisfy all the equations in the system.

Using the Gauss-Jordan (GJ) elimination method discussed laterin Section 1.16, it can be determined that this system has the uniquesolution:

x = (x1, x2, x3, x4) = (0.25, 0.34, 0.39, 0.02)

which happens to be also nonnegative. Thus a mixture of desiredcomposition can only be obtained by mixing the scrap metals in thefollowing way:

Scrap metal % by weight in blendSM1 25SM2 34SM3 39SM4 2

The solution x = (x1, x2, x3, x4) = (0.25, 0.34, 0.39, 0.02) is called asolution vector to the system of equations. A solution, or solutionvector for a system of linear equations, gives specific numerical values

1.2 Example applications 13

to each variable in the system that together satisfy all the equations inthe system.

The word vector in this book refers to a set of numbers arrangedin a specified order, that’s why it is called an ordered set of num-bers. It is an ordered set because the first number in the vector isthe proportion by weight of the specific raw material SM1, the secondnumber is the proportion by weight of SM2, etc. Hence, if we changethe order of numbers in a vector, the vector changes; i.e., for exam-ple (0.25, 0.34, 0.39, 0.02) and (0.34, 0.25, 0.39, 0.02) are not the samevector.

A vector giving specific numerical values to each variable in a systemof linear equations is said to be a solution or feasible solution forthe system if it satisfies every one of the equations in the system. Avector violating at least one of the equations in the system is said tobe infeasible to the system.

Since it has four entries, x above is said to be a four dimensionalvector. An n dimensional vector will have n entries.

The definition of vector that we are using here is standard in allmathematical literature. The word “vector” is also used to representa directed line segment joining a pair of points in the two dimensionalplane in geometry and physics. Using the same word for two differentconcepts confuses young readers. To avoid confusion, we will use theword “vector” to denote an ordered set of numbers only in this book.For the other concept “the direction from one point towards anotherpoint” in space of any dimension we will use the word “direction” itselfor the phrase “direction vector”(see Section 3.5).

As we will learn later, vectors can be of two types, row vectors andcolumn vectors. If the entries in a vector are written horizontally oneafter the other, the vector is called a row vector. The solution vectorx is a row vector. If the entries in a vector are written vertically onebelow the other, the vector is called a column vector.

The symbol Rn denotes the set of all possible n dimensional vectors(either row vectors or column vectors). If u is an n dimensional vectorwith entries u1, . . . , un in that order, each ui is a real number; and weindicate this by u ∈ Rn (i.e., u is contained in Rn). An n dimensionalvector is also called an n-tuple or an n-tuple vector.

14 Ch. 1. Linear eqs.

In the n-dimensional vector x = (x1, . . . , xj−1, xj , xj+1, . . . , xn), xj

is known as the j-th entry or j-th component or j-th coordinateof the vector x, for j = 1 to n.

Rn is called the n-dimensional vector space, or the space of alln-dimensional vectors.

Historical note on vectors: The original concept of a vector(ordered set of numbers, also called a sequence now-a-days) goes backto prehistoric times in ancient India, in the development of the positionnotation for representing numbers by an ordered set or sequence ofdigits. There are 10 digits, these are 0, 1, . . ., 9. When we write thenumber 2357 as a sequence of digits ordered from left to right, we areusing a notation for the number which is actually 7 + 5(10) + 3(102) +2(103).

In the symbol “2357” the rightmost digit 7 is said to be in the“digits position”. The next digit 5 to the left is in the “10s position”.The next one 3 to the left is in the “100s position”, or “102s position”;and so on. Clearly changing the order of digits in a number changesits value, that’s why it is important to treat the digits in a number asan ordered set or sequence, which we are now calling a vector.

In the number 904 = 4 + 9(102), there is no entry in the 10s po-sition. For representing such numbers the people who developed theposition notation in ancient India realized that they had to introduce asymbol for “nothing” (“sunya” in Sanskrit), and introduced the symbol“0” (sunna) for zero or “nothing”. It is believed that zero got intro-duced officially as a digit through this process, even though it was notrecognized as a number until then.

Ex ample 2 : Athlete’s Supplemental Diet Formu-lation

The nutritionist advising a promising athlete preparing for an in-ternational competition has determined that his client needs 110 unitsof Vitamin E, 250 units of Vitamin K, and 700 calories of stored mus-cular energy (SME) per day in addition to the normal nutrition theclient picks up from a regular diet. These additional nutrients can be

1.2 Example applications 15

picked up by including five different supplemental foods with the fol-lowing compositions (remember that the units of measurement of eachsupplemental food and each nutrient may be different, for example onemay be measured in liters, another in Kg, etc.).

Nutrient Nutrient units/unit of foodFood 1 Food 2 Food 3 Food 4 Food5

Vit E 50 0 80 90 4Vit K 0 0 200 100 30SME 100 300 50 250 350

A supplemental diet specifies the quantity of each of foods 1 to 5 toinclude in the athlete’s daily intake. In order to express mathematicallythe conditions that a supplemental diet has to satisfy to meet all therequirements, we define the decision variables: for j = 1 to 5

xj = units of food j in the supplemental diet

The total amount of Vit E contained in the supplemental diet repre-sented by the vector x = (x1, x2, x3, x4, x5) is 50x1+80x3+90x4+4x5

units. Unfortunately, not all of this quantity is absorbed by the body,only a fraction of it is absorbed and the rest discarded. The fractionabsorbed varies with the food and the nutrient, and is to be deter-mined by very careful (and tedious) metabolic analysis. This data isgiven below.

Nutrient Prop. nutrient absorbed, in foodFood 1 Food 2 Food 3 Food 4 Food 5

Vit E 0.6 − 0.8 0.7 0.9Vit K − − 0.4 0.6 1.0SME 0.5 0.6 0.9 0.4 0.8

So the units of Vit E absorbed by the athlete’s body in the supple-mental diet represented by x is 50×0.6x1+80×0.8x3 +90×0.7x4+4 × 0.9x5, which is required to be 110. Arguing the same way withVit K, and Calories of SME, we find that the decision variables x1 to

16 Ch. 1. Linear eqs.

x5 must satisfy the following system of linear equations in order toyield a supplemental diet satisfying all the nutritionist’s requirements:

30x1 + 64x3 + 63x4 + 36x5 = 110

80x3 + 60x4 + 30x5 = 250

50x1 + 180x2 + 45x3 + 100x4 + 280x5 = 700

This is a system of three equations in five variables. Of course, herealso, in addition to these equations, the variables have to satisfy thenonnegativity restrictions, which we ignore.

Example 3: A Coin Assembly Problem

A class in an elementary school was assigned the following projectby their teacher. They are given an unlimited supply of US coins con-sisting of: pennies (1 cent), nickels (5 cents), dimes (10 cents), quarters(25 cents), half dollars (50 cents), and dollars. They are required toassemble a set of these coins satisfying the following constraints:

Total value of the set has to be equal to $31.58Total number of coins in the set has to be equal to 171Total weight of the set has to be equal to 735.762 grams.

Here is the weight data:

Coin Weight in gramsPenny 2.5Nickel 5.0Dime 2.268

Quarter 5.67Half Dollar 11.34

Dollar 8.1

Formulate the class’s problem as a system of linear equations.

Let the index j = 1 to 6 refer to penny, nickel, dime, quarter, half

1.2 Example applications 17

dollar, and dollar, respectively. For j = 1 to 6 define the decisionvariable:

xj = number of coins of coin j in the assembly

Then the constraints to be satisfied by the solution vector x =(x1, x2, x3, x4, x5, x6) corresponding to an assembly satisfying theteacher’s requirements are:

0.01x1 + 0.05x2 + 0.10x3 + 0.25x4 + 0.5x5 + x6 = 31.58

x1 + x2 + x3 + x4 + x5 + x6 = 171

2.5x1 + 5.0x2 + 2.268x3 + 5.67x4 + 11.34x5 + 8.1x6 = 735.762

Of course, in addition to these equations, we require the variablesto be not only nonnegative, but nonnegative integers; but we ignorethese requirements for the moment.

Example 4 : Balancing chemical reactions

In a chemical reaction, two or more input chemicals combine and re-act to form one or more output chemicals. Balancing this chemicalreaction deals with the problem of determining how many moleculesof each input chemical participate in this reaction, and how manymolecules of each output chemical are produced in the reaction. Thisleads to a system of linear equations for which the simplest positiveinteger solution is needed.

As an example, consider the well known reaction in which moleculesof hydrogen (H) and oxygen (O) combine to produce molecules of water.A molecule of hydrogen (oxygen) consists of two atoms of hydrogen(oxygen) and is represented as H2 (O2); in the same way a molecule ofwater consists of two atoms of hydrogen and one atom of oxygen andis represented as H2O. The question that arises about this reaction is:how many molecules of hydrogen combine with how many molecules ofoxygen in the reaction, and how many molecules of water are producedas a result. Let

18 Ch. 1. Linear eqs.

x1, x2, x3 denote the number of molecules of hydrogen, oxy-gen, water respectively, involved in this reaction.

Then this chemical reaction is represented as:

x1H2 + x2O2 −→ x3H2O

Balancing this chemical reaction means finding the simplest positiveintegral values for x1, x2, x3 that leads to the same number of atoms ofhydrogen and oxygen, the two elements appearing in this reaction, onboth sides of the arrow.

Equating the atoms of each element on both sides of the arrow leadsto an equation that the variables have to satisfy. In this reaction thoseequations are: 2x1 = 2x3 for hydrogen atoms, and 2x2 = x3 for oxygenatoms. This leads to the system of equations:

2x1 − 2x3 = 0

2x2 − x3 = 0

in x1, x2, x3. x1 = 2, x2 = 1, x3 = 2 leads to a simple integer solutionto this system, leading to the balanced reaction 2H2 + O2 −→ 2H2O.

In the same way balancing any chemical reaction leads to a systemof linear equations in which each variable is associated with either aninput, or an output chemical; and each constraint corresponds to achemical element in some of these chemicals. We require the simplestpositive integer solution to this system, but for the moment we willignore these requirements.

Example 5: Parameter Estimation in Curve Fit-ting Using t he Metho d of Least Squares

A common problem in all areas of scientific research is to obtaina mathematical functional relationship between a variable, say t, anda variable y whose value is known to depend on the value of t; fromdata on the values of y corresponding to various values of t, usuallydetermined experimentally. Suppose this data is:

1.2 Example applications 19

when t = tr, the observed value of y is yr; for r = 1 to k.

Using this data, we are required to come up with a mathematicalexpression for y as a function of t. This problem is known as a curvefitting problem, handling it involves the following steps.

Algorithm for Solving a Curve Fitting Problem

BEGIN

Step 1: Select a model function f(t)

An explicit mathematical function, let us denote it by f(t), contain-ing possibly several unknown parameters (or numerical constants withunknown values) that seems to provide a good fit for y is selected in thisstep. The selected function is called the model function. There maybe theoretical considerations which suggest a good model function. Orone can create a plot of the observed data on the 2-dimensional Carte-sian plane, and from the pattern of these points think of a good modelfunction.

Step 2: Parameter estimation

Let a0, a1, . . . , an denote the parameters in the model function f(t)whose numerical values are unknown.

Measuring the values of y experimentally is usually subject to un-known random errors, so even if the model function f(t) is an exactfit for y, we may not be able to find any values of the parametersa0, a1, . . . , an that satisfy:

yr = observed value of y when t = tr is = f(tr), for r = 1to k.

In this step we find the best values for the parameters a0, . . . , an

that make f(tr) as close to yr as possible, for r = 1 to k.The method of least squares for parameter estimation selects the

values of the parameters a0, . . . , an in f(t) to be those that minimizethe sum of squares of deviations (yr − f(tr)) over r = 1 to k; i.e.,——

20 Ch. 1. Linear eqs.

minimize L2(a0, . . . , an) =k∑

r=1

(yr − f(tr))2

The vector (a0, . . . , an) that minimizes L2(a0, . . . , an) is known asthe least squares estimator for the parameter vector; and with thevalues of the parameters (a0, . . . , an) fixed at (a0, . . . , an), the functionf(t) is known as the least squares fit for y as a function of t.

From calculus we know that a necessary condition for (a0, . . . , an) tominimize L2(a0, . . . , an) is that it must satisfy the equations obtainedby setting the partial derivatives of L2(a0, . . . , an) WRT a0, . . . , an

equal to 0.

∂L2(a0, . . . , an)

∂aj= 0 j = 0 to n.

This system is known as the system of normal equations.When the model function f(t) satisfies certain conditions, the sys-

tem of normal equations is a system of linear equations that can besolved by the methods to be discussed later on in this chapter. Also,any solution to this system is guaranteed to minimize L2(a0, . . . , an).A general class of model functions that satisfy these conditions are thepolynomials. In this case,

f(t) = a0 + a1t + a2t2 + . . . + antn

where the coefficients a0, . . . , an are the parameters in this model func-tion. We will discuss this case only.

If n = 1 (i.e., f(t) = a0 + a1t), we are fitting a linear function of tfor y. In Statistics, the problem of finding the best linear function of tthat fits y (i.e., fits the given data as closely as possible) is known asthe linear regression problem. In this special case, the estimatesfor the parameters obtained are called regression coefficients, andthe best fit is called the regression line for y in terms of t.

If n = 2, we are trying to fit a quadratic function of t to y. And soon.——

1.2 Example applications 21

When the model function f(t) is the polynomial a0+a1t+. . .+antn,

we have L2(a0, . . . , an) =∑k

r=1(yr − a0 − a1tr − . . . − antnr )2. So,

∂L2(a0, . . . , an)

∂aj

= 2(k∑

r=1

yrtjr − a0

k∑r=1

tjr − a1

k∑r=1

tj+1r − . . .− an

k∑r=1

tn+jr ).

for j = 0 to n. Hence the normal equations are equivalent to

a0

k∑r=1

tjr + a1

k∑r=1

tj+1r + . . . + an

k∑r=1

tn+jr =

k∑r=1

yrtjr for j = 0 to n

which is a system of n + 1 linear equations in the n + 1 parametersa0, . . . , an.

If (a0, . . . , an) is the solution of this system of linear equations, thenf(t) = a0 + a1t + . . . + antn is the best fit obtained for y.

Step 3: Checking how good the best fit obtained is

Let (a0, . . . , an) be the best values obtained in Step 2 for the un-known parameters in the model function f(t); and let f(t) denote thebest fit for y, obtained by substituting aj for aj for j = 0 to n in f(t).Here we check how well f(t) actually fits y.

The minimum value of L2(a0, . . . , an) is L2(a0, . . . , an), this quantityis called the optimal residual or the sum of squared errors.

The optimal residual will be 0 only if yr − f(tr) = 0 for all r = 1to k; i.e., the fit obtained is an exact fit to y at all values of t observedin the experiment.

The optimal residual will be large if some or all the deviationsyr − f(tr) are large. Thus the value of the optimal residual can beused to judge how well f(t) fits y. The smaller the optimal residualthe better the fit obtained.

If the optimal residual is considered too large, it is an indicationthat the model function f(t) selected in Step 1 does not provide a goodapproximation for y in terms of t. In this case Step 1 is repeated toselect a better model function, and the whole process repeated.

END

22 Ch. 1. Linear eqs.

Example: Example o f a curve fi tting problem

As a numerical example consider the yield y from a chemical reac-tion, which depends on the temperature t in the reactor. The yield ismeasured at four different temperatures on a specially selected scale,and the data is given below.

Temperature t Yield y−1 7

0 91 122 10

We will now derive the normal equations for fitting a quadraticfunction of t to y using the method of least squares. So, the modelfunction is f(t) = a0 + a1t + a2t

2, involving three parameters a0, a1, a2.In this example the sum of squared deviations

L2(a0, a1, a2) = (7 − a0 + a1 − a2)2 + (9 − a0)

2 + (12 − a0 − a1 − a2)2

+(10 − a0 − 2a1 − 4a2)2

So, ∂L2

∂a0= −2(7− a0 + a1 − a2)− 2(9− a0) −2(12− a0 − a1 − a2)−

2(10 − a0 − 2a1 − 4a2)2 = −76 + 8a0 + 4a1 + 12a2. Hence the normal

equation ∂L2

∂a0= 0 leads to

8a0 + 4a1 + 12a2 = 76

Deriving ∂L2

∂a1= 0 and ∂L2

∂a2= 0, in the same way, we have the system

of normal equations given below.

8a0 + 4a1 + 12a2 = 76

4a0 + 12a1 + 16a2 = 50

12a0 + 16a1 + 36a2 = 118

This is a system of three linear equations with three unknowns. If(a0, a1, a2) is its solution, f(t) = a0 + a1t + a2t

2 is the quadratic fitobtained for y in terms of t.

1.2 Example applications 23

We considered the case where the variable y depends on only oneindependent variable t. When y depends on two or more independentvariables, parameter estimation problems are handled by the methodof least squares in the same way by minimizing the sum of squareddeviations.

Historical note on the method of least squares: The methodof least squares is reported to have been developed by the Germanmathematician Carl Friedrich Gauss at the beginning of the 19th cen-tury while calculating the orbit of the asteroid Ceres based on recordedobservations in tracking it earlier. It was lost from view when the as-tronomer Piazzi tracking it fell ill. Gauss used the method of leastsquares and the Gauss-Jordan method for solving systems of linearequations in this work. He had to solve systems involving 17 linearequations, which are quite large for hand computation. Gauss’s accu-rate computations helped in relocating the asteroid in the skies in afew months time, and his reputation as a mathematician soared.

The Importance of Systems of Linear Equations

Systems of linear equations pervade all areas of scientific computa-tion. They are the most important tool in scientific research, develop-ment and technology. Many other problems which do not involve linearequations directly in their statement are solved by iterative methodsthat usually involve in each iteration, one or more systems of linearequations to be solved.

The Scope of this Book, Limitations

The branch of mathematics called Linear Algebra initially origi-nated for the study of systems of linear equations and for developingmethods to solve them. This occurred more than 2000 years ago. Clas-sical linear algebra concerned itself with only linear equations, and nolinear inequalities. We limit the scope of this book to linear algebrarelated to the study of linear equations.

24 Ch. 1. Linear eqs.

Extensions to Linear Inequalities, Linear Pro-gramming, Integer Programming, & CombinatorialOptimization

Historical note on linear inequalities, linear programming:Methods for solving systems of linear constraints including linear in-equalities and/or nonnegativity restrictions or other bounds on thevariables belong to the modern subject called Linear Programmingwhich began with the development of the Simplex Method to solvethem by George B. Dantzig in 1947. Most of the content of this book,with the possible exception of eigen values and eigen vectors is an es-sential prerequisite for the study of linear programming. All usefulmethods for solving linear programs are iterative methods that need tosolve in each iteration one or more systems of linear equations formedwith data coming from the current solution. The fundamental theoreti-cal result concerning linear inequalities, called Farkas’ Lemma, is thekey for the development of efficient methods for linear programming. Itappeared in J. Farkas, “Uber die Theorie der einfachen Ungleichun-gen”, (in German), J. Reine Angew. Math., 124(1902)1-24. Thuslinear programming is entirely a 20th century subject. Linear pro-gramming is outside the scope of this book.

Linear Alge-bra

Study of linearequations.Originated over2000 years ago.

→

Linear Pro-gramming

Study of lin-ear constraintsincluding in-equalities.Developed in20th century

→

Integer Pro-gramming

Study for findinginteger solu-tions to linearconstraintsincluding in-equalities.Developed in20th centuryafter linearprogramming.

Figure 1.4

1.2 Example applications 25

We noticed that some applications involved obtaining solutions tosystems of linear equations in which the variables are required to benonnegative integers. The branch of mathematics dealing with meth-ods for obtaining integer solutions to systems of linear equations (noinequalities) is called linear Diophantine equations, again a clas-sical subject that originated a long time ago. Unfortunately, linearDiophantine equations has remained a purely theoretical subject with-out many applications. The reason for this is that in applications whichcall for an integer solution to a system of linear constraints, the sys-tem usually contains nonnegativity restrictions or other bounds on thevariables and/or other inequalities. Problems in which one is requiredto find integer solutions to systems of linear constraints including someinequalities and/or bounds on variables fall under the modern subjectsknown as Integer Programming, and Combinatorial Optimiza-tion. These subjects are much further extensions of linear program-ming, and were developed after linear programming in the 20th century.Approaches for solving integer programs usually need the solution of asequence of linear programs. All these subjects are outside the scopeof this book.

Figure 1.4 summarizes the relatioships among these subjects. Thesubject at the head-end of each arrow is an extension of the one at thetail-end.

Exercises

1.2.1: Data on the thickness of various US coins is given below.

Coin Thickness in mmPenny 1.55Nickel 1.95Dime 1.35Quarter 1.75Half dollar 2.15Dollar 2.00

Formulate a version of the coin assembly problem in Example 3 in

26 Ch. 1. Linear eqs.

which the total value of the coin assembly has to be $55.93, the totalnumber of coins in the assembly has to be 254, the total weight of theassembly has to be 1367.582 grams, and the total height of the stackwhen the coins in the assembly are stacked up one on top of the otherhas to be 448.35 mm, as a system of linear equations by ignoring thenonnegative integer requirements on the decision variables.

1.2.2: In the year 2000, the US government introduced two newone dollar coins. Data on them is given below.

Coin Thickness in mm Weight in gramsSacagawea golden 1.75 5.67

Sacagawea Silver toned 2 8.1

Consider the problem stated in Exercise 1.2.1 above. The set ofcoins assembled has to satisfy all the constraints stated in Exercise1.2.1. It may consist of a nonnegative integer number of pennies, nick-els, dimes, quarters, half dollars, and dollars mentioned in Exercise1.2.1; in addition, the set is required to contain exactly two Saca-gawea golden dollars, and exactly 3 Sacagawea silver toned dollars.Discuss how this changes the linear equations model developed in Ex-ercise 1.2.1.

1.2.3: Data on the composition of various US coins is given below.Consider a version of the coin assembly problem in Example 3 in whichthe total value of the coin assembly has to be $109.40, the total numberof coins in the assembly has to be 444, the total weight of the assemblyhas to be 2516.528 grams, the total height of the stack when the coinsin the assembly are stacked up one on top of the other has to be 796.65mm; and the total amount of Zn, Ni, Cu in the assembly has to be268.125, 256.734, and 1991.669 g, respectively. Also, among these coins,the penny and the nickel are plain; while the dime, quarter, half dollar,dollar are reeded. The number of plain and reeded coins in the assemblyhas to be 194 and 250, respectively. Again, among these coins, onlythe quarter, half dollar, dollar have an eagle on the reverse; and thenumber of such coins in the assembly is required to be 209. Formulate

1.2 Example applications 27

this problem as a system of linear equations by ignoring the nonnegativeinteger requirements on the decision variables.

Coin Fraction in coin, by weight, of elementZn Ni Cu

Penny 0.975 0 0.025Nickel 0 0.25 0.75Dime 0 0.0833 0.9167

Quarter 0 0.0833 0.9167Half dollar 0 0.0833 0.9167

Dollar 0 0.0833 0.9167

1.2.4: A container has been filled with water from two taps in twodifferent experiments. In the first experiment, the first tap was lefton at full level for six minutes, and then the second tap was left onat full level for three minutes; resulting in a total flow of 114 quartsinto the container. In the second experiment a total of 66 quarts hasflown into the container when the first tap was left on at full level fortwo minutes, and then the second tap was left on at full level for threeminutes. Assuming that each tap has the property that the flow ratethrough it at full level is a constant independent of time, find these ratesper minute. Formulate this problem as a system of linear equations(from Van Nostrand Reinhold Concise Encyclopedia of Mathematics,2nd edition, 1989).

1.2.5: An antifreeze has a density of 1.135 (density of water = 1).How many quarts of antifreeze, and how many quarts of water haveto be mixed in order to obtain 100 quarts of mixture of density 1.027?(From Van Nostrand Reinhold Concise Encyclopedia of Mathematics,2nd edition, 1989.)

1.2.6: A person is planning to buy special long slacks (SLS), fancyshirts (FS) that go with them; special half-pants (SHP), and specialjackets (SJ) that go with them. Data on these items is given below.

The person wants to spend exactly $1250, and receive exactly 74bonus coupons from the purchase. Also, the total number of items

28 Ch. 1. Linear eqs.

Item Price ($/item) Bonus coupons∗

SLS 120 10FS 70 4

SHP 90 5SJ 80 3

∗awarded per item purchased

purchased should be 14. The number of SLS, FS purchased should beequal; as also the number of SHP, SJ purchased. Formulate the person’sproblem using a system of linear equations, ignoring any nonnegativeinteger requirements on the variables.

1.2.7: A person has taken a boat ride on a river from point A topoint B, 10 miles away, and back, to point A on a windless day. Theriver water is flowing in the direction of A to B at a constant speedand the boat always travels at a constant speed in still water. The tripfrom A to B took one hour, and the return trip from B to A took 5hours. Formulate the problem of finding the speeds of the river andthe boat using a system of linear equations.

1.2.8: A company has two coal mines which produce metallurgicalquality coal to be used in its three steel plants. The daily productionand usage rates are as follows:

Mine Daily productionM1 800 tonsM2 400 tons

Plant Daily requirementP1 300 tonsP2 200 tonsP3 700 tons

Each mine can ship its coal to each steel plant. The requirement ateach plant has to be met exactly, and the production at each mine hasto be shipped out daily. Define the decision variables as the amounts ofcoal shipped from each mine to each plant, and write down the systemof linear equations they have to satisfy.

1.2.9: The yield from a chemical reaction, y, is known to vary as athird degree polynomial in t − 105 where t is the temperature

1.2 Example applications 29

inside the reactor in degrees centigrade, in the range 100 ≤ t ≤ 110.Denoting the yield when the temperature inside the reactor is t byy(t), we therefore have

y(t) = a0 + a1(t − 105) + a2(t − 105)2 + a3(t − 105)3

in the range 100 ≤ t ≤ 110, where a0, a1, a2, and a3 are unknownparameters. To determine the values of these parameters, data on ob-servations on the yield taken at four different temperatures is given be-low. Write the equations for determining the parameters a0, a1, a2, a3

using this data.

Temperature Yield103 87.2105 90107 96.8109 112.4

1.2.10: A pizza outlet operates from noon to midnight each day.Its employees begin work at the following times: noon, 2 PM, 4 PM,6 PM, 8 PM, and 10 PM. All those who begin work at noon, 2 PM, 4PM, 6 PM, or 8 PM, work for a four hour period without breaks andthen go home. Those who begin work at 10 PM work only for twohours and go home when the outlet closes at midnight.

On a particular day the manager has estimated that the outlet needsthe following number of workers during the various two hour intervals.

Time interval No. workers needed12 noon − 2 PM 62 PM − 4 PM 104 PM − 6 PM 96 PM − 8 PM 88 PM − 10 PM 5

10 PM − 12 midnight 4

The manager needs to figure out how many workers should be calledfor work at noon, 2 PM, 4 PM, 6 PM, 8 PM, and 10 PM on that day to

30 Ch. 1. Linear eqs.

meet the staffing requirements given above exactly. Write the systemof equations that these decision variables have to satisfy.

1.2.11: A shoe factory makes two types of shoes, J (jogging shoes)and W (walking shoes). Shoe manufacturing basically involves twomajor operations: cutting in the cutting shop and assembly in theassembly shop. Data on the amount of man-minutes of labor requiredin each shop for each type and the profit per pair made, is given below.

Type Man-minutes needed/pair in Profit/pairCutting shop Assembly shop

J 10 12 10$W 6 3 8$

Man-mts. 130,000 135,000available

Assuming that all man-minutes of labor available in each shop areused up exactly on a particular day, and that the total profit made onthat day is $140,000, develop a linear equation model to determine howmany pairs of J and W are manufactured on that day.

1.2.12: A farmer has planted corn and soybeans on a 1000 acresection of the family farm, using up all the land in the section.

An acre planted with soybeans yields 40 bushels of soybeans, and anacre planted with corn produces 120 bushels of corn. Each acre of cornrequires 10 manhours of labor, and each acre of soybeans needs 6 man-hours of labor, over the whole season. The farmer pays $5/manhourfor labor.

The farmer sells soybeans at the rate of $8/bushel, and corn at therate of $4/bushel.

The farmer had to pay a total of $38,000 for labor over the season,and obtained a total of $384,000 from the sale of the produce.

Develop a system of linear equations to determine how many acresare planted with corn and soybeans and how many bushels of corn andsoybeans are produced.

1.2 Example applications 31

1.2.13: In a plywood company, logs are sliced into thin sheetscalled green veneer. To make plywood sheets, the veneer sheets aredried, glued, pressed in a hot press, cut to an exact size, and thenpolished.

There are different grades of veneer and different grades of plywoodsheets. For our problem assume that there are two grades of veneer (A and B ), and that the company makes two grades of plywood sheets(regular and premium). Data on inputs needed to make these sheetsand the availability of these inputs is given below.

Input Needed for 1 sheet of Available/dayRegular Premium

A-veneer sheets 2 4 100,000B-veneer sheets 4 3 150,000

Pressing mc. minutes 1 2 50,000Polishing mc. minutes 2 4 90,000Selling price($/sheet) 15 20

On a particular day, the total sales revenue was $575,000. 10,000A-veneer sheets and 20,000 B-veneer sheets of that day’s supply wereunused. The polishing machine time available was fully used up, but5,000 minutes of available pressing time was not used. Use this infor-mation and develop a system of all linear equations that you can writefor determining the number of regular and premium plywood sheetsmanufactured that day.

1.2.14: A couple, both of whom are ardent nature lovers, havedecided not to have any children of their own, because of concerns thatthe already large human population is decimating nature all over theworld. One is a practicing lawyer, and the other a practicing physician.The physician’s annual income is four times that of the lawyer.

They have a fervent desire to complete two projects. One is to buya 1000 acre tract of undeveloped land and plant it with native trees,shrubs and other flora, and provide protection so that they can growand flourish without being trampled by humanity. This is estimatedto require 10 years of the wife’s income plus 20 years of the husband’s

32 Ch. 1. Linear eqs.

income.

The other project is to fund a family planning clinic in their neigh-borhood, which is exactly half as expensive as the first project, andrequires 6 years of their combined income.

They read in a magazine article that Bill Gate’s official annualsalary at his company is half-a-million dollars, and thought “Oh!, thatis our combined annual income”.

Express as linear equations, all the constraints satisfied by theirannual incomes described above.

1.2.15: A university professor gives two midterm exams and a finalexam in each of his courses. After teaching many courses over severalsemesters he has developed a theory that the score of any student inthe final exam of one of his courses, say x3, can be predicted from thatstudent’s midterm exam scores, say x1, x2; by a function of the forma0 + a1x1 + a2x2 where a0, a1, a2 are unknown parameters.

In the following table, we give data on the scores obtained by 6students in one of his courses in all the exams. Using it, it is requiredto determine the best values for a0, a1, a2 that will make x3 as close aspossible to a0 + a1x1 + a2x2 by the method of least squares. Developthe system of normal equations from which these values for a0, a1, a2

can be determined.

Student no. Score obtained inMidterm 1 Midterm 2 Final

1 58 38 572 86 76 733 89 70 584 79 83 685 77 74 776 96 90 84

1.2.16: The following table gives data on x1 = height in inches, x2

= age in years, and x3 = weight in lbs of six male adults from a smallethnic group in a village. It is believed that a relationship of the form:x3 = a0 + a1x1 + a2x2 holds between these variables, where a0, a1, a2

1.2 Example applications 33

are unknown parameters. Develop the normal equations to determinethe best values for a0, a1, a2 that make this fit as close as possible usingthe method of least squares.

Person Height Age Weight1 62 40 1522 66 45 1713 72 50 1844 60 35 1355 69 54 1686 71 38 169

1.2.17: They have conducted experiments at an agricultural re-search station to determine the effect of the application of the threeprincipal fertilizer components, nitrogen (N), phosphorous (P), andpotassium (K), on yield in the cultivation of sugar beets. Variousquantities of N, P, K are applied on seven research plots and the yieldmeasured (all quantities are measured in their own coded units). Datais given in the following table.

Let N, P, K, and Y denote the amounts of N, P, K applied and theyield respectively. It is required to find the best fit for Y of the form a0+a1N+a2P +a3K, where a0, a1, a2, a3 are unknown parameters. Developthe normal equations for finding the best values for these parametersthat give the closest fit, by the method of least squares.

Plot no. Quantity applied YieldN P K

1 4 4 4 902 5 4 3 823 6 2 4 874 7 3 5 1005 6 8 3 1156 4 7 7 1107 3 5 3 80

1.2.18: Following table gives information on the box-office revenue

34 Ch. 1. Linear eqs.

(in million dollar units) that some popular movies released in 1998earned during each weekend in theaters in USA. The 1st column is theweekend number (1 being the first weekend) after the release of themovie, and data is provided for all weekends that the movie remainedamong the top-ten grossing films.

Weekend Armageddon Rush hour The wedding singer1 36.1 33.0 18.92 23.6 21.2 12.23 16.6 14.5 8.74 11.2 11.1 6.25 7.6 8.2 4.76 5.3 5.9 3.27 4.1 3.88 3.3

For each movie, it is believed that the earnings E(w) during week-end w can be represented approximately by the mathematical formula

E(w) = abw

where a, b are positive parameters that depend on the movie.For each movie separately, assuming that the formula is correct,

write a system of linear equations from which a, b can be determined(you have to make a suitable transformation of the data in order tomodel this using linear equations).

Also, if this system of linear equations has no exact solution, discusshow the least squares method for parameter estimation discussed inExample 5 can be used to get the best values for the parameters a, bthat minimizes the sum of squared deviations in the above system oflinear equations. Using the procedure discussed in Example 5 deriveanother system of linear equations from which these best estimates fora, b can be obtained.

(Data obtained from the paper: R. A. Powers, “Big Box OfficeBucks”, Mathematics Teacher, 94, no. 2 (112-120)Feb. 2001.)

1.2.19:Minnesota’s shrinking farm population: Given below

1.2 Example applications 35

is the yearly data on the number of farms in the State of Minnesotabetween 1983 (counted as year 0) to 1998 from the Minnesota Dept. ofAgriculture. Here a farm is defined to be an agricultural establishmentwith $1000 or more of agricultural sales annually. Let

t = year number (1983 corresponds to t = 0)y(t) = number of farms in Minnesota in year t in thousands.

t 0 1 2 3 4 5 6 7y(t) 102 97 96 93 92 92 90 89

t 8 9 10 11 12 13 14 15y(t) 88 88 86 85 83 82 81 80

Assume that y(t) = a + bt where a, b are unknown real parameters.Write all the linear equations that a, b have to satisfy if the assumptionis correct.

This system of equations may not have an exact solution in a, b. Inthis case the method of least squares takes the best values for a, b to bethose that minimize the sum of square deviations L(a, b) =

∑15t=0(y(t)−

a − bt)2. From calculus we know that these values are given by thesystem of equations:

∂L(a, b)

∂a= 0,

∂L(a, b)

∂b= 0

Derive this system of equations and show that it is a system of twolinear equations in the two unknowns a, b.

1.2.20:Archimedes’ Cattle Problem, also known as Archimedes’Problema Bovinum: The Greek mathematician Archimedes com-posed this problem around 251 BC in the form of a poem consistingof 22 elegiac couplets, the first of which says “If thou art diligent andwise, O stranger, compute the number of cattle of the Sun, who onceupon a time grazed on the fields of the Thrinacian isle of Sicily.” TheSun god’s herd consisting of bulls and cows, is divided into white,

36 Ch. 1. Linear eqs.

black, spotted, and brown cattle. The requirement is to determine thecomposition of this herd, i.e., the number of cattle of each sex in eachof these color groups. Ignoring the nonnegative integer requirements,formulate this problem as a system of linear equations, based on thefollowing information given about the herd.

Among the bulls:

• the number of white ones is greater than the brown ones by (onehalf plus one third)the number of black ones;

• the number of black ones is greater than the brown ones by (onequarter plus one fifth)the number of the spotted ones;

• the number of spotted ones is greater than the brown ones by(one sixth plus one seventh)the number of white ones.

Among the cows:

• the number of white ones is (one third plus one quarter)of thetotal black cattle;

• the number of black ones is (one quarter plus one fifth)the totalof the spotted cattle;

• the number of spotted is (one fifth plus one sixth)the total of thebrown cattle;

• and the number of brown ones is (one sixth plus one seventh)thetotal of the white cattle.

1.2.21: Monkey and Coconut Problem: There are 3 monkeys,and a pile of N coconuts.

A traveling group of six men come to this scene one after the other.Each man divides the number of coconuts in the remaining pile by 6,and finds that it leaves a remainder of 3. He takes one sixth of thatremaining pile himself, gives one coconut each to the three monkeys,

1.2 Example applications 37

leaves the others in the pile, and exits from the scene. The monkeys im-mediately transfer the coconuts they received to their private treasurepiles.

After all the men have left, a group of 5 female tourists show up onthe scene together. They find that if the number of remaining coconutsis divided by 5, it leaves a remainder of 3. Each one of them take onefifth of the remaining pile, and give one coconut to each of the threemonkeys.

Write the system of equations satisfied by N and the number ofcoconuts taken by each man, and each female tourist.

1.2.22: The Classic Monkey and the Bananas Problem:Three sailors were shipwrecked on a lush tropical island. They spentthe day gathering bananas, which they put in a large pile. It wasgetting dark and they were very tired, so they slept, deciding to dividethe bananas equally among themselves next morning. In the middle ofthe night one of the sailors awoke and divided the pile of bananas into3 equal shares. When he did so, he had one banana left over, he fedit to a monkey. He then hid his share in a secure place and went backto sleep. Later a 2nd sailor awoke and divided the remaining bananasinto 3 equal shares. He too found one banana left over which he fed tothe monkey. He then hid his share and went back to sleep. The 3rdsailor awoke later on, and divided the remaining bananas into 3 equalshares. Again one banana was left over, he fed it to the monkey; hidhis share and went back to sleep.

When all the sailors got up next morning, the pile was noticeablysmaller, but since they all felt guilty, none of them said anything; andthey agreed to divide the bananas equally. When they did so, onebanana was left over, which they gave to the monkey.

Write the system of linear equations satisfied by the number ofbananas in the original pile, and the number of bananas taken by eachof the sailors in the two stages. Find the positive integer solution tothis system of equations that has the smallest possible value for thenumber of bananas in the original pile.

1.2.23: (From C. Ray Wylie, Advanced Engineering Mathematics,

38 Ch. 1. Linear eqs.

6th ed., McGraw-Hill, 1995) Commercial fertilizers are usually mix-tures containing specified amounts of 3 components: potassium (K),phosphorus (P), and nitrogen (N). A garden store carries three stan-dard blends M1, M2, M3 with the following compositions.

Blend % in blend, ofK P N

M1 10 10 30M2 30 10 20M3 20 20 30

(i): A customer has a special need for a fertilizer with % of K, P,N equal to 15, 5, 25 respectively. Discuss how the store should mixM1, M2, M3 to produce a mixture with composition required by thecustomer. Formulate this problem as a system of linear equations.

(ii): Assume that a customer has a special need for a fertilizerwith % of K, P, and N equal to k, p, and n respectively. Find out theconditions that k, p, n have to satisfy so that the store can produce amixture of M1, M2, M3 with this specific composition.

1.2.24: Balancing chemical reactions: Develop the system oflinear equations to balance the following chemical reactions, and if pos-sible find the simplest positive integer solution for each of these sys-tems and express the chemical reaction using it. Besides hydrogen (H),oxygen (O) mentioned above, the elements involved in these reactionsare: lead (Pb), carbon (C), sulpher (S) nitrogen (N), phosphorous (P),chlorine (Cl), sodium (Na), cerium (Ce), molybdenum (Mo), tantalum(Ta), Chromium (Cr), arsenic (As), and manganese (Mn).

(i): CnH2n+2 is the formula for a basic hydrocarbon for positiveinteger values of n. n = 8, 10, 12 correspond to common liquid hy-drocarbons. Each of these hydrocarbons combines with oxygen in achemical reaction (combustion) to produce carbon dioxide (CO2) andwater (H2O). Balance this reaction for n = 8, 10, 12.

1.2 Example applications 39

(ii): CH4 is the formula for methane which is the main compo-nent in natural gas. Balance the chemical reaction in which methanecombines with oxygen to produce carbon dioxide and water.

(iii): C2H22O11 is the formula for sucrose (sugar). Balance thereaction in which sucrose combines with oxygen to produce carbondioxide and water.

(iv): Balance the chemical reaction in which NO2 and H2O com-bine to form HNO2 and HNO3.

(v): Balance the reaction in which the chemical compounds PbN6

and CrMn2O8 combine to produce Pb3O4, CrO3, MnO2 and NO.

(vi): Balance the reaction in which the chemical compounds MnS,As2Cr10O35, and H2SO4 combine to produce HMnO4, AsH3, CrS3O12

and H2O.

(vii): Balance the reaction in which the chemical compounds

Ta2(NH2)4(PH3)2H2 and O2 combine to produce Ta2O5, H2O, P2O5,and NO2.

(viii): Balance the reaction in which the chemical compounds

Ta2(NH2)4(PH3)2H2 and Cl2 combine to produce TaCl5, PCl5, HCl,and NCl3.

(ix): Balance the reaction in which the chemical compounds

Na8CeMo12O42 and Cl2 combine to produce NaCl, CeCl4, MoCl5,and Cl2O.

The reactions (vii), (viii), (ix) are given to me by K. Rengan.

1.2.25: Two couples C1, C2 who are very close friends, had theirwedding anniversaries on the same date. They decided to celebrate ittogether.

First they had dinner at a nice restaurant. To cover the restaurantbill couple C2 paid twice as much as C1. Then they went to a theater

40 Ch. 1. Linear eqs.

show. For the tickets couple C1 paid twice as much as C2. Together,the theater tickets cost 50% more than the restaurant bill.

Then they went for dancing in a nightclub. For the entrance feeshere couple C1 paid as much as C2 paid at the restaurant and theatertogether; and C2 paid as much as C1 paid at the restaurant and theatertogether.

After it was all over, they realized that the evenings program costthem together a total of $750. Formulate a system of linear equationsto determine who spent how much at each place, and who spent moreoverall.

1.2.26: Two couples (W1, H1), (W2, H2) met at a weight loss clinic.They found out that all of them together weigh 700 lbs, that the twowives have the same weight which is the average weight of the twohusbands, and that H2 is one-third heavier than H1. Formulate asystem of linear equations to determine their weights individually.

1.2.27: In an enclosed area in a cattle ranch there are cows grazingand cowboys riding on horseback. The total number of legs (heads) ofall the cows, horses, and cowboys put together is 2030 (510). Formulatea system of linear equations to determine the number of cows andcowboys in the area.

1.2.28: A college student went into a jewellery store in which allthe items are on sale for $40, with $1000 in her purse. She boughtsome ear ring pairs (sold only in pairs), necklaces, single bangles, andordinary rings; spending 60% of her money.

The number of ear ring pairs and ordinary rings that she purchasedis twice the number of necklaces. The number of single bangles andordinary rings that she purchased is the total number of ear rings in allthe pairs she purchased. The number of necklaces and single banglesthat she purchased is 50% more than the number of ear ring pairs andordinary rings. The number of necklaces and ordinary rings that shepurchased is one-half the number of ear ring pairs and single bangles.Formulate a system of linear equations to determine how she spent hermoney.

1.2 Example applications 41

1.2.29: A graduate who just got a new job went to the depart-ment store with $1000 in his pocket. He purchased some $30 shirts,$60 trousers, $80 sweaters, and $100 shoe pairs and stopped when herealized that he will have only $40 left in his pocket after paying for allof them. At that time he had purchased a total of 21 items.

The amount spent on shirts is equal to the amount spent on trousers.The number of sweaters purchased is equal to the number of shoe pairspurchased. Formulate a system of linear equations to determine whatthings were purchased by him.

1.2.30: (From Mahaaviiraa’s Ganiita saara sangraha, 850 AD) Thetotal price of 9 citrons and 7 wood-apples is 107 rupees; and that of7 citrons and 9 wood-apples is 101 rupees. Write the system of linearequations to determine the price of each fruit.

1.2.31: (Adopted from Kordemsky’s The Moscow Puzzles, CharlesScribner’s Sons, NY 1972).

(a): A boy has as many sisters as brothers, but each sister has onlyhalf as many sisters as brothers. Formulate a system of linear equationsto determine how many brothers and sisters are in the family.

(b): There are three tractor stations in the city. Tractors are inshort supply, and they lend each other equipment as needed. The firststation lent the 2nd and 3rd as many tractors as they each already had.A few months later the 2nd station lent the 1st and the 3rd as manyas they each had at that time. Still later, the 3rd lent the 1st and the2nd as many as they each had at that time. Each station now had 24tractors. Formulate a linear equation model to determine how manytractors each station had originally.

(c): Three brothers shared 24 apples, each getting a number equalto his age three years before. Then the youngest proposed the followingswap “I will keep only half the apples I got and divide the rest betweenyou two equally. Then the middle brother, keeping half his accumulatedapples, must divide the rest equally between the oldest brother and me;

42 Ch. 1. Linear eqs.

and then the oldest brother must do the same.”

Everyone agreed, the result was that each ended with 8 apples.Formulate a linear equation model to determine how old the brothersare.

(d): A sister S and brothers B1 to B4 went mushroom hunting. Scollected 45 mushrooms, but B1 to B4 all returned empty handed. Outof sympathy she distributed all her mushrooms among her brothers.

On the way back B1 picked up 2 new mushrooms, B2 picked up newmushrooms to double what he had, B3 lost two mushrooms, and B4 losthalf his mushrooms. At the end all the brothers had the same numberof mushrooms. Formulate a linear equation model to determine howmany mushrooms S gave each boy originally.

1.2.32: Magic squares: (i) Lo Shu, the most famous magicsquare: Associate a variable with each cell of a 3 × 3 array as in thefollowing:

x1 x2 x3

x4 x5 x6

x7 x8 x9

It is required to find values for all the variables satisfying the mag-ical property that the sum of the variables in every row, column, andthe two diagonals is the same, here 15. Write this system of linearequations.

Find a solution to this system, in which x1 to x9 assume distinctinteger values from 1 to 9 (this yields Lo Shu (Shu means writing ordocument in Chinese) which is said to have first been revealed on theshell of a sacred turtle which appeared to the mythical emperor Yufrom the waters of the Lo river in the 23rd century BC).

(ii): Magic square from India (more than 2000 year old): Sameas the above, except that we now have 16 variables x1 to x16 arrangedin a 4 × 4 array; and the sum of all the variables in each row, column,and the two diagonals is required to be 34. Write this system of linear

1.2 Example applications 43

equations. In the magic square the variables are required to assumedistinct integer values from 1 to 16, find it.

1.2.33: A group of 12 students from a class went to eat lunch ina restaurant, agreeing to split the bill equally. When the bill for $aarrived, two of them discovered that they did not have any moneywith them. Others in the group made up the difference by paying anextra $b each.

Then one of these two students who did not have any money tookthem all to an ice cream parlor where he maintains an account. Eachone in the group took ice cream worth $b, and the total bill this timecame to one-fifth of the total bill at the restaurant, which the studentcharged to his account.

Develop a system of linear equations to determine how much eachstudent paid.

1.2.34: A kid’s piggybank has cents, nickels, dimes, quarters andhalf-dollars. The number of cents is the total number of nickels anddimes together. The number of dimes is the total number of quartersand half-dollars together. The number of nickels is twice the numberof half-dollars. The value of all the dimes is the same as the value of allthe quarters. The total value in the bank is $29.10. Develop a systemof linear equations to find the total number of coins in the piggy bank.

1.2.35: A group of 12 people consists of doctors, lawyers, and en-gineers. The avarage ages of doctors, lawyers, engineers in the group is60, 30, 40 years respectively. The average age of all the group membersis 40 years. The total ages of the doctors is the same as the total agesof the lawyers, and this number is one half of the total ages of theengineers. Write a system of linear equations to determine the numberof doctors, lawyers, and engineers in the group.

1.2.36: The sum of the first 5 terms in an arithmetic sequence is40, and the sum of the first 10 terms is 155. Develop a system of linearequations to find the sum of the first 20 terms.

44 Ch. 1. Linear eqs.

1.2.37: A fundraiser for storm victims was organized in a theaterwith tickets priced at $10, $50, $100; and a total of 1600 tickets weresold. The number of $50 tickets sold was the sum of two times thenumber of $100 tickets plus the number of $10 tickets sold. The sumof the numbers of $10, $50 tickets sold was three times the number of$100 tickets sold. The sum of the numbers of $50, $100 tickets soldwas seven times the number of $10 tickets sold. Formulate a system oflinear equations to determine the amount of money raised.

1.2.38: A batch of 30 lbs of blended coffee was made by blendingtwo varieties of coffee beans valued at $2/lb, and $5/lb respectively.The mixture has a value of $3/lb. Develop a system of linear equationsto determine how many lbs of each variety were used in the blend.

1.2.39: A lady has a total of $35,000 invested in three separateinvestments I1, I2, I3 that yield annual interest of 5%, 10%, 15% re-spectively. If she can rollover the amount invested in I1 completelyinto the investment I2 (I3) her net annual yield will change to 12.86%(13.57%). If she can roll over the amount invested in I2 completely intothe investment I3 her net annual yield will change to 13.57%. Formu-late a system of linear equations to determine her present net annualyield.

1.2.40: A husband and wife team went on a car trip to a destination800 miles away. They started at 6 AM. At the beginning the husbanddrove the car nonstop at an average speed of 70 miles/hour for sometime. Then they stopped and did some touring and sightseeing on foot.Then the wife took the wheel and drove nonstop at an averag speedof 60 miles/hour to the destination. The time spent sightseeing in themiddle was one-sixth of the total time spent driving. The distancedriven by the wife was 30% of the total distance. Formulate a systemof linear equations to determine when they reached their destination.

1.2.41: Problems dealing with snow shoveling rates of peo-ple: In these problems assume that each person always shovels snow ata constant rate independently of any others who may also be shoveling

1.2 Example applications 45

snow on the same driveway.

(i): Husband and wife working together can clear their driveway in36 minutes. Working alone, the lady requires half an hour more thanher husband to complete the job. Develop a system of linear equationsto determine how long it would take each of them to complete the jobworking alone.

(ii): A family of father, mother, son have to clear their driveway.Father and son working together can finish it in 4 hours. If son andmother work on it they will finish in 3 hours. If father and mother workon it they will need 2.4 hours. Formulate a system of linear equationsto determine how long it would take each of them to complete the jobworking alone.

1.2.42: Problems dealing with ages of people: (i): PersonsP1, P2, P3 all have birthdays on 9 September. P1’s present age is 5 yearsless than the sum of the present ages of P2, P3. In 10 years P1 will betwo and a half times as old as P2 will be then. 5 years ago P2 wasone-sixth as old as P3 was. Develop a system of linear equations todetermine their present ages.

(ii): (From Metrodorus, Greek Anthology, about 500 AD) A personnamed Demochares lived one-fourth of his life as a boy, one-fifth as ayouth, one-third as a man, and the remaining 13 years in his dotagebefore he passed away. Develop a linear equation model to determinehow old he was when he died.

(iii): In 10 years a father will be twice as old as his son will bethen. 5 years ago the father was 5 times as old as his son was then.Develop a system of linear equations to determine their present ages.

(iv): 6 years ago Rod was 4 times as old as Chandra. 2 years agoRod was twice as old as Chandra. Develop a system of linear equationsto determine their present ages.

46 Ch. 1. Linear eqs.

(v): Brother B and sisters S1, S2 have discovered that when B washalf the age of S1, S2 was 30 years old. When S1 was half as old as S2,B was just born. The sum of the ages of B and S2 is twice the age ofS1. Develop a system of linear equations to find all their ages.

(vi): A teenage girl fell in love with an older man twice her age.The sum of the digits in both their ages are equal. Also, one year agothe man was one year older than twice her age. Develop a system oflinear equations to determine their ages.

(vii): B1, B2, B3 are brothers, and B1’s age is the sum of the agesof B2, B3 two years ago. The sum of the ages of B1, B3 is one morethan twice the age of B2. Also, the difference between the ages of B1

and B2 is one more than half the age of B2. Formulate a system oflinear equations to determine their ages.

(viii): Three brothers all of whose ages are of two digits, have theinteresting property that the sum of the ages of any two of them isthe number obtained by writing the age of the third in reverse order.Formulate a system of linear equations to determine their ages.

(ix): Three people P1, P2, P3 discovered that their present ages total90. When P1 was twice as old as P2, P3 was 5 years old. When P2 wastwice as old as P3, P1 was 50. Formulate a system of linear equationsto find their ages.

1.2.43: A clothing store received a truckload consisting of $20shirts, $30 slacks, and $60 sweaters. The total value of the slacksreceived was 65.22% of the sum of the values of all the shirts andsweaters received. The total value of the shirts received was 111.11%of the sum of the values of all the slacks and sweaters received. If eachoh the slacks had been valued $10 more, then the total value of theslacks received would have been equal to the total value of the shirtsreceived. Formulate a system of linear equations to determine the totalvalue of the truckload received.

1.2 Example applications 47

1.2.44: A carpenter and her assistant did some work at a home.The assistant arrived first and started the work. 4 hours later thecarpenter arrived and both of them worked for 3 more hours to finishthe work. The homeowner paid both of them together $360 for thework. If each of them had worked 2 hours longer, the total bill wouldhave been $520. Formulate a system of linear equations to determinetheir hourly rate of pay.

1.2.45: A girl’s piggybank consists of a total of 220 coins consistingof nickels, dimes, and quarters, and has a value of $35.

If the numbers of nickels and dimes are doubled, the value wouldgo up by 28.57% of the present value.

If the number of quarters is tripled, and the number of nickels ischanged to 11 times their present number the value will be tripled.

If the number of dimes is tripled, and the number of nickels in-creased by 75%, the value will go up by 50%.

Formulate a system of linear equations to determine the number ofcoins of each type in the piggybank.

1.2.46: A person receives yearly income of $13,600 from invest-ments he made during his working years in bonds bearing 5%, 7%,8%, and 10% interest annually. If the amounts invested in the 5%,7% bonds are interchanged, he will earn $400 less annually. If theamounts invested in the 5%, 8% bonds are interchanged, he will earn$1200 less annually. If the amounts invested in the 5%, 10% bondsare interchanged, he will earn $1500 less annually. If he could transferthe amounts invested in the 5% and 7% bonds to the 10% bond, hewill earn $2200 more annually. Develop a system of linear equations todetermine the total amount of money invested in all the bonds.

1.2.47: The cupola already contains 100 lbs of an alloy consistingof 20% Cu (copper), 5% Sn (tin), 50% Al (aluminium), and 25% Zn(zinc) by weight. Two other alloys: R1, R2 with % composition byweight of (Cu, Sn, Al, Zn) equal to (10%, 10%, 60%, 20%), (10%, 20%,40%, 30%) respectively are available to add to the cupola; as also pureCu and Zn. Formulate a system of linear equations to determine how

48 Ch. 1. Linear eqs.

much R1, R2, Cu, Zn to add to the cupola to make a batch of 400 lbswith % composition by weight of (Cu, Sn, Al, Zn) equal to (16.25%,8.75%, 47.5%, 27.5%).

1.2.48: 4 lbs of coffee, 10 lbs of sugar and 2 lbs of butter in 2000cost $41. By 2001 coffee is 10% more expensive, sugar is 20% moreexpensive, and butter is 25% more expensive; and the same order costs$43.64. Also, in 2000, 10 lbs each of coffee, sugar, and butter togethercost $95. Formulate a system of linear equations to determine the 2001prices of coffee, sugar, and butter.

1.2.49: Problems dealing with speeds of vehicles: (Adoptedfrom M. R. Spiegel, Theory and Problems of College Algebra, Schaum’sOutline Series, McGraw-Hill, 1956). In these problems assume thateach vehicle always travels at a constant speed, but the speeds of dif-ferent vehicles may be different. Formulate a system of linear equationsto determine the speeds of the vehicles discussed from the informationgiven.

(i): Two race cars racing around a circular mile track starting fromthe same point at the same time, pass each other every 18 seconds whentraveling in opposite directions; and every 90 seconds when travelingin the same direction.

(ii): A paasenger at the front of a train A notices that he passesthe complete length of 330 feet of train B in 33 seconds when travelingin the same direction as B, and in 3 seconds when traveling in oppositedirections.

1.2.50: (Adopted from the issues of Mathematics Teacher).

(i): In a school each student earns between 0 to 100 points on eachtest. A student has already taken n + 2 tests.

At the end of the nth test his average score per test was m.

On the (n + 1)th test he got a grade of 98 and his average pointsper test became m + 1.

On the last test he got a grade of 70 and his average points per test

1.2 Example applications 49

became m − 1.

Develop a system of linear equations to determine the number oftests taken so far, and the average points per test at this stage.

(ii): When a pile of dimes is distributed equally into 100 red boxes,50 dimes were left over. When the same pile was distributed equallyinto 110 blue boxes none was left over, but each blue box contained5 fewer dimes than the red ones before. Develop a system of linearequations to determine the number of dimes in the pile.

(iii): A train traveling at constant speed takes 20 seconds from thetime it first enters a 300 meter long tunnel until it completely emergesfrom the tunnel at the other end. A stationary ceiling light in thetunnel was directly above the train for 10 seconds. Find the length ofthe train and its traveling speed using a linear equation model.

1.2.51: Three boys have some $ cash in their pockets. If each ofthem put $2 more in their pocket, boy 1 will have twice the amountof boys 2, 3 combined. If boy 1 transfers $6 from his pocket to thepockets of each of boys 2, 3; he will have half as much as they bothhave. If boy 3 transfers $1 from his pocket to the pockets of each ofthe others, he will have one-tenth of the money in all the pockets. If $2is taken away from the pocket of each boy, boy 1 will have 5 times asmuch money as the other two combined. Formulate a system of linearequations to determine how much money each boy posesses.

1.2.52: To encourage energy conservation, California has adoptedthe following electricity rate structure: each household consuming be-tween 0 to 500 kwh in a month will pay at the rate of a cents/kwh. Ifthe consumption in a month is between 500 kwh to 1500 kwh, the first500 kwh is charged at a cents/kwh, and any amount beyond that atthe greater rate of b cents/kwh. If the consumption in a month exceeds1500 kwh the first 1500 kwh are charged as mentioned before, and anyamount above 1500 kwh is charged at the highest rate of cents c/kwh.Given that the monthly electricity bill when the consumption is 300,600, 1600 kwh; is $30, $65, $220 respectively; use a linear equation

50 Ch. 1. Linear eqs.

2 3

1

54 6

(b)

2

3

1 5

4

7

6

(a)

Figure 1.5:

model to find the various rates.

1.2.53: Problems based on network diagrams: (i): Given be-low are several network diagrams, each consisting of nodes representedby little circles, and edges joining some pairs of nodes. Each diagramleads to a separate exercise, in each do the following. For each nodej, associate a decision variable xj . You are required to find values forthe variables so that the variables along any n-ball straight path is bmentioned there. Write these conditions on the variables as a systemof linear equations. If possible try to find a solution to this systemin which the values of the variables form a permutation of the nodenumbers.

For (a) n = 3 and b = 12.For (b) n = 3 and do separately for b = 9, 10, 11, 12 respectively.For (c) n = 4 and b = 34.(ii): Same as in (i) with n = 4 for the pentagram in figure (d), but

here b is not given; all you are required to do is to find the values for

1.2 Example applications 51

2

3

1

5

4

6

10

9 8

7(d)

2

3

1

54

16

6

151413

1211

10

9

8

7

(c)

Figure 1.6:

the variables so that the sums of all the variables along every 4-nodestraight path is the same. Write this as a system of linear equations.

(iii): Magic circles: (Adopted from J. S. Madachy, Madachy’sMathematical Recreations) Associate a variable xj with point j in fig-ure (e). Find the values for the variables such that the sum over allintersections is the same constant for all the circles.

1.2.54: Finding new girlfriend’s house number: While think-ing fondly about the four-digit house number of his new girlfriendRama, Venu discovered some of its amazing properties. For exam-ple, to divide it by 7, you only have to delete its second digit from left.The sum of its two rightmost digits is five times the sum of its twoleftmost digits. To multiply by three, all you have to do is to add 21 tothe number consisting of its two leftmost digits. If you interchange thetwo rightmost digits, its value goes down by 4.29%. Develop a systemof linear equations to determine Rama’s house number.

1.2.55: Is this a linear system?: On returning from the depart-ment store John explained to his wife Mary that the prices of one typeof shirts and one type of slacks that he liked were integer numbers in$. He purchased as many shirts as the sum of the prices of a shirt anda slack; and as many slacks as the sum of the price of a shirt and

52 Ch. 1. Linear eqs.

3

1

54 6

2

(e)

Figure 1.7:

one-half the price of a slack. He purchased 50 items in all, and spent$700. Develop a system of equations to determine how many shirts andslacks he purchased. Is this a system of linear equations? Explain.

1.2.56: Squashes galore!: On a hot summer day they were sellingyellow, black, white, and big-8 summer squashes at 20, 25, 30, 35 centseach respectively in a Farmer’s market. A restaurant owner purchasedall four types and paid exactly the same total amount for each type,getting 319 squashes in all. Develop a system of linear equations todetermine how many of each type she purchased.

1.2.57: Problem from ancient Chinese text Chiu-chang Suan-shu (Nine Chapters on the Mathematical Art): The yield fromone sheaf of inferior grain, two sheaves of medium grain, and threesheaves of superior grain is 39 tou. The yield from one sheaf of inferiorgrain, three sheaves of medium grain, and two sheaves of superior grainis 34 tou. The yield from three sheaf of inferior grain, two sheaves of

1.2 Example applications 53

medium grain, and one sheaf of superior grain is 26 tou. What is theyield of inferior, medium, and superior grains? Formulate the systemof linear equations to determine these. (This Chinese text is estimatedto be over 2000 years old.)

1.2.58: A farmer grows hogs, geese, sheep, and ostrichs on his smallfarm. The number of birds in his farm is 50 plus two times the numberof animals. His birds and animals together have a total of 350 headsand 900 legs. The number of sheep and ostrichs is 10 less than halfthe number of geese and hogs. Develop a system of linear equations todetermine how many of each type he has on his farm.

1.2.59: Sita purchased a total of 90 bulbs consisting of tulip, daf-fodil, and gladiola bulbs to plant in her garden. For each type shepurchased as many bulbs as the price per bulb of that type in cents,and paid a total of $29. She purchased twice as many gladiola bulbs asdaffodil bulbs. Develop a system of equations to determine the priceand the number of each type of bulb that she purchased. Is this asystem of linear equations? Explain.

1.2.60: Each year Rabi purchases individual gifts for his sister andher two children, a niece and a nephew.

Last year he spent a total of $240 for these gifts. That year hissister’s gift and nephew’s gift together cost two times his niece’s gift.His nephew’s gift plus three times his niece’s gift cost three times hissister’s gift.

This year his sister’s gift will cost twice as much as his niece’s giftlast year. His niece’s gift will be twice as expensive as his nephew’sgift last year. And his nephew’s gift will cost only half as much as hissister’s gift last year.

Develop a system of linear equations to determine his total giftexpense this year.

1.2.61: Cities 1, 2, 3 form a triangle.

From 1 to 3 via 2 is three times the distance by direct route.

54 Ch. 1. Linear eqs.

From 2 to 1 via 3 is twice the distance by direct route.

From 3 to 2 via 1 is 20 miles more than the distance bydirect route.

Formulate a system of linear equations to determine the lengths ofdirect routes between pairs of cities.

1.2.62: Three students S1, S2, S3 took a test which had 50 ques-tions, some easy, others tough. S1 answered three times as many ques-tions as the sum that S2, S3 missed. S2, S3 answered the same numberof questions, each of them missing a fourth of what S1 missed. Formu-late a system of linear equations to determine how many each of themanswered.

1.2.63: A middle school teaches grades 7, 8, 9, 10 and has a totalenrollment of 330 students. They receive annual funding at the rate of$4000 per 7th grade student, $4300 per 8th grade student, $4500 per9th garde student, and $4700 per 10th grade student; and received atotal funding of $1,457,000 for the year. 30% of students in 7th grade,25% in 8th grade, and 20% in grades 9 and 10 take music classes; thisyear they have a total of 76 students enrolled in music classes. Also, thisyear they have a total of 31 students enrolled in AP courses, consistingof 15% of students of 8th grade and 10% of students in each of grades9 and 10. Formulate a system of linear equations to determine thenumber of students in each grade.

1.2.64: The automobile route to a scenic mountainous area is 50miles each way with only 10 miles of it level road each way, the restis either uphill or downhill. A person maintained an average speed of20 miles/hour uphill, 50 miles/hour downhill and 40 miles/hour on thelevel road on a car trip to this region. The return trip took 44.44%more than the riding time to get there (not counting stops). Formulatea system of linear equations to determine the total driving time for thetrip.

1.2.65: A family purchased cantaloupes (50 cents each), honeydews

1.2 Example applications 55

($1 each), oranges (20 cents each), grape fruit (15 cents each), andmangoes (75 cents each) at a road side fruit stall on their way backfrom vacation, and paid $26.30 for all of them, obtaining 80 fruit inall. They purchased 6 times as many citrus fruit as melons; as manymangoes as one-fourth of oranges and cantaloupes put together; andas many oranges as 20% more than the honeydews and grapefruit puttogether. Formulate a syatem of linear equations to determine howmany fruit of each type they purchased.

1.2.66: ((Adopted from J. A. H. Hunter Entertaining MathematicalTeasers and How to Solve Them, Dover, NY, 1983).

(a): A kid’s mother organized a birthday party for him. She pur-chased enough apples at 50 cents/apple to give two to each child invited,but a quarter of the invited kids did not show up. So she distributedtwo apples to each of the 16 boys who showed up, and then had suffi-cient apples to give three to each of the girls who showed up. Formulatea system of linear equations to determine how much money she spenton the apples.

(b): A dealer of speciality teas imports pure oolong tea at $3.29/lb,and lapsang souchong tea at $2.38/lb from China; blends them andsells the blend at $4.48/lb which gives him a 60% markup. Formulatea system of linear equations to determine the proportions in the blend.

(c): In an office there are both men and women working as clerks.If they had twice as many girls’ or three times as many men, they wouldhave 30 clerks. Formulate a system of linear equations to determinehow many clerks they have.

1.2.67: In an ordered list of six numbers every one (except the endones) is equal to the sum of the two adjacent numbers in the list. Thesum of all the numbers is the sum of the first three. The third numberis −6. Formulate a system of linear equations to find all the numbersin the list.

56 Ch. 1. Linear eqs.

1.2.68: There are 60 girls in an undergraduate class. Each is eithera blonde or brunette, and has either blue or brown eyes. Among them40 are blondes and 22 have blue eyes. Formulate a system of linearequations to determine the number of girls in this class in each of thefour hair and eye color combinations.

1.2.69: A four digit integer number has the property that mul-tiplying it by 9 yields the four digit number obtained by writing thedigits in the original number in reverse order. Formulate a system oflinear equations to determine the original number.

1.2.70: A school principal noticed that the number of students inher school has the property that when it is divided by k, a reminder ofk−1 is left for k = 2, 3, . . ., 10. Formulate a system of linear equationsto find the number of students in the school.

1.2.71: Three numbers are in arithmetic progression, three othernumbers are in geometric progression. Adding coorresponding termsof these two progressions we obtain 9, 16, and 28 respectively. Addingall three terms of the arithmetic progression we obtain 18. Formulatea system of equations to find the terms of both progressions. Is it asystem of linear equations?

1.2.72: (Adopted from G. Polya, J. Kilpatrick, The Stanford Math-ematics Problem Book With Hints and Solutions, Teachers CollegePress, Columbia University, NY, 1974) (i): Unlimited numbers ofcoins- cents, nickels (5 cent coins), dimes (10 cent coins), quarters (25cent coins), and half-dollars are available. It is required to determinein how many different ways you can assemble a collection of these coinsso that the total value of this collection is n cents for a positive integern, say, for n varying between 0 to 50. Formulate a system of linearequations to solve this problem, and show that for n = 50, the numberof different ways is 50.

(ii): 10 people are sitting around a round table. $20 is to bedistributed among them satisfying the rule that each person receives

1.2 Example applications 57

exactly one-half of the sum of what his (her) two neighbors receive.Formulate a system of linear equations to determine how to distributethe money. Using the special properties of this system (or the resultsyou will learn in Chapters up to 4) argue that there is only one way todistribute the money here.

1.2.73: A magical card trick: As a child, I was enthralled by acard trick performed by the great magician Sirkar in India. It involvesa deck of playing cards. The deck has 52 cards divided into 4 suitsof 13 cards each. As they call them in India, these are kalaavaru andispeetu (the black suits), and aatiinu and diamond (the red suits).

Sirkar gave me the deck and said “shuffle it as much as you want,then divide it into two piles of 26 cards each and place the piles onthe table”. He then covered the two piles with his hands and utteredthe magical chant “Om, bhurbhuvasvuh” three times saying that he isusing his power of magic to make sure that the number of black cardsin the first pile becomes equal to the number of red cards in the secondpile. He then asked me to count and verify the truth of his statement,which I did.

Define four decision variables; the number of black, red cards in piles1, 2. Formulate the system of linear equations that these variables haveto satisfy. Using this system, unravel the magic in this trick.

1.2.74: The sports department in a middleschool offers classes inbicyling, swimmimg, skating, and tennis. A class in this school has 25pupils. In this class 17 pupils are enrolled in the bicycling class, 13 inthe swimming class, 8 in the skating class, and 6 in the tennis class.None of the pupils enrolled in any of bicycling, swimming, or skatingclasses is enrolled in the tennis class. It is required to determine howmany pupils of this class are enrolled in various combinations of sportsclasses. Formulate this using a system of linear equations.

1.2.75: People can be divided into four distinct blood groups:O, A, B, and AB. Blood transfusion with donated blood will be with-out danger to the receiver only under the following criteria: Every onecan accept blood for transfusion from a person of the O blood group,

58 Ch. 1. Linear eqs.

or from a person in his/her own blood group. Also, a person in theAB group can accept blood from any person.

Among the members in a club willing to donate blood, there are47 (64) [67] {100} members from whom an O (A) [B] {AB} can ac-cept blood for transfusion. Formulate a system of linear equations todetermine how many of these members belong to the various bloodgroups.

1.2.76: At a club meeting they served: coffee, a chacolate drink(for drinks); pie, a cacolate dessert (for dessert); and a nonvegetariandish with meat, a vegetarian dish (for main dish); and members selectwhat they like. There was a bill for each item separately for the wholeclub. The drink items together cost $35. The dessert items togethercost $50. The cacolate items together cost $55. The main dishes together cost $110. All the items which have no meat, together cost $135.The tip which is calculated at 10% of the total cost came to $19.50.Formulate a system of linear equations to determine the cost of eachitem separately.

1.2.77: Newspaper buying habits of households in Ann Ar-bor: In any quarter of the year, the households in Ann Arbor can beclassified into four classes: C1 = those who do not subscribe to anydaily newspaper, C2 = those who subscribe to the Detroit Free Pressonly, C3 = those who subscribe to Ann Arbor News only, and C4 =those who subscribe to both the Detroit Free Press and the Ann ArborNews.

Consumer research has shown that at the end of every quarter,households move from the class in which they currently belong, toanother class, in the following proportions.

Table: Fraction of households belonging to the class defined by therow, who stay in same class or move to class defined by column, at endof a quarter.

1.2 Example applications 59

C1 C2 C3 C4

C1 0.6 0.2 0.1 0.1C2 0.5 0.3 0.2 0C3 0.5 0.2 0.1 0.2C4 0.2 0.7 0.1 0

These transition fractions have remained very stable at these valuesfor a very long time.

Let p1, p2, p3, p4 denote the proportion of households in Ann Arborbelonging to the classes C1 to C4 respectively, in a quarter. Typicallywe would except the values of p1 to p4 to change from one quarter to thenext, but surprisingly their values have remained essentially unchangedfor a long time in spite of the transitions of households taking placefrom one quarter to the next mentioned above. Use this informationto construct a system of linear equations satisfied by these proportionsp1 to p4.

1.2.78: Figure 1.7 shows a rectangle with sides a, b cut into squareswith sides x1 to x6. Write the system of linear equations that x1, . . . , x7

have to satisfy.

a

b

x1

x2

x5

x6x 7

3x

4x

Figure 1.8:

60 Ch. 1. Linear eqs.

1.2.79: An avid gardener has created nine mounds for plantingdifferent types of zucchini plants in his garden. He planted green zuc-chini seeds on some mounds, yellow zucchini seeds on some, and whitezucchini seeds on the rest. On each mound as many plants sproutedas the number of mounds allotted to zucchini of that color; and allthe sprouted plants started growing vigorously. On a certian morningeach zucchini plant had as many squashes as the total number of plantsof that zucchini color, and he counted a total of 353 squashes in thegarden. He noticed that only half the yellow squashes, one third of thewhite squashes, and one eighth of the green squashes are of harvestablesize and he harvested all of them (67 total) that morning.

Develop a system of equations to determine how many mounds heallocated to zucchinis of each color. Is this a system of linear equations?

1.2.80: A small farmer grows three types of chicken, each in itsown pen. He has noticed that the chicken in pens 1, 2, 3 lay 24, 22, 19eggs per chicken per month; and he has collected 5110 eggs in all onemonth.

All his chicken are either white or brown, and the proportions ofbrown chicken in pens 1, 2, 3 are 0.5, 0.25, 0.4 respectively. He counteda total of 90 brown chickens in all the pens together.

He estimates that the chicken in pens 1, 2, 3 consume on an average4, 3, 2 units of feed per chicken per day respectively. He has been using740 units of feed daily.

Develop a system of linear equations to determine how many chick-ens are housed in each pen.

1.2.81: On a holiday, players 1, 2, 3, 4 played singles tennis all day.

The total number of games in which player 1, 2, 3, 4 participatedwas 13, 10, 15, 14 respectively. The total number of games in whichplayer 1, 2, 3, 4 was not one of the players was 13, 16, 11, 12 respec-tively.

Develop a system of linear equations to determine how many gameswere played by each pair of players.

1.2.82: Problems adopted from “Ganitamlo podupu kad-

1.3 Fundamental Computational Tool 61

halu” in Telugu by CSRC Murthy: (i): A person has loans fromthree different banks. One month he saved $1000 which he used topayoff part of these loans. He paid Bank 2 $100 more than Bank 1,and he paid Bank 3 as much as the sum he paid to Banks 1, 2. Developa system of linear equations to determine how much he paid each bank.

(ii): A man married a woman who is four years younger than him.They have two children, a boy and his younger sister. On a certainday, the father’s age is 1 less than three times the age of the boy; andthe mother’s age is 1 more than three times the daughter’s age. Also,the sum of the ages of the children on that day is 24.

Develop a system of linear equations to determine the ages of mem-bers of this family.

(iii): At a party people had hot coffee or tea, or cold OJ (orangejuice) for drinks, a total of 15 drinks. The number of coffees orderedwas five more than the sum of teas and OJ ordered. Also, the numberof hot drinks ordered was four times the number of cold drinks ordered.Develop a system of linear equations to determine how many drinks ofeach type were ordered.

1.3 Solving a System of Linear Equations

Is a Fundamental Computational Tool

In the previous section, we have seen a variety of applications in which asystem of linear equations to be solved appears directly. Actually, solv-ing a system of linear equations is a very fundamental computationaltool used in almost all areas of scientific computation. Many problemswhich are not themselves systems of linear equations are solved either

• by approximating them through a system of linear equations, or

• by algorithms using iterative steps, each of which involves solvingone or more systems of linear equations.

Here we provide an example of the first type of problem in dif-ferential equations, which are equations that involve functions and

62 Ch. 1. Linear eqs.

their derivatives. They appear in many studies in engineering, physics,chemistry, and biology. Most differential equations appearing in theseareas are very hard to solve through exact mathematical techniques, sothey are usually handled numerically by approximating them througha system of linear equations. For a specific example, consider a realvalued differentiable function f(t) of one variable t, defined over theinterval a ≤ t ≤ b. We denote this interval by [a, b]. Let f ′(t), f ′′(t) de-note the first and second derivatives of f(t). Suppose we do not knowthe function f(t) explicitly, but know that it and its derivatives satisfythe following equations

u(t)f ′′(t) + v(t)f ′(t) + w(t)f(t) = z(t) for all t in [a, b]

f(a) = α, f(b) = β

where u(t), v(t), w(t), z(t) are known real valued functions, and α, β aregiven constants.

Since the values of the unknown function f(t) at the two boundarypoints a, b of the interval [a, b] are given, the problem of finding thefunction f(t) satisfying the above equations is known as a two-pointboundary value problem. It is an example of a differential equationthat appears in many applications.

To handle such problems, numerical methods are employed to findapproximate values of f(t) at a set of discrete points inside the inter-val [a, b]. The approximating technique is called discretization, andproceeds as follows. Divide the interval [a, b] into n + 1 subintervalsof equal length h = (b − a)/(n + 1). h is called the step size. Theboundary points of the subintervals, called the grid points are

t0 = a, t1 = a+h, t2 = a+2h, . . . tn = a+nh, tn+1 = a+(n+1)h = b.

The aim now is to find the values of f(t) at the grid points, i.e.,f(t1), . . . , f(tn), which are the unknowns. Discretization converts thedifferential equations corresponding to the grid points into a system oflinear equations, using approximations for the derivatives at the gridpoints in terms of the unknowns. Derivative approximations calledcentral difference approximations, given below for i = 1 to n, areused.

1.4 Tableau Form 63

f ′(ti) ≈ (f(ti+1) − f(ti−1))/2h

f ′′(ti) ≈ (f(ti−1) − 2f(ti) + f(ti+1))/h2

Substituting these expressions for f ′(ti), f ′′(ti) in

u(ti)f′′(ti) + v(ti)f

′(ti) + w(ti)f(ti) = z(ti)

for i = 1 to n, we get a system of n linear equations in the n un-knowns f(t1), . . . , f(tn). Solving this system of linear equations givesapproximations to the values of the function f(t) at the grid points.Once approximations to f(t1), . . . , f(tn) are obtained, an approxima-tion to the value of f(t) at any point t which is not a grid point, canbe obtained using standard interpolation techniques.

For better derivative approximations, we need to make the stepsize h small; this makes n large and leads to a large system of linearequations.

1.4 The Detached Coefficient Tableau Form

The detached coefficient tableau form is a neat way of organizingthe data in a system of linear equations so that:

• the coefficients of all the variables in each constraint appear in arow of the tableau,

• the coefficients of each variable in all the constraints appear in acolumn of the tableau.

To represent a system of m linear equations in n variables (say,x1, . . . , xn) in detached coefficient tableau form:

• Set up an m × (n + 1) two dimensional array.

• Arrange the variables in the system in some order, say x1 to xn

in the natural order of increasing subscripts, and allocate one

64 Ch. 1. Linear eqs.

variable to each of columns 1 to n in that order, and record thisby writing that variable in that column in a top row of the array.Then allocate the last column on the right, column n + 1, to theRHS (right hand side) constants.

• Take the constraints in any order, say constraint 1 to constraintm in natural order, and record the coefficients of the variablesand the RHS constant in that constraint, in the order selectedabove, in a row of the tableau, one after the other.

Example 1:

Consider the 5 constraint, 4 variable system of linear equationsderived in Example 1, Section 1.2, which is reproduced below.

5x1 + 7x2 + 2x3 + x4 = 4.43

3x1 + 6x2 + x3 + 2x4 = 3.22

4x1 + 5x2 + 3x3 + x4 = 3.89

88x1 + 82x2 + 94x3 + 96x4 = 88.46

x1 + x2 + x3 + x4 = 1

By taking the variables and constraints in natural order, the de-tached coefficient tableau form for this system is:

x1 x2 x3 x4

5 7 2 1 4.433 6 1 2 3.224 5 3 1 3.8988 72 94 96 88.461 1 1 1 1

Example 2:

1.4 Tableau Form 65

Consider the following system of linear equations.

3x4 − 7x2 + 8x1 = −13

−x1 + 2x3 = −2

x2 − 13x5 − 7x4 + 4x3 = 18

−2x3 + 7x1 = 12

20x5 − 19x3 = −1

Recording the variables and constraints in their natural order, thissystem in detached coefficient form is:

x1 x2 x3 x4 x5

8 −7 0 3 0 −13−1 0 2 0 0 −2

0 1 4 −7 −13 187 0 −2 0 0 120 0 −19 0 20 −1

Discussion: Rows Correspond to Con-straints, Columns to Variables

Notice that in the detached coefficient tableau form, the “=” symbolfor each constraint is not explicitly recorded, but is implicitly under-stood.

Clearly, the ith constraint in the system can be read directly fromthe ith row in the tableau by multiplying the coefficients in this rowby the corresponding variables and forming the equation. For example,from the fourth row of the above tableau we read the equation 7x1−2x3

= 12, which is the same as the fourth equation in the system in thisexample. Hence, rows in the tableau correspond to constraints in thesystem. That’s why people sometimes refer to constraints in a systemas “rows” in informal conversations.

Similarly, the column in the tableau under a variable is the columnvector of coefficients of that variable in the various constraints of the

66 Ch. 1. Linear eqs.

system. Thus each column in the tableau corresponds to a variable inthe system.

Let the symbol A denote the detached coefficient tableau for a sys-tem of constraints, in which the variables are listed in the order x1 toxn. The ith row in the tableau (without the RHS constant in this row)forms a row vector which we denote by Ai. (the dot following the i inthe subscript indicates that it is the ith row vector in the tableau A).Ai. is the row vector of the coefficients of the variables in the order x1

to xn in the ith constraint of the system.

Since each entry in Ai. corresponds to a specific variable, if wechange the order of the entries in Ai., it may lead to a different con-straint. For example, in the above tableau, A4. = (7, 0,−2, 0, 0). If weinterchange the first two entries here, we get the row vector (0, 7,−2, 0, 0),which leads to the constraint 7x2 −2x3 = 12, not the same as the fourthconstraint in the system. That’s why the order of entries in a vector isvery important.

The jth column in the tableau forms a column vector which wedenote by A.j (the dot before the j in the subscript indicates it isthe jth column vector in the tableau A), it is the column vector ofcoefficients of the variable xj in the various constraints of the system.

The Column Vector of Every Variable in the TableauMust Contain Atleast One Nonzero Entry

A variable appears in a system of linear equations only if it has anonzero coefficient in at least one equation in the system. Hence animportant property of the detached coefficient form is that the columnvector corresponding to every variable in the system must have at leastone nonzero entry.

Of course the rightmost column in the tableau, the column of theRHS constants, may consist of all zero entries.

Exercises

1.4.1 Write the following systems of equations in detached coeffi-cient form.

1.4 Tableau Form 67

(i) −2x6 − 3x1 + x5 = −12x1 − 7x4 + 2x3 = 17−5x2 + 8x3 = 13

(ii) Here variables have two subscripts (x11 etc.)x11 + x12 + x13 = 800x21 + x22 + x23 = 300x11 + x21 = 400x12 + x22 = 500x13 + x23 = 200

(iii) −3x4 + 7x2 = −10 + 6x5 − 17x1

8x6 − 3x1 = 15 − 13x2 + 12x4 − 9x3

−22 + 7x5 − 4x2 + 14x6 = 33 + 2x1 + 6x3

88x2 + 65x7 − 5x2 = 29 − 4x3 − 3x4

x5 − 7x3 = −27 + 6x7 − 8x5

39 + 4x7 − 13x4 + 11x2 = 00 = 13x7 − 5x3 + 25x1 − 9x5

(iv) x1 + x2 = 110x1 = x3 + x4

x2 = x5 + x8

x6 = x3 + x5

x7 = x4 + x8

x7 + x6 = 110

1.4.2 Write all the constraints in the system corresponding to thefollowing detached coefficient forms:

x1 x2 x3 x4 x5 x6

−5 0 0 17 −13 8 −230 0 21 −13 0 0 37

−97 −15 0 0 0 33 −560 11 0 −2 0 8 07 0 −6 0 26 0 29

68 Ch. 1. Linear eqs.

x1 x2 x3 x4 x5 x6 x7 x8 x9

1 1 1 0 0 0 0 0 0 10 0 0 1 1 1 0 0 0 10 0 0 0 0 0 1 1 1 11 0 0 1 0 0 1 0 0 10 1 0 0 1 0 0 1 0 10 0 1 0 0 1 0 0 1 1

1.5 Operations on Vectors, the Zero and

the Unit Vectors, Subspaces and Co-

ordinate Subspaces of Rn

Transposition of Vectors

A row vector can be transformed into a column vector by writingthe entries in it one below the other instead of side by side and viceversa. This operation is called transposition. Thus the transpose ofa row vector is a column vector and vice versa. The transpose of avector u is usually denoted by the symbol uT . For example:

(3, 8, 4)T =

384

;

7−1

03

T

= (7,−1, 0, 3)

In particular, we will find it very convenient to write a column

vector, for example

384

, as (3, 8, 4)T to conserve vertical space in

the text.

Dot (Inner) Product of Two Vectors

Now consider the detached coefficient tableau for the system inExample 2, Section 1.4, which is reproduced below.

1.5. Operations on Vectors 69

x1 x2 x3 x4 x5

8 −7 0 3 0 −13−1 0 2 0 0 −2

0 1 4 −7 −13 187 0 −2 0 0 120 0 −19 0 20 −1

Let us denote by x = (x1, x2, x3, x4, x5)T the vector of variables in

this system written as a column vector. Also, let b = (b1, b2, b3, b4, b5)T

= (−13,−2, 18, 12, 1)T denote the column vector of RHS constants inthis tableau.

Denoting this tableau by A, the first row vector of coefficients in it,A1. = (8, −7, 0, 3, 0), and it corresponds to the first constraint in thesystem

8x1 − 7x2 + 0x3 + 3x4 + 0x5 = b1 = −13

The quantity on the left of this constraint, 8x1−7x2+0x3+3x4+0x5

is called the dot product or inner product of the row vector A1. withthe column vector x. It is written as A1.x in this specific order. Usingthis notation, we can write the ith constraint in the system as

Ai.x = bi

The dot product or inner product of two vectors is defined onlyif both the vectors have the same dimension (i.e., the same numberof entries). Then it is the sum of the products of the correspondingentries. Thus if u ∈ Rn with entries u1, . . . , un in that order, and v ∈ Rn

with entries v1, . . . , vn in that order, then the dot (inner) product of uand v is the number u1v1 + . . . + unvn. It is denoted in mathematicalliterature in two ways:

• as uv where the left vector u is written as a row vector, andthe right vector v is written as a column vector, or

• as < u, v > or < v, u > where either of u or v may be row orcolumn vectors.

70 Ch. 1. Linear eqs.

We caution the reader that the product written as uv whereu ∈ Rn is a column vector and v ∈ Rn is a row vector has a totallydifferent meaning to be explained in the next chapter.

Scalar Multiplication of Vectors

Let u be a (row or column) vector in Rn with entries u1, . . . , un. Letα be any real number. Multiplying the vector u by α is called scalarmultiplication of u by the scalar α. It has the effect of multiplyingeach entry in u by α. Thus:

α(u1, . . . , un) = (αu1, . . . , αun)

α

u1...

un

=

αu1...

αun

For example:

10(3, 0,−4) = (30, 0,−40)

−15

30−4

=

−45060

0(3, 0,−4) = (0, 0, 0)

Addition of Vectors, Linear Combination of Vec-tors

Let u, v be two vectors. The operations

u + v or u − v

1.5. Operations on Vectors 71

are only defined if either both u and v are row vectors of the samedimension, or both are column vectors of the same dimension. Theresult is a vector of the same type obtained by performing the operationcomponent by component. Thus:

(u1, . . . , un) + (v1, . . . , vn) = (u1 + v1, . . . , un + vn)

(u1, . . . , un) − (v1, . . . , vn) = (u1 − v1, . . . , un − vn)

u1...

un

+

v1...

vn

=

u1 + v1...

un + vn

,

u1...

un

−

v1...

vn

=

u1 − v1...

un − vn

Thus the operation

(u1, . . . , um) + (v1, . . . , vn)

is not defined if m �= n. And the operation

(u1, . . . , un) +

v1...

vn

is not defined.In general, when α, β are scalars, the general linear combination of

two vectors is defined by:

α(u1, . . . , un) + β(v1, . . . , vn) = (αu1 + βv1, . . . , αun + βvn)

α

u1...

un

+ β

v1...vn

=

αu1 + βv1...

αun + βvn

Examples:

72 Ch. 1. Linear eqs.

−3708

+

8−17

3−20

=

5−10

3−12

,

−3708

−

8−17−328

=

−11243

−12

2(4,−6, 8, 2) − 10(6,−5, 0, 8) = (−52, 38, 16,−76)

Some Properties of the Addition Operation onVectors

In summing two vectors u and v, the order in which you write themin the sum is immaterial, i.e., u + v = v + u

When you want to find the sum of three vectors u, v, w, youcan either add w to the sum of u, v; or u to the sum ofv, w; both will give the same result; i.e., (u + v) + w = u + (v + w).This property is known as the associative law of addition.

Also, for all vectors u, we have u + 0 = u where 0 isthe zero vector of the same type as u and in the same space.

In the same way, for r = 1 to k, let

xr =

xr1...

xrn

be a column vector in Rn. The general linear combination of these kcolumn vectors {x1, . . . , xk} with scalar multipliers α1, . . . , αk, each ofwhich can be an arbitrary real number, is

k∑r=1

αrxr = α1x

1 + . . . + αkxk =

∑kr=1 αrx

r1∑k

r=1 αrxr2

...∑kr=1 αrx

rn

and the set of linear combinations of a set of row vectors in Rn isdefined in a similar way.

1.5. Op erations on Vectors 73

The set of all possible linear combinations of the set of vectors{x1, . . . , xk}, {α1x

1 + . . . + αkxk : where α1, . . . , αk each take arbitrary

real values} is called the linear hull or linear span of {x1, . . . , xk}.For example, let

x1 =

11

−1

, x2 =

1−1

1

, x3 =

−111

Then

3x1 + 4x2 − 5x3 =

3 + 4 − (−5)3 + 4(−1) + 53(−1) + 4 + 5

=

1246

and so the vector (12, 4, 6)T is contained in the linear hull of {x1, x2, x3 }.Given a set of vectors {x1, . . . , xk} (either all row vectors, or all

column vectors) from Rn, the linear combination α1x1 + . . . + αkx

k iscalled an affine combination if the scalar multipliers in it satisfy thecondition

α1 + . . . + αk = 1.

The set of all possible affine combinations of {x1, . . . , xk} is calledthe affine hull or affine span of {x1, . . . , xk}.

For the set of vectors {x1, x2, x3 } from R 3 given above, the linear

combination 3x1 + 4x2 − 5x3 is not an affine combination because 3 +4 − 5 = 2 is different from 1. But

−8x1 − 4x2 + 13x3 =

−8 − 4 + 13(−1)−8 − 4(−1) + 13−8(−1) − 4 + 13

=

−259

17

is an affine combination of {x1, x2, x3 } because −8−4+13 = 1. Hence(−25, 9, 17)T is in the affine hull of {x1, x2, x3 }.

Exercises

1.5.1 Find the following if they are defined:

74 Ch. 1. Linear eqs.

(6,−2, 8) + (0, 10)

(3, 3, 3) −

6811

15(1,−1, 1, 0, 1)− 21(0, 1, 5, 2, 3) + 3(0, 0, 0, 0, 1)

<

3−84

, (1, 2, 3) >

(1, 2, 3)

2−34

< (1, 2, 3, 0),

8−173

>

The Zero Vector and the Unit Vectors

We now define some important special vectors in Rn.

The vector in Rn in which all the entries are zero is called the zerovector and is itself denoted by the symbol “0”, whether it is a row ora column vector. From context, one can determine whether the symbol0 is denoting the row or the column zero vectors.

A vector in Rn in which the ith entry is 1 and all the other entriesare 0 is called the ith unit vector in Rn. Thus there are n differentunit vectors in Rn. For example:

(0, 0, 1) is the third unit row vector in R3.

(1, 0, 0)T =

100

is first unit column vector in R3.

1.5. Op erations on Vectors 75

Caution: For n ≥ 2, the vector in Rn in which all the entries are1, for example (1, 1, 1) ∈ R3 is not a unit vector.

Spanning Property of the Unit Vectors

The set of n unit vectors in Rn have the special property that theirlinear hull is Rn itself, i.e., every vector in Rn is a linear combinationof the set of n unit vectors in Rn.

For instance, let n = 3. Consider the column vector (3,−69, 75)T .We have

3−69

75

= 3

100

− 69

010

+ 75

001

and hence (3,−69, 75)T is contained in the linear hull of the set of threeunit vectors in R3.

Similarly, let {I.1, . . . , I.n} denote the set of n unit column vectorsin Rn. Consider the general column vector x = (x1, . . . , xn)T ∈ Rn. Wehave

x =

x1...

xn

= x1I.1 + . . . + xnI.n

and hence x is in the linear hull of {I.1, . . . , I.n}.

Subspaces of Rn

A subset S of n-dimensional vectors (either all row vectors or all col-umn vectors) is said to be a subspace of the n-dimensional vectorspace Rn, if it satisfies the following property:

Subspace Property: for every pair of vectors x, y ∈ S, everylinear combination αx+βy for any real numbers α, β is also containedin the set.

Here are some examples of subspaces of Rn.

76 Ch. 1. Linear eqs.

• The singleton set {0} containing the 0-vector only, is a sub-space of dimension 0. This is the only subspace containing onlyone vector in it. All other subspaces contain an infinite numberof vectors.

• The j-th coordinate subspace of Rn. For any j = 1 to n, thisis the set of all vectors {x = (x1, . . . , xj−1, xj = 0, xj+1, . . . , xn) :in which the component xj = 0, while all the other componentsmay take any real values}. This subspace has dimension n − 1.

• Lower dimensional coordinate subspaces of Rn of dimen-sion n − r < n − 1. Select any subset T of r componentsamong x1, . . . , xn, where r ≥ 2. Then the set of all vectors{x = (x1, . . . , xn) : each component xj ∈ T is 0} is an (n − r)-dimensional coordinate subspace of Rn defined by the subset of rcomponents T . For example the set of vectors {x = (x1 = 0, x2 =0, . . . , xr = 0, xr+1, . . . , xn)} is the n− r-dimensional subspace inwhich all the components {x1, . . . , xr} are fixed equal to 0.

• Set of all linear combinations of a given set of vectors∆ ⊂ Rn is a subspace of Rn containing all the vectors in ∆.

1.6 Notation to Denote the General Sys-

tem of Linear Equations

We now describe the algebraic notation that we will use for denotinga general system of m linear equations in n variables. (Here, m =the number of equations; may be <, or =, or > n = the number ofvariables.)

Let x = (x1, . . . , xn)T denote the column vector of variables in thesystem.

Let aij denote the coefficient of xj in the ith constraint, for i = 1to m, j = 1 to n.

Let b = (b1, . . . , bm)T be the column vector of the RHS constants inthe constraints 1 to m. Then the system of constraints is:

1.6. General System 77

a11x1 + a12x2 + . . . + a1nxn = b1

a21x1 + a22x2 + . . . + a2nxn = b2

......

am1x1 + am2x2 + . . . + amnxn = bm

In detached coefficient tableau form, it is:

x1 . . . xj . . . xn RHS constantsa11 . . . a1j . . . a1n b1...

......

...ai1 . . . aij . . . ain bi...

......

...am1 . . . amj . . . amn bm

Denoting this tableau by A, in this system, the row vector of coef-ficients of the variables in the ith constraint is: Ai. = (ai1, . . . , ain), i= 1 to m.

The column vector corresponding to the variable xj in this tableauis: A.j = (a1j , . . . , amj)

T , j = 1 to n.

Denoting the System in Terms of the Rows of theTableau

So, the ith constraint in the system can be written as: Ai.x = bi,i = 1 to m, using our notation. So, the above system of equations canalso be written as:

Ai.x = bi i = 1 to m

Denoting the System in Terms of the ColumnVectors Corresponding to the Variables

The column vectors corresponding to the variables x1 to xn are A.1

to A.n. It can be verified that the left hand side of the general system of

78 Ch. 1. Linear eqs.

equations given above is the linear combination of the column vectorsA. 1, . . . , A.n with x1, . . . , xn as the coefficients. Thus this system ofequations can also be written as a vector equation as:

x1A. 1 + . . . + xjA.j + . . . + xnA.n = b

Thus the values of the variables x1 to xn in any solution of the sys-tem, are the coefficients in an expression of the RHS constants columnvector b as a linear combination of the column vectors correspondingto the variables, A. 1 to A.n in the system.

Exercises

1.6.1 Write the systems of constraints in the following detachedcoefficient tableaus as vector equations, expressing a column vector asa linear combination of other column vectors.

x1 x2 x3 x4 x5

0 −7 0 0 14 -36−2 0 12 3 −3 2213 17 −11 8 0 00 3 0 5 6 −12

x1 x2 x3 x4 x5

1 2 1 2 −1 112 1 2 1 −4 10

1.7 Constraint Functions, Linearity As-

sumptions, Linear and Affine Func-

tions

Consider the following general system of m linear equations in n vari-ables x = (x1, . . . , xn)T .

ai1x1 + . . . + ainxn = bi, i = 1, . . . , m

1.7. Constraint Functions, Linearity Assumptions 79

The left hand side expression in the ith constraint, ai1x1 + . . . +ainxn is a real valued function of the variables called the constraintfunction for this constraint. Denoting it by gi(x), the constraints inthis system can be written as

gi(x) = bi, i = 1, . . . , m

Hence each constraint in the system requires that its constraintfunction should have its value equal to the RHS constant in it.

As an example, consider the system of 5 equations in 4 variables inExample 1 of Section 1.2, which is reproduced below.

5x1 + 7x2 + 2x3 + x4 = 4.43

3x1 + 6x2 + x3 + 2x4 = 3.22

4x1 + 5x2 + 3x3 + x4 = 3.89

88x1 + 82x2 + 94x3 + 96x4 = 88.46

x1 + x2 + x3 + x4 = 1

The left hand side expression in the first equation, 5x1 + 7x2 +2x3 + x4 = g1(x) measures the percentage by weight of Al in themixture represented by the vector x, and this is required to be =4.43. Similarly, the left hand side expressions in equations 2, 3, 4 inthe system, g2(x) = 3x1 +6x2 +x3 +2x4, g3(x) = 4x1 +5x2 +3x3 +x4,g4(x) = 88x1+82x2+94x3+96x4, respectively measure the percentageby weight of Si, C, Fe respectively in the mixture represented bythe vector x. g5(x) = x1 + x2 + x3 + x4 from the fifth equation isthe sum of the proportions of SM-1 to SM-4 in the mixture, and unlessthis is equal to 1 the vector x cannot be a solution to the problem.

With these constraint functions defined as above, we can denotethis system in the form

g1(x)g2(x)g3(x)g4(x)g5(x)

=

4.433.223.8988.46

1

80 Ch. 1. Linear eqs.

In the same way, in any system of linear equations, the constraintfunction associated with any constraint measures some important quan-tity as a function of the variables, and the constraint requires it to beequal to the RHS constant in that constraint.

Linear Functions

Definition 1: A real valued function, h(x), of variables x =(x1, . . . , xn)T is said to be a linear function if it satisfies the fol-lowing two assumptions, which together are called the linearity as-sumptions.

Additivity or Separability assumption: This assumption re-quires that the function h(x) should be the sum of n functions,in which the jth function involves only one variable xj , for j = 1 ton; i.e.,

h(x) = h1(x1) + h2(x2) + . . . + hn(xn)

Proportionality assumption: A real valued function of a singlevariable, say xj , satisfies this assumption if the function is equal tocjxj for some constant cj.

The function h(x) of n variables x = (x1, . . . , xn)T whichsatisfies the additivity assumption as in the above equation, is said tosatisfy this proportionality assumption, if each of the functions hj(xj),j = 1 to n satisfy it. ��

Therefore, the real valued function h(x), where x = (x1, . . . , xn)T ,is a linear function iff there exist constants c1, . . . , cn such thath(x) = c1x1+. . .+cnxn for all x ∈ Rn. In this case c = (c1, . . . , cn)T

is called the coefficient vector of the variables in the linear functionh(x). Consider the following examples.

1.7. Constraint Functions, Linearity Assumptions 81

The function Commentsof x = (x1, . . . , x5)

T

3x2 − 7x5 A linear function with coefficient vec-tor (0, 3, 0, 0,−7).

x2 − 3x2x3 − 4x4 Not a linear function, as it does notsatisfy the additivity assumption be-cause of the term −3x2x3 in it.

x21 + x2

2 + x23 + x2

4 Satisfies additivity assumption, butnot the proportionality assumption,because of the square terms. Hencenot a linear function.

Another equivalent definition of a linear function is the following:

Definition 2: A real valued function h(x) of variables x =(x1, . . . , xn)T is said to be a linear function if for every pair of vectorsy = (y1, . . . , yn)

T and z = (z1, . . . , zn)T and scalars α, β, we have

h(αy + βz) = αh(y) + βh(z)

or equivalently, for every set of vectors {x1, . . . , xr} in Rn, and scalarsα1, . . . , αr, we have

h(α1x1 + . . . + αrx

r) = α1h(x1) + . . . + αrh(xr)

To show that Definition 1 implies Definition 2: Supposeh(x) satisfies Definition 1. Then h(x) = c1x1 + . . . + cnxn for someconstants c1, . . . , cn. Given y = (y1, . . . , yn)

T and z = (z1, . . . , zn)T ,and scalars α, β,

h(αy+βz) =n∑

j=1

cj(αyj +βzj) = αn∑

j=1

cjyj +βn∑

j=1

cjzj = αh(y)+βh(z).

So, h(x) satisfies Definition 2.

To show that Definition 2 implies Definition 1: Supposeh(x) satisfies Definition 2. Let I.j be the jth unit vector in Rn forj = 1 to n. Then the vector x satisfies

82 Ch. 1. Linear eqs.

x = (x1, . . . , xn)T = x1I.1 + . . . + xnI.n

Applying Definition 2 using this equation, we have

h(x) = x1h(I.1) + x2h(I.2) + . . . + xnh(I.n).

So, if we denote h(I.j) (the value of the function corresponding tothe jth unit vector) by cj, we have

h(x) = c1x1 + . . . + cnxn

and hence h(x) satisfies Definition 1.

Affine Functions

Definition 3: A real valued function, a(x), of variables x =(x1, . . . , xn)T is said to be an affine function if it is equal to a con-stant + a linear function of variables x; i.e., if there are constantsc0, c1, . . . , cn such that

a(x) = c0 + c1x1 + . . . + cnxn

for all x ∈ Rn. ��

Hence every linear function is also an affine function (if the constantc0 = 0), but the converse may not be true.

As an example, the function −300+2x2−3x3 is an affine function,but not a linear function.

Another equivalent definition of an affine function is the following:

Definition 4: A real valued function h(x) of variables x =(x1, . . . , xn)T is said to be an affine function if for every pair of vectorsy = (y1, . . . , yn)

T and z = (z1, . . . , zn)T and scalars α, β satisfyingα + β = 1, we have

h(αy + βz) = αh(y) + βh(z)

1.7. Constraint Functions, Linearity Assumptions 83

or equivalently, for every set of vectors {x1, . . . , xr} in Rn, andscalars α1, . . . , αr satisfying α1 + . . . + αr = 1, we have

h(α1x1 + . . . + αrx

r) = α1h(x1) + . . . + αrh(xr) ��

Notice the difference between the conditions stated in Definitions 2and 4. Definition 2 requires the condition to hold for all linear combi-nations of vectors, but Definition 4 requires it to hold only for all affinecombinations of vectors (i.e., only when the scalar multipliers in thelinear combination sum to 1).

We leave it to the reader to verify that Definitions 3 and 4 areequivalent using arguments similar to those used above to show thatDefinitions 1 and 2 are equivalent. Remember that a function a(x)satisfying Definition 4 is of the form c0 + c1x1 + . . . + cnxn where

c0 = a(0) where 0 = (0, . . . , 0)T is the zero vector inRn.

and cj = a(I.j) for j = 1 to n, where I.j is the jth unitvector in Rn.

Sometimes, in informal conversations, people refer to affinefunctions also as linear functions.

A real valued function f(x) of variables x = (x1, . . . , xn)T

which is neither linear nor affine is known as a nonlinear function.An example of a nonlinear function in the variables x = (x1, x2)

T isf(x) = 9x2

1 + 16x1x2 + 6x22.

Hence, given a system of m linear equations in variables x = (x1,. . . , xn)T , if the constraint functions associated with the m constraintsare g1(x), . . . , gm(x) and the RHS constants vector is b = (b1,. . . , bm)T , then the system can be written as

gi(x) = bi, i = 1, . . . , m.

For i = 1 to m, by transferring the RHS constant term in the ithequation to the left, we get the affine function ai(x) = gi(x) − bi. Inthis case, the above system of equations can be written in an equivalentmanner as

84 Ch. 1. Linear eqs.

ai(x) = gi(x) − bi = 0, i = 1, . . . , m

This shows that each constraint in a system of linear equationsrequires that the constraint function associated with it should have avalue equal to the RHS constant in that equation. This constraint isin the model only because of this requirement.

In constructing mathematical models for real world problems, veryoften the constraint functions that we have to deal with do not satisfythe linearity assumptions exactly. For instance, in Example 2 of Section1.2, we assumed that the amount of nutrient (like a vitamin) absorbedby the body from the food consumed, is proportional to the amountof the nutrient present in the food. This is only approximately true.In reality, the percentage of vitamins in the diet absorbed by the bodygoes up if some protein is present in the diet.

If the constraint functions violate the linearity assumptions by hugeamounts, then we cannot model the equality constraints in the prob-lem as a system of linear equations. However, in many applications,the constraint functions, while not satisfying the linearity assumptionsexactly, do satisfy them to a reasonable degree of approximation. Inthis case, a linear equation model for the system of equality constraintsin the problem provides a reasonable approximation.

The reader should read the examples and exercises in Section 1.2and check whether the constraint functions in them do satisfy the lin-earity assumptions either exactly or to a reasonable degree of approxi-mation.

Admonition to engineers: In engineering there is an oft quotedadmonition: Be wise, linearize!. This refers to the fact that engineersvery often simplify models of problems by approximating constraintfunctions with judiciously chosen linear functions. ��

A system of equations of the form

fi(x) = 0, i = 1, . . . , m

1.7. Constraint Functions, Linearity Assumptions 85

where at least one of the constraint functions f1(x), . . . , fm(x) is non-linear (i.e., not linear or affine) is called a system of nonlinear equa-tions. Systems of nonlinear equations are much harder to solve thansystems of linear equations. Usually nonlinear equations are solved byiterative methods. These require in each iteration the solution of atleast one system of linear equations based on the linear approxima-tions of functions at the current point. Solving nonlinear equations isoutside the scope of this book.

Bilinear Functions

A real valued function of two or more variables is said to be abilinear function if the variables can be partitioned into two vectorsu, v say, such that when u is fixed at any u, the function is a linearfunction in the remaining variables v; and similarly when v is fixed atany v, the function is a linear function in u.

Thus, if the function is f(u, v), it is bilinear if

for any u, f(u, v) is a linear function of v, and

for any v, f(u, v) is a linear function of u.

An example of a bilinear function is the dot product of two vectors.Let u = (u1, . . . , un), v = (v1, . . . , vn)T . Let the function

f(u, v) = uv = dot product of u and v.

f(u, v) is a nonlinear function of all the variables u1, . . . , un,v1,. . . ,vn;however it is bilinear since it is a linear function of v when u is fixedat any u, and a linear function of u when v is fixed at any v. Forinstance, let n = 3, u = (3,−1,−7), v = (−2, 0, 1)T , u = (u1, u2, u3),v = (v1, v2, v3)

T . Then the dot product of u and v, f(u, v) satisfies

f(u, v) = 3v1 − v2 − 7v3; f(u, v) = −2u1 + u3

both of which are linear functions. We state this formally in the fol-lowing result.

86 Ch. 1. Linear eqs.

Result 1.7.1: The dot product of two vectors is a bilinearfunction of the two vectors: The dot product of two vectors is abilinear function, whenever any of these vectors is fixed equal to somenumerical vector, the dot product is a linear function of the other vector.

1.8 Solutions to a System, General Solu-

tion; Nonbasic (Independent), Basic

(Dependent) Variables in Parametric

Representation

Three Possibilities

Given a system of linear equations, there are three different possi-bilities, which we discuss below.

(a): System Has No Solution: In this case we say that thesystem in infeasible or inconsistent. An example of such a systemis:

x1 + x2 = 2

−x1 − x2 = −1

If there is a solution (x1, x2)T satisfying both these equations, then

by adding them we would get 0 = 1, a contradiction. Hence thissystem is infeasible, or inconsistent.

(b): System Has a Unique Solution: System may have a so-lution, and that solution may be the only one, i.e., unique solution.An example of such a system is the following:

x1 = 1

x1 + x2 = 3

1.8. General Solution 87

By subtracting the first equation from the second, we see that thissystem has the unique solution (x1, x2)

T = (1, 2)T .We will see later that if a system of m equations in the variables

x = (x1, . . . , xn)T has the unique solution x = (c1, . . . , cn)T , then thesystem of equations is equivalent to, i.e., can be reduced to thesimple system:

x1 = c1

...

xn = cn

(c): System Has an Infinite Number of Solutions: When thesystem has more than one solution, it actually has an infinite numberof solutions. An example of such a system is:

x1 + x3 = 10

x2 + x3 = 20

For this system we see that x = (10−α, 20−α, α)T is a solutionfor every real value of α.

In this case the set of all solutions of the system is called the solu-tion set for the system.

When the solution to the system is unique, the solution set consistsof that single solution.

Two systems of linear equations in the same variables, are said tobe equivalent, if they have the same solution set.

Representation of the General Solution in Para-metric Form

Consider a system of m equations in variables x = (x1, . . . , xn)T

which has more than one solution. In this case we will show that thesystem of equations is equivalent to another, which expresses a subset

88 Ch. 1. Linear eqs.

of r variables as linear functions of the other variables for some r ≤ m;i.e., that the system of equations can be reduced to one in the followingform (here for simplicity we are assuming that the r variables expressedin terms of the others are x1, . . . , xr).

x1 = α1 + β1,r+1xr+1 + . . . + β1,nxn

x2 = α2 + β2,r+1xr+1 + . . . + β2,nxn

......

xr = αr + βr,r+1xr+1 + . . . + βr,nxn

where α1, . . . , αr, and the β’s are constants. This system does not sayanything about the values of the variables xr+1, . . . , xn; hencethey can be given any arbitrary real values; and then the values ofx1, . . . , xr can be computed by substituting those values for xr+1, . . . ,xn in the above expressions to yield a solution to the system. Ev-ery solution of the system can be obtained this way. Hence a generalsolution for the original system is of the form:

x1...xr

xr+1...

xn

=

α1 + β1,r+1xr+1 + . . . + β1,nxn...

αr + βr,r+1xr+1 + . . . + βr,nxn

xr+1...

xn

where xr+1, . . . , xn are parameters that can take arbitrary real val-ues. Such an expression is known as a representation of the generalsolution to the system in parametric form with xr+1, . . . , xn

as real valued parameters.In this representation, the variables xr+1, . . . , xn which can

be given arbitrary real values are known as independent or free ornonbasic variables. The variables x1, . . . , xr whose values inthe solution are computed from the values of independent variables areknown as dependent or basic variables in this representation.

1.8. General Solution 89

In the general solution given above, the expressions for the ba-sic variables in terms of the nonbasic variables are listed in the orderx1, . . . , xr. Hence x1, x2, . . . , xr are known as the first basic vari-able, second basic variable, ..., rth basic variable respectively inthis general solution.

When the basic variables are put in this order as a vector, (x1, . . . , xr),this vector is called the basic vector for this representation.

The particular solution obtained by giving all nonbasic variablesthe value zero, namely:

x1...xr

xr+1...

xn

=

α1...

αr

0...0

is known as the basic solution for the system of equations corre-sponding to the basic vector (x1, . . . , xr).

The choice of basic, nonbasic variables leading to such a represen-tation of the general solution is usually not unique; i.e., the variablesx1, . . . , xn can be partitioned into basic, nonbasic variables in manydifferent ways leading to such a representation of the general solution.One important property is that in all such partitions, the number ofbasic variables, r will always be the same. This number r is calledthe rank of the original system of equations. The number of nonbasicvariables in such a representation = n− r (where n is the total numberof variables in the system, and r is the number of basic variables in therepresentation) and is called the dimension of the solution set forthe system.

As an example, consider the system:

x1 + x3 = 10

x2 + x3 = 20

For this system, one parametric representation of the general solu-tion is:

90 Ch. 1. Linear eqs.

x1

x2

x3

=

10 − x3

20 − x3

x3

for arbitrary real values of the parameter x3. In this representation:

basic vector is: (x1, x2); nonbasic variable is: x3; rank of systemis 2; dimension of the solution set is 1. The basic solution of thesystem corresponding to this basic vector (x1, x2) is: (x1, x2, x3)

T =(10, 20, 0)T .

Another representation of the general solution of this system is:

x1

x2

x3

=

x1

10 + x1

10 − x1

this one corresponds to the selection of (x2, x3) as the basic vector,and x1 as the nonbasic variable; and this also exhibits the same rankof 2 for the system, and dimension of 1 for the solution set. Also, thebasic solution of the system corresponding to the basic vector (x2, x3)is: (x1, x2, x3)

T = (0, 10, 10)T .Efiicient algorithms for determining the general solution of a given

system of linear equations are discussed later on.

Exercises

1.8.1: Problem of the incorrectly cashed cheque: (Adoptedfrom M. H. Greenblatt, Mathematical Entertainments, A collection ofIlluminating Puzzles New and Old, T. Y. Crowell co., NY, 1965) Longtime ago there was a man who cashed his pay cheque at a bank wherethe teller made the mistake of giving him as many dollars as thereshould have been cents, and as many cents as there should have beendollars. The man did not notice the error. He then made a 5 centpayphone call (phone rates were low in those days). Then he discoveredthat he had precisely twice as much money left as the pay cheque had

1.8. General Solution 91

been written for. Construct a linear equation model for this problem,find its general solution, and using the limits for the variables thatcome from the above facts on it deduce what the amount of his paycheque is.

1.8.2: For each of the following problems construct a linear equa-tion model, find its general solution, and using all the facts stated inthe problem on it, deduce the required answer.

(i):A brother B and sister S are in their twenties now. When Swas one year older than half of Bs present age, B was as old as S isnow. Find their present ages.

(ii): A brother B is 6 years older than his sister S. The 1st digitin B’s age is twice the 2nd digit in S’s age; the 2nd digit in B’s age istwice the 1st digit in S’s age. Find their ages.

(iii): A four digit number has the property that if you move theleftmost digit in it to the rightmost position, its value changes to 1 +three-fourths of the original number. Find the number.

(iv): A two digit number is equal to 34 + one-third of the numberobtained by writing the digits in it in reverse order. Find this number.

(v): A kid’s piggybank has a total of 70 coins consisting of pennies(cents), nickels (5-cent coins), and dimes (10-cent coins) for a totalvalue of $1.17. Determine how many coins of each type it has.

(vi): A brother B and sister S took the same examination in whichthere were several questions. Both B and S attempted all the questions.S got a third of the answers wrong. B had 5 of them wrong. Both ofthem put together got three-quarters of the questions right. Find outhow many questions were there, and how many each answered correctly.

92 Ch. 1. Linear eqs.

1.9 Linear Dependence Relation Among

Constraints, Redundant Constraints,

Linearly Independent Systems

Linear Combinations of Constraints

In a system with m linear equations, let R1 to Rm denote them equations. A linear combination of these constraints is of theform:

∑mi=1 αiRi, i.e., multiply Ri by αi on both sides of

the equality symbols and add over i = 1 to m. Here all αi arearbitrary real numbers.

Consider the system of equations:

x1 + x3 = 10

x2 + x3 = 20

If x = (x1, x2, x3)T is a solution to this system, it satisfies

both the first and the second constraints, and hence also any linearcombination of these constraints, namely:

α1(x1 + x3) + α2(x2 + x3) = 10α1 + 20α2

for every α1, α2 real. For example, taking α1 = 5, α2 = 10 leadsto the linear combination of constraints in above system:

5x1 + 10x2 + 15x3 = 250

which will be satisfied by every solution of the system.For exactly the same reason, we have the following result:

Result 1.9.1: Every solution satisfies all linear combina-tions of equations in system:Every solution of a system of linearequations satisfies every linear combination of constraints in the sys-tem.

Redundant Constraints in a System of LinearEquations

1.9. Linearly Dependent (Independent) Systems 93

Consider the following system of linear constraints in detached co-efficient tableau form:

x1 x2 x3 x4 x5

1 1 1 1 1 101 −1 0 0 0 51 0 −1 0 0 151 0 0 −1 0 204 0 0 0 1 50

Let R1 to R5 denote the constraints in the system from topto bottom. The following relationship can be verified to hold amongthese constraints:

R5 = R1 + R2 + R3 + R4

In a system of linear equations, a particular constraint can be saidto be a redundant constraint if it can be obtained as a linear com-bination of the other constraints. So, in the system above, R5 canbe treated as a redundant constraint. If we delete R5 from thatsystem, we get the following system:

x1 x2 x3 x4 x5

1 1 1 1 1 101 −1 0 0 0 51 0 −1 0 0 151 0 0 −1 0 20

Since the second system has one constraint less than the first, wehave:

Solution set of 2nd system ⊃ Solution set of 1st system

However, since R5 is a linear combination of R1 to R4, byResult 1.9.1, every solution of the 2nd system satisfies R5. Henceboth systems above have the same solution set. Using exactly the sameargument on a general system, we have the following result:

94 Ch. 1. Linear eqs.

Result 1.9.2: Deletion of a redundant constraint leads toan equivalent system: Consider a system of linear equations. If oneof the equations in the system can be expressed as a linear combinationof the other equations, that equation can be considered as a redundantconstraint and deleted from the system without affecting the set of so-lutions; i.e., this deletion leads to an equivalent system with one lessequation. The same process can be repeated on the remaining system.

Linear Dependence Relations for a System of Lin-ear Equations

In the system in the first tableau above, the constraints R1 to R5

satisfy R5 = R1 +R2 +R3 +R4. So, if we take the linear combinationof the constraints R1 + R2 + R3 + R4 − R5, we get the equation:

0x1 + 0x2 + 0x3 + 0x4 + 0x5 = 0

with all the coefficients of the variables on the LHS as 0, and the RHSconstant 0. We will call this the 0 = 0 equation.

A linear combination of these constraints R1 to R5 is of theform

∑5i=1 αiRi. In this linear combination if we take αi = 0 for

all i = 1 to 5, we get the trivial linear combination, and it of courseresults in the 0=0 equation.

If at least one of the αi is �= 0, the linear combination is calleda nontrivial linear combination.

In the same way, in a general system of linear equations with equa-tions R1 to Rm, a nontrivial linear combination of these equationsis of the form:

∑mi=1 αiRi with at least one αi �= 0; and we say

that the equation Ri participates in this linear combinationiff αi �= 0.

As an illustration, in the linear combination R1+R2+R3+R4−R5

in the example above, all the equations R1 to R5 are participatingbecause all of them have nonzero coefficients in it.

A nontrivial linear combination of a system of equations is said tobe a linear dependence relation for the system if it leads to the0=0 equation.

1.9. Linearly Dependent (Independent) Systems 95

In the example considered above, the linear combination R1 +R2 +R3 +R4−R5 is a linear dependence relation because it is 0=0.

Among the equations participating in a linear dependence relation,any one can be expressed as a linear combination of the others. Forinstance, in the linear dependence relation R1 + R2 + R3 + R4 − R5

in the example above, all the equations R1 to R5 are participating,and we can verify that:

R1 = R5 − R2 − R3 − R4

R2 = R5 − R1 − R3 − R4

R3 = R5 − R2 − R1 − R4

R4 = R5 − R2 − R3 − R1

R5 = R1 + R2 + R3 + R4

Therefore we have the following result:

Result 1.9.3: Any one constraint in a linear dependencerelation can be considered redundant: Among the constraintsparticipating in a linear dependence relation for a system of linear equa-tions. any one of them can be considered as a redundant equation andeliminated from the system without affecting the set of solutions.

A system of linear equations is said to be linearly dependent ifthere is at least one linear dependence relation for it. In a linearlydependent system of linear equations, there is a redundant equationthat can be eliminated without affecting the set of solutions.

A system of linear equations is said to be linearly independentif there is no linear dependence relation for it; i.e., if no nontriviallinear combination of constraints in it leads to the 0=0 equation.For instance, the system obtained by deleting R5 from the systemconsidered above, which is reproduced below

96 Ch. 1. Linear eqs.

x1 x2 x3 x4 x5

1 1 1 1 1 101 −1 0 0 0 51 0 −1 0 0 151 0 0 −1 0 20

is linearly independent. To verify this, the equations in this tableaufrom top to bottom are R1 to R4. The linear combination

∑4i=1 αiRi

is the equation:

(α1 + α2 + α3 + α4)x1 + (α1 − α2)x2 + (α1 − α3)x3

+(α1 − α4)x4 + α1x5 = 10α1 + 5α2 + 15α3 + 20α4

For this to be the 0=0 equation, we must have the coefficientof x5, α1 = 0. Given α1 = 0, the coefficients of x2, x3, x4 are 0 onlyif α2 = α3 = α4 = 0. Hence the only linear combination of the fourequations in this system that is the 0=0 equation is the triviallinear combination; i.e., the system is linearly independent.

A system of equations which is linearly dependent is said to beminimally linearly dependent if the system obtained by eliminat-ing any one constraint from it is always linearly independent. Hencea system of linear equations is minimally linearly dependent iff it islinearly dependent, and there is no linear dependence relation for it inwhich at least one constraint does not participate. An example of aminimally linearly dependent system of linear equations is the systemconsidered above consisting of equations R1 to R5, which is reproducedbelow:

x1 x2 x3 x4 x5

1 1 1 1 1 101 −1 0 0 0 51 0 −1 0 0 151 0 0 −1 0 204 0 0 0 1 50

Exercises

1.9. Linearly Dependent (Independent) Systems 97

1.9.1: Show that the following systems of linear equations are lin-early dependent. For each of them check whether it is minimally lin-early dependent.

(i): x1 + x2 + x3 = 100x4 + x5 + x6 = 200x7 + x8 + x9 = 300x1 + x4 + x7 = 200x2 + x5 + x8 = 200x3 + x6 + x9 = 200

(ii): x1 = 6x2 = 7x1 + x2 = 132x1 + x2 = 19

(iii): x1 + x2 + x3 + x4 = 1002x1 = 20x1 + 2x2 = 50x1 + x2 + 2x3 = 905x1 + 4x2 + 3x3 + x4 = 260

1.9.2: Show that the following systems of linear equations are lin-early independent.

(i): x1 + x2 + x3 + x4 + x5 = −18x2 + x3 + x4 + x5 = −12x3 + x4 + x5 = 9x4 + x5 = 6

(ii): x1 + x2 + x3 = 800x4 + x5 + x6 = 300x1 + x4 = 500x2 + x5 = 200

98 Ch. 1. Linear eqs.

1.10 The Fundamental Inconsistent Equa-

tion “0 = 1”

The equation 0 = 1 is known as the fundamental inconsistentequation.

Consider the following system of linear equations. Let R1 to R4

denote the constraints in it from top to bottom.

x1 x2 x3 x4 x5

1 1 −1 1 1 41 2 4 3 2 62 3 3 4 3 9

−1 2 1 3 2 15

Taking the linear combination R1 + R2 − R3 leads to theequation:

0x1 + 0x2 + 0x3 + 0x4 + 0x5 = 1

which is the fundamental inconsistent equation 0 = 1. If the abovesystem of equations has any solution, it has to satisfy this inconsistentequation too, an impossibility. This implies that the above system hasno solution.

We will show later (see item 3 in “Discussion of Some Features ofthe GJ Method”, Section 1.16) that the converse must be true also.This leads to the following result:

Result 1.10.1: Fundamental condition for inconsistency: Asystem of linear equations has no solution iff the fundamental incon-sistent equation 0=1 can be obtained as a linear combination ofequations in it.

An algorithm to determine whether this condition holds is givenlater.

Exercises

1.11. Row Operations 99

1.10.1: A problem from Bulgaria: A person A claims that heand his three brothers B, C, D live in a building, each occupying aseparate apartment. Their statement contains the following details:

(i) Apartment of A has 3 windows and 2 doors.

(ii) B has as many windows as the doors of C.

(iii) C has as many windows as the doors of B.

(iv) D says that in total his brothers have the same numberof windows as doors.

Represent these statements through a system of linear equationsand solve it. What possible conclusion can you draw about the buildingfrom this?

1.11 Elementary Row Operations on a Tableau

Leading to an Equivalent System

Row operations are the most heavily used computational tool today. Itis truly amazing that such a simple tool plays such an important role inso many complex tasks. We now discuss two different row operations(also called elementary row operations). These are:

1. Scalar Multiplication: Multiply each entry in a rowby the same nonzero scalar

2. Add a Scalar Multiple of a Row to Another:Multiply each element in a row by the same nonzero scalarand add the result to the corresponding element of the otherrow.

Here are the results for two rows in a general tableau:

100 Ch. 1. Linear eqs.

Row Con-dition

R1 a11 a12 . . . a1n b1

R2 a21 a22 . . . a2n b2

αR1 αa11 αa 12 . . . αa1n αb1 α �= 0R2 + βR1 a21 + βa11 a22 + βa12 . . . a2n + βa1n b2 + βb1 β �= 0

Here are numerical illustrations:

RowR1 1 2 −1 2 1 −10R2 −1 1 −2 −1 1 156R1 6 12 −6 12 6 −60

R2 + 2R1 1 5 −4 3 3 −5

There is another row operation called row interchange (inter-changing two rows in the tableau) which we will discuss later in Section1.20, as it is used by the Gaussian elimination method discussed there.

We state some important facts about row operations on the de-tached coefficient tableau for a system of linear equations in the fol-lowing result:

Result 1.11.1: Row operations lead to an equivalent sys-tem, and nonzero columns remain nonzero: (a): Since we aredealing with a system of equations, row operations lead to an equivalentsystem with the same set of solutions.

(b): A nonzero column vector in a tableau remains nonzero aftersome row operations are performed on the tableau.

As discussed in Section 1.4, in the detached coefficient tableau fora system of linear equations, every column vector is always nonzero.From (b) of Result 1.11.1 we conclude that every column vector contin-ues to remain nonzero after performing any sequence of row operationson the tableau.

Exercises

1.12. Memory Matrix 101

1.11.1 Perform the following row operations and record the results:

RowR1 3 2 4 1 5 −7R2 1 −2 −1 1 0 −2R3 0 −4 −2 1 1 −7R4 −1 4 −3 4 −2 8

R2 + R1

R3 + 2R1

R4 − 2R1

(1/2)R1

1.12 Memory Matrix to Keep Track of

Row Operations

A tableau changes whenever row operations are performed on it. Clearlyevery row vector in the resulting tableau is a linear combination of rowvectors in the original tableau, but we do not know what that linearcombination is. Sometimes, when performing a series of row opera-tions, it is a good idea to have the formula for each row vector in thecurrent tableau as a linear combination of rows in the original tableau.A tool called memory matrix helps us to keep track of these linearcombinations at the expense of slight extra work. Here we explain whatthis memory matrix is, how to set it up and how to use it.

Let R1, . . . , Rm denote the row vectors in the original tableau.

If a row vector in the current tableau is the linear combinationα1R1+. . .+αmRm, the memory matrix will store this in detached coef-ficient form, i.e., it consists of column vectors labeled with R1, . . . , Rm

at the top, and in these column vectors it will have entries α1, . . . , αm

in this row; and updates this whenever new row operations are carriedout.

102 Ch. 1. Linear eqs.

Algorithm: How to Set Up the Memory Matrix and UseIt

BEGIN

To use this facility, before doing any row operations, on the lefthand side of the original tableau open a memory matrix consisting ofm column vectors labeled R1, . . . , Rm at the top. For each i =1 to m, the column vector under Ri should be the ith unit vector ofdimension m. This will be the initial memory matrix set up on theoriginal tableau itself. All it says is that the first row vector in theoriginal tableau is 1R1 + 0R2 + . . . + 0Rm = R1, the second rowvector is 0R1 + 1R2 + 0R3 + . . . 0Rm = R2, etc.

Then, whenever you perform any row operations on the tableau,carry out exactly the same operations on the memory matrix also (thisis the extra work in operating this memory matrix). This updates thememory matrix appropriately, and at any stage, the ith row of thememory matrix will contain the coefficients of R1, . . . , Rm in theexpression of this row in the current tableau as a linear combination ofR1 to Rm.

END

We will illustrate with an example of a detached coefficient tableaufor a system consisting of 4 equations in 5 variables.

x1 x2 x3 x4 x5

1 −1 1 0 1 52 1 −2 1 −1 60 0 1 2 1 −21 2 0 0 0 3

Denoting the constraints from top to bottom as R1 to R4, we openthe memory matrix as below.

1.12. Memory Matrix 103

Memory matrix Original tableauR1 R2 R3 R4 x1 x2 x3 x4 x5

1 0 0 0 1 −1 1 0 1 50 1 0 0 2 1 −2 1 −1 60 0 1 0 0 0 1 2 1 −20 0 0 1 1 2 0 0 0 3

So, the information in the memory matrix on the original tableauindicates that the first row in the original tableau is 1R1 +0R2 +0R3 +0R4 = R1, etc.

Now suppose we perform the following row operations:

Multiply Row 1 by 2

Subtract two times Row 1 from Row 2

Subtract Row 1 from Row 4.

We carry on the same operations on the memory matrix too. Thisleads to the current tableau which is the tableau after the above rowoperations are carried out.

Memory matrix Current tableauR1 R2 R3 R4 x1 x2 x3 x4 x5

2 0 0 0 2 −2 2 0 2 10−2 1 0 0 0 3 −4 1 −3 −4

0 0 1 0 0 0 1 2 1 −2−1 0 0 1 0 3 −1 0 −1 −8

According to the memory matrix,

the first row in the current tableau is 2R1,the second row is −2R1 + R2,the third is R3,and the fourth is −R1 + R4

which are true. In the same way, we can continue performing additionalrow operations on the current tableau, and the memory matrix will

104 Ch. 1. Linear eqs.

update and always show the coeffients in the expressions of rows of thecurrent tableau as linear combinations of rows in the original tableau.

Thus when performing a sequence of row operations on a tableau,we use the memory matrix if we need to have the expressions for rowsin the current tableau as linear combinations of rows in the originaltableau.

1.13 What to Do With “0=0”, Or “0=1”

Rows Encountered After Row Oper-

ations ?

The “0=0” Equation

We start with an example first. Consider the detached coefficienttableau given below for a system of linear equations, in which we callthe equations as R1 to R4.

Original Tableau-1Eq. name x1 x2 x3 x4 x5

R1 1 1 1 1 1 −2R2 2 −1 2 −1 2 −4R3 5 −1 5 −1 5 −10R4 3 4 2 1 −2 6

We introduce the memory matrix, and then perform the followingrow operations: subtract Row 1 from Rows 3 and 4. This leads to thefollowing tableau:

Memory matrixR1 R2 R3 R4 x1 x2 x3 x4 x5

1 0 0 0 1 1 1 1 1 −20 1 0 0 2 −1 2 −1 2 −4

−1 0 1 0 4 −2 4 −2 4 −8−1 0 0 1 2 3 1 0 −3 8

1.13. “0=0” Or “0=1” Rows 105

In this tableau we subtract two times Row 2 from Rows 3 and 4.This leads to the following Current Tableau-1.

Memory matrix Current Tableau-1R1 R2 R3 R4 x1 x2 x3 x4 x5

1 0 0 0 1 1 1 1 1 −20 1 0 0 2 −1 2 −1 2 −4

−1 −2 1 0 0 0 0 0 0 0−1 −2 0 1 −2 5 −3 2 −7 16

In this tableau Row 3 has become the “0=0” equation. What shouldbe done about it?

This implies that the set of 4 constraints in the original system islinearly dependent, and that a linear dependence relation for them hasjust been discovered by the row operations we carried out. Whenever a“0=0” row appears in the current tableau after carrying out some rowoperations on the detached coefficient tableau for a system of linearequations, it is an indication of the discovery of a linear dependencerelation among the original constraints, and consequently the equa-tion corresponding to this row in the original system is a redundantconstraint.

That linear dependence relation can be read out from the entriesin the memory matrix in the row of the “0=0” row in the currenttableau. Here from Row 3 in the memory matrix in the current tableauwe find that the linear dependence relation is: −R1 − 2R2 + R3,because it leads to the “0=0” equation. The vector of coefficients ofthe equations in the original system (in the order in which the equationsare listed in the original system) in this linear dependence relation iscalled the evidence (or certificate) of redundancy for treatingthe equation corresponding to Row 3 as a redundant constraint in theoriginal system. The reason for the name is the fact that one can formthe linear combination of the constraints in the original system withcoefficients in this row and actually verify that it leads to the “0 = 0”equation. This evedence is provided by Row 3 of the memory matrixin the current tableau.

This evidence (−1,−2, 1, 0) implies that among the constraints in

106 Ch. 1. Linear eqs.

the original tableau, we have the relation: R3 = R1 + 2R2. So, R3

can be eliminated from the original system as a redundant constraint,leading to an equivalent system. The effect of this is to eliminate the“0=0” row in the current row (i.e., third row) and the correspondingcolumn, column 3 of R3, from the memory matrix. After this elimina-tion, the resulting equivalent original tableau, and current tableau areshown below.

Original Tableau-2Eq. name x1 x2 x3 x4 x5

R1 1 1 1 1 1 −2R2 2 −1 2 −1 2 −4R4 3 4 2 1 −2 6

Memory matrix Current Tableau-2R1 R2 R4 x1 x2 x3 x4 x5

1 0 0 1 1 1 1 1 −20 1 0 2 −1 2 −1 2 −4

−1 −2 1 −2 5 −3 2 −7 16

The processing of these equivalent systems can now continue.In the same way, whenever a “0=0” row is encountered after some

operations on the detached coefficient tableau for a system of linearequations, do the following:

• Suppose the “0=0” equation is in Row r of the current tableau.

It is an indication that the equation in Row r of the originalsystem is a redundant constraint. If the memory matrix is beingmaintained, you can get an expression for this constraint as alinear combination of the other constraints from Row r of thememory matrix in the current tableau (this row of the memorymatrix is the evidence for the redundancy of the equation in Rowr), as illustrated in the above example.

• Eliminate Row r (i.e., the row of the “0=0” equation) from thecurrent tableau, and also eliminate column r of the memory ma-trix if it is being maintained. Also, eliminate Row r from the

1.13. “0=0” Or “0=1” Rows 107

original tableau. Continue processing the remaining tableaus,which represent an equivalent system with one less constraint.

Equivalently, you can leave this row r in both the current andoriginal tableaus (and row r and column r of the memory matrixtoo) as they are in these tableaus, but ignore them in any futurecomputation to be carried out.

Equivalent Systems

Suppose we are given a system of linear equations, say System 1.Any system, say System 2, obtained by performing a series of rowoperations on System 1 and eliminating any redundant equations dis-covered during the process is said to be equivalent to System 1. Allsuch systems have the same set of solutions as System 1.

Given a system of linear equations to solve, the strategy followedby algorithms discussed in the following sections is to transform thissystem through a series of row operations into an equivalent simplerone for which the general solution can be easily recognized.

The “0=1” Equation

Again we begin with an example. Consider the following system ofequations in detached coefficient tableau form.

Original TableauEq. name x1 x2 x3 x4 x5

R1 1 1 1 1 1 15R2 1 2 2 2 2 22R3 1 3 3 3 3 35R4 2 5 4 5 4 40

We introduce the memory matrix and perform the following rowoperations: subtract Row 1 from Rows 2 and 3; and subtract twotimes Row 1 from Row 4. This leads to the following tableau.

108 Ch. 1. Linear eqs.

Memory matrixR1 R2 R3 R4 x1 x2 x3 x4 x5

1 0 0 0 1 1 1 1 1 15−1 1 0 0 0 1 1 1 1 7−1 0 1 0 0 2 2 2 2 20−2 0 0 1 0 3 2 3 2 10

On this tableau subtract two times Row 2 from Row 3, and subtractthree times Row 2 from Row 4. This leads to the following tableau.

Memory matrixR1 R2 R3 R4 x1 x2 x3 x4 x5

1 0 0 0 1 1 1 1 1 15−1 1 0 0 0 1 1 1 1 7

1 −2 1 0 0 0 0 0 0 61 −3 0 1 0 0 −1 0 −1 −11

The third row in this tableau is the equation “0=6”. So, fromthe memory matrix entries in this row we conclude that the linearcombination 1

6(R1 − 2R2 + R3) is the fundamental inconsistent

equation “0=1”, and hence that the original system has no solution.

The vector of coefficients of the equations in the original system(in the order in which the equations are listed in the original system)that produce the “0 = 1” equation or the “0 = a” equation for somea �= 0 has been called by the obscure name supervisor principle forinfeasibility. The idea behind this name is that if your supervisoris suspicious about your claim that the system is infeasible, he/shecan personally verify the claim by obtaining the linear combination ofconstraints in the original system with this vector of coefficients andverify that it is “0 = 1” or “0 = a” for a �= 0. But we will call this vectorby the simpler name evidence (or certificate) of infeasibility. Thisevidence is provided by the row in the memory matrix in the currenttableau that corresponds to the “0 = a” equation for a �= 0.

In the same way for any system of equations, if we end up with arow corresponding to the equation “0 = a” where a �= 0 after some rowoperations on it, we conclude that the system is inconsistent and has

1.14. Elimination Approach 109

no solution, and terminate.When row operations are carried out on the detached coefficient

tableau of a system of linear equations without introducing the memorymatrix, we will still be able to discover all the redundant constraints inthe system, and determine if the system is infeasible; but then we willnot have the evedences (or certificates) of redundancy or infeasibility.To have those evidences we need to introduce the memory matrix onthe system before row operations are commenced, and carry out all thesame row operations on the columns of the memory matrix as well.

1.14 The Elimination Approach to Solve

Linear Equations

The classical elimination approach for solving systems of linearequations was figured out by Chinese and Indian mathematicians morethan 2000 years ago. Without trying to list all possibilities that canoccur individually, we want to present the main idea behind this ap-proach, which is the following:

Algorithm: The Elimination Approach For SolvingLinear Equations

BEGIN

Select any equation and take it out of the system. Selecta variable which appears with a nonzero coefficient in itas the basic variable in this equation. Use this equationto derive the expression for this basic variable in terms ofthe other variables. Substitute this expression for the basicvariable in all the other equations, thereby eliminating thisvariable from the remaining system.

Once the values of the other variables are determined fromthe remaining system, the corresponding value of this vari-able can be computed from the expression for it obtainedabove.——

110 Ch. 1. Linear eqs.

The remaining system is now smaller. It has one equationless, and one variable less than the previous system. If ithas a “0 = a” equation in it for some a �= 0, the originalsystem is inconsistent (i.e., it has no solution), terminate.If it has some “0 = 0” equations, eliminate them. If theremaining system now has only one equation, obtain thegeneral solution for it, and using the expressions for ba-sic variables generated in previous steps obtain the generalsolution of the original system. If the remaining systemconsists of more than one equation, apply the eliminationstep on it and repeat the same way.

END

Example: Solving a Syatem o f L inear Equationsby the Elimination Approach

We will apply this approach on the following system of 3 equationsin 4 variables as an illustration:

Original System = System 12x1 +4x2 −6x3 = 10x1 −x2 +x3 −x4 = 5

−x1 +2x2 −2x3 +x4 = 3

Selecting the first equation to take out from the system, and thevariable x1 with the coefficient of 2 as the basic variable in it, we getthe expression:

x1 =1

2(10 − 4x2 + 6x3) = 5 − 2x2 + 3x3

We now substitute this expression for x1 in the remaining equations2 and 3 leading to:

(5 − 2x2 + 3x3) − x2 + x3 − x4 = 5

−(5 − 2x2 + 3x3) + 2x2 − 2x3 + x4 = 3

1.14. Elimination Approach 111

Rearranging terms, this leaves the reduced system:

System 2−3x2 +4x3 −x4 = 0

4x2 −5x3 +x4 = 8

The reduced system, System 2 has two equations in three variables.Taking out the top equation from it, and the variable x4 with thecoefficient of −1 as the basic variable in it, we get the expression:

x4 = −3x2 + 4x3

Substituting this expression for x4 in the remaining equation leadsto the system with one equation only

4x2 − 5x3 + (−3x2 + 4x3) = 8

or

x2 − x3 = 8

The remaining system is a single equation in two variables x2, x3.Selecting x2 as the basic variable in it, we get the general solution forit as:

(x2

x3

)=

(8 + x3

x3

)

x3 is an arbitrary real valued parameter. Plugging this solution in theexpression for x4 above leads to the general solution

x2

x3

x4

=

8 + x3

x3

−24 + x3

for System 2. Plugging this in the expression for x1 above leads to thegeneral solution for the original system

112 Ch. 1. Linear eqs.

x1

x2

x3

x4

=

−11 + x3

8 + x3

x3

−24 + x3

where x3 is an arbitrary real parameter. In particular, taking x3 = 0leads to the basic solution WRT the basic vector (x1, x4, x2), which is:

x1

x2

x3

x4

=

−1180

−24

Exercises

1.14.1 Apply the elimination approach to get the general solutionof the following systems. For each, check whether the solution obtainedis unique, and explain your conclusions carefully.

(i) An infeasible system−x1 + 7x2 − 9x3 = 10x1 − 8x2 + 12x3 = −152x1 − 11x2 + 7x3 = 52x1 − 12x2 + 10x3 = 03x1 − 19x2 + 19x3 = −10

(ii) An infeasible system3x1 − x2 + 2x3 − 7x4 = 102x1 + 2x2 + 15x4 = 37x1 + 3x2 + 2x3 + x4 = 168x1 + 4x3 + x4 = 30

1.15. Pivot Step 113

(iii) A system with a unique solution5x1 − 7x2 − x3 = 5−8x1 + 15x2 + 2x3 = 15−2x1 + 9x2 + x3 = −13

(iv) A system with multiple solutions4x1 + 5x2 − x3 − 6x4 + 3x5 = 810x1 − 6x2 + x3 + 13x4 + 2x5 = −424x1 − 7x2 + x3 + 20x4 + 7x5 = 013x1 + x4 + 17x5 = −5

1.14.2: Solve the following systems of equations through simplemanipulations.

(a) x1 x2 x3 x4

1 7 3 5 168 4 6 2 −162 6 4 8 165 3 7 1 −16

(b) x1 x2 x3 x4

1 1 1 0 40 1 1 1 −51 0 1 1 01 1 0 1 −8

1.15 The Pivot Step, a Convenient Tool

For Carrying Out Elimination Steps

Here we discuss a step known as the Gauss-Jordan pivot step orGJ pivot step or sometimes just pivot step which offers a veryconvenient tool for carrying out the elimination steps discussed in theprevious section. In a later section we will discuss a slightly differentversion of the pivot step called the Gaussian pivot step.

Consider the general system of m equations in n variables in de-tached coefficient tableau form. A GJ pivot step on this tableau is a

114 Ch. 1. Linear eqs.

computational tool specified by:

• A row of the tableau selected as the pivot row for the pivot step

• A column of the tableau selected as the pivot column for thepivot step

• The element in the pivot row and the pivot column as the pivotelement for the pivot step, which should be nonzero.

The GJ pivot step itself consists of a series of row operations onthe tableau to convert the pivot column into the unit column with asingle nonzero entry of “1” in the pivot row. We give the operationsfor the general system first. We use the abbreviations “PR” for “pivotrow”, and “PC” for “pivot column”, and indicate the pivot element byputting a box around it. The general system in detached coefficienttableau form is:

Original Tableaux1 x2 . . . xs . . . xn RHS constantsa11 a12 . . . a1s . . . a1n b1

a21 a22 . . . a2s . . . a2n b2...

......

......

ar1 ar2 . . . ars . . . arn br PR...

......

......

am1 am2 . . . ams . . . amn bm

PC

This GJ pivot step is only possible if the pivot element ars �= 0.This pivot step involves the following row operations:

• for each i �= r such that ais = 0, leave Row i unchanged

• for each i �= r such that ais �= 0, carry out Row i −(

ais

ars

)Row r

• for i = r carry out (Row r)/ars

1.15. Pivot Step 115

The entries in the tableau will change as a result of the pivot step,we denote the new entries by aij , bi etc. with a bar on top

Tableau after pivot stepx1 x2 . . . xs . . . xn RHS constantsa11 a12 . . . 0 . . . a1n b1

a21 a22 . . . 0 . . . a2n b2...

......

......

ar1 ar2 . . . 1 . . . arn br...

......

......

am1 am2 . . . 0 . . . amn bm

Hence the GJ pivot step eliminates the variable corresponding tothe pivot column from all rows other than the pivot row.

We now provide some numerical examples. Consider the tableaugiven below. We will carry out the GJ pivot step on this tableau withRow 2 as the pivot row, Column 3 as the pivot column, and the boxedentry 2 in both of them as the pivot element.

Original Tableaux1 x2 x3 x4 x5 RHS−1 2 1 1 0 4

4 −2 2 0 −4 −10 PR3 7 0 −4 8 −2

−2 4 −1 1 0 −7PC

This GJ pivot step involves the following row operations:

Row 1 −(

12

)Row 2

Row 4 +(

12

)Row 2(

12

)Row 2

No change in Row 3 because the entry in pivot column in Row 3 isalready zero.

116 Ch. 1. Linear eqs.

Tableau after the pivot stepx1 x2 x3 x4 x5 RHS−3 3 0 1 2 9

2 −1 1 0 −2 −53 7 0 −4 8 −20 3 0 1 −2 −12

Notice that the pivot column, column of x3, became the 2nd unitcolumn (with a “1” entry in the pivot row, Row 2) after the pivot step.

Here is another example:

Original Tableaux1 x2 x3 x4 x5 RHS1 0 −4 2 1 −7

−1 2 2 3 −1 81 −2 −1 −5 0 −12

−2 3 4 0 1 4 PRPC

Tableau after the pivot stepx1 x2 x3 x4 x5 RHS0 3

2−2 2

32

−5

0 12

0 3 − 32

6

0 − 12

1 −5 12

−10

1 − 32

−2 0 −12

−2

Exercises

1.15.1 Perform the GJ pivot steps indicated by the pivot row, pivotcolumn, and the pivot element in a box, in the following tableaus.

1.15. Pivot Step 117

(i) x1 x2 x3 x4 x5

1 −2 −2 2 3 2

0 1 2 −1 1 −7 PR−12 −8 19 0 17 18

3 −2 −1 −2 2 −3PC

(ii) x1 x2 x3 x4 x5

0 21 48 37 0 732 −4 5 6 1 8

−3 −1 −2 −5 −1 −7

4 −4 2 6 2 20 PRPC

(iii) x1 x2 x3 x4 x5 x6

1 0 0 −1 −2 3 40 1 0 1 −1 2 5

0 0 1 2 1 −1 2 PRPC

The GJ Pivot Step as a Tool for Eliminating aVariable From a System

In Section 1.14 we talked about eliminating a variable from a sys-tem of linear equations using one of the equations in which it appearswith a nonzero coefficient. We will show that this is exactly what aGJ pivot step does. In fact, the GJ pivot step offers a very conve-nient tool for carrying out this elimination step when the system isexpressed in detached coefficient form. In the following table we showthe correspondence between various elements in the two steps.

Following this table, we reproduce the system solved in Section 1.14by the elimination method. There, we first eliminated the variable x1

from it using the first equation in the system.

118 Ch. 1. Linear eqs.

Element in Elimination step Correponding element in GJpivot step

1. Eq. used to eliminatea variable

Pivot row. Row corre-sponding to that eq. intableau is the PR in GJpivot step.

2. Variable eliminated Pivot column. Columncorresponding to that vari-able in tableau is the PC inGJ pivot step.

3. Coeff. of variableeliminated, in eq. usedto eliminate it

Pivot element.

Original System = System 12x1 +4x2 −6x3 = 10x1 −x2 +x3 −x4 = 5

−x1 +2x2 −2x3 +x4 = 3

After this elimination step, we got the following:

x1 = −2x2 + 3x3 + 5 Expression for x1 in terms ofother variables.

−3x2 + 4x3 − x4 = 04x2 − 5x3 + x4 = 8

Remaining system after x1

eliminated using 1st eq.

The PR, PC, and the boxed pivot element for the GJ pivot step onthe detached coefficient tableau for the original system, correspondingto this elimination step are indicated below:

1.15. Pivot Step 119

Original Tableaux1 x2 x3 x4 RHS

2 4 −6 0 10 PR1 −1 1 −1 5

−1 2 −2 1 3PC

Carrying out this GJ pivot step leads to the following tableau

Tableau after pivot stepx1 x2 x3 x4 RHS1 2 −3 0 50 −3 4 −1 00 4 −5 1 8

In this tableau, the equation corresponding to Row 1 is:

x1 + 2x2 − 3x3 = 5

We get the expression for x1 in terms of the other variables bytransferring the terms of those variables to the right in this equation.Also, rows 2 and 3 in this tableau contain the reduced system after x1

is eliminated from the original system.

Thus we see that the GJ pivot step eliminates the variable corre-sponding to the pivot column from all the equations other than thatrepresented by the pivot row. Hence the GJ pivot step offers a veryconvenient computational tool for carrying out elimination steps.

In Section 1.16, we will formally discuss a method for solving sys-tems of linear equations based on the elimination approach, using theGJ pivot step as the main computational tool.

Note: To carry out a sequence of pivot steps on a tableau, theyare performed one after the other in sequence. Each pivot step leads toa new tableau, on which the next pivot step is performed. Usually thevarious tableaus obtained are listed one below the other in the orderin which they are generated.

120 Ch. 1. Linear eqs.

1.16 The Gauss-Jordan (GJ) Pivotal Method

for Solving Linear Equations

We will now discuss the method for solving linear equations using theelimination approach based on GJ pivot steps. This is one of themost commonly used methods for solving linear equations. It is knownas the GJ (Gauss-Jordan) pivotal (or elimination) method andalso known as the Row reduction algorithm or Row reductionmethod.

Historical note on the GJ pivotal method: The essential ideasbehind the method were figured out by Chinese and Indian mathemati-cians more than 2000 years ago, but they were unknown in Europeuntil the 19th century when the famous German mathematician CarlFriedrich Gauss discovered them. Gauss discovered the method whilestudying linear equations for approximating the orbits of celestial bod-ies, in particular the orbit of the asteroid Ceres (whose diameter is lessthan 1000 km). The positions of these bodies are known from obser-vations which are subject to errors. Gauss was trying to approximatethe values of the unknown coefficients in an algebraic formula for theorbit, that fit the observations as closely as possible, using the methodof least squares described in Example 5 of Section 1.2. This methodgives rise to a system of linear equations in the unknown coefficients.He had to solve a system of 17 linear equations in 17 unknowns, whichhe did using the elimination approach. His computations were so ac-curate that astronomers who lost track of the the asteroid Ceres, wereable to locate it again easily.

Another German, Wilhelm Jordan, popularized the algorithm in alate 19th century book he wrote. From that time, the method has beenpopularly known as the Gauss-Jordan elimination method. ��

The method involves a series of GJ pivot steps on the detached co-efficient tableau form. To solve a system of m equations in n variables,the method needs at most m (actually min{m, n}, but as shown later,we can assume without loss of generality that m ≤ n) GJ pivot steps.

At each pivot step, the variable corresponding to the pivot column

1.16 GJ Method 121

is recorded as the basic variable selected in the pivot row, and is elimi-nated from all equations other than that represented by the pivot row(this is the property of the GJ pivot step). The equations correspondingto the expressions of each basic variable in terms of the other variablesare kept in the tableau itself.

We first give the statement of the method, and then give illustrativeexamples for all the possibilities that can occur in the method. In thedescription of each step, instead of just giving a curt statement of thework to be done in that step, we also give some explanation of thereasons for it to help the reader understand it easily.

If you need evidence for each redundant constraint identified andeliminated during the method, and evidence for infeasibility when themethod terminates with the infeasibility conclusion, you need to in-troduce the memory matrix into the computation as pointed out inSection 1.13. We omit those to keep the discussion simple. Also, theversion of the method presented below is the simplest version fromancient times. A computationally more efficient version of the samemethod using the memory matrix is presented in Section 4.11 (it isbased on computational improvements introduced by George Dantzigin his paper on the revised simplex method for linear programming).That version has the additional advantage of automatically producingthe “evidences” without increasing the computational burden.

The GJ Pivot Method

BEGIN

Step 1: Put the system in detached coefficient tableau form andobtain the original tableau. Open a column on the left hand side torecord the basic variables. Select the top row as the pivot row in theoriginal tableau and go to Step 2.

Step 2: General Step: Let r denote the number of the presentpivot row in the present tableau.

If the coefficients of all the variables in the pivot row are 0, go toStep 2.1 if the RHS constant in the pivot row, br = 0, or to Step 2.2 if——

122 Ch. 1. Linear eqs.

br �= 0. If at least one of the variables has a nonzero coefficient in thepivot row, go to Step 2.3.

Step 2.1: Redundant Constraint Elimination: In thepresent tableau, the pivot row represents the equation “0= 0”, a redundant constraint. Eliminate the correspondingrow (i.e., the one with the same number from the top) in theoriginal tableau as a redundant constraint. If this pivot rowis the last row in the present tableau, eliminate it from thepresent tableau and go to Step 3 with the resulting tableau.Otherwise, eliminate this row from the present tableau, inthe resulting tableau select the next row as the pivot rowand with these repeat Step 2. (Equivalently, you can leavethese redundant rows in both the tableaus, but ignore themin future work.)

Step 2.2: Concluding Original System Inconsistent:In the present tableau, the pivot row represents the equa-tion “0 = br” where br �= 0, an inconsistent equation. Sothe original system is inconsistent and has no solution. Ter-minate.

Step 2.3: GJ Pivot Step: Select a variable with a nonzerocoefficient in the pivot row in the present tableau as the ba-sic variable in the pivot row (i.e., the rth basic variable) andrecord it. This variable is also called the entering vari-able for this pivot step. The column vector of this basicvariable in the present tableau is the pivot column. Per-form the GJ pivot step on the present tableau, and let theresulting tableau be the next tableau.

If row r is the last row in this next tableau, go to Step 3with the next tableau. If row r is not the last row in thenext tableau, with the next row in it as the pivot row repeatStep 2.

Step 3: Checking Uniqueness of Solution, and Writing the——

1.16 GJ Method 123

General Solution:

If every variable in the system has been selected as a basic variablein some row (i.e., there are no nonbasic variables at this stage), allthe column vectors in the present tableau are unit vectors. So, thenumber of rows in the present tableau is n (n may be less than m =number of equations in original system, if some redundant equationswere eliminated during the method). In this case, the column vectorof the basic variable in the ith row is the ith unit vector in Rn for i= 1 to n. So, each row vector in the present tableau represents theequation giving the value of a different variable in the solution to thesystem. Collect these values and put them in a column vector in theproper order of the variables to yield the solution. This solution is theunique solution of the system in this case.

If there are some nonbasic variables in the present tableau, theoriginal system has an infinite number of solutions. In the equationcorresponding to the ith row, if you transfer all the terms containingnonbasic variables to the right, it gives the expression for the ith basicvariable in terms of the nonbasic variables. Collecting these expressionsfor all the basic variables together, we have a parametric representationfor the general solution of the system as discussed in Section 1.8. Thevalues of the nonbasic variables are the parameters in it, these can begiven arbitrary real values to obtain a solution for the system, and allthe solutions for the system can be obtained this way. In particular,making all the nonbasic variables equal to 0, leads to the basic solutionof the system WRT present basic vector.

END.

Numerical Examples for the GJ Method

We now provide numerical examples illustrating the three possibleways in which the method can terminate. In this method, on eachtableau one GJ step is performed yielding the next tableau on which thenext GJ step is performed, and so on. We provide the various tableausone below the other indicating the pivot row (PR), pivot column (PC),and the boxed pivot element if a GJ pivot step is performed on it;

124 Ch. 1. Linear eqs.

or marking it as RC (redundant constraint to be eliminated), or IE(inconsistent equation) otherwise. We use the abbreviation “BV” for“basic variable”.

We number all the rows (equations) serially from top to bottom,and record these numbers in a rightmost column, to make it easy tokeep track of redundant constraints eliminated.

The original detached coefficient tableau for the original system isalways the first one.

Example 1:

x1 x2 x3 x4 x5 RHS Eq. no.

−1 0 1 1 2 10 PR 11 −1 2 −1 1 5 21 −2 5 −1 4 20 32 0 0 −2 1 25 43 −2 5 −3 5 45 55 −2 5 −5 6 70 6

BV PC↑x3 −1 0 1 1 2 10 1

3 −1 0 −3 −3 −15 PR 26 −2 0 −6 −6 −30 32 0 0 −2 1 25 48 −2 0 −8 −5 −5 5

10 −2 0 −10 −4 20 6PC↑

x3 −1 0 1 1 2 10 1x2 −3 1 0 3 3 15 2

0 0 0 0 0 0 RC 3

2 0 0 −2 1 25 PR 42 0 0 −2 1 25 54 0 0 −4 2 50 6

PC

1.16 GJ Method 125

BV x1 x2 x3 x4 x5 RHS Eq. no.x3 −5 0 1 5 0 −40 1x2 −9 1 0 9 0 −60 2x5 2 0 0 −2 1 25 4

0 0 0 0 0 0 RC 50 0 0 0 0 0 RC 6

Final Tableaux3 −5 0 1 5 0 −40 1x2 −9 1 0 9 0 −60 2x5 2 0 0 −2 1 25 4

So, the constraints eliminated as redundant constraints are equa-tions 3, 5, 6 in the original tableau. When we eliminate the corre-sponding rows from the original tableau, we get the following detachedcoefficient tableau form with the redundant constraints deleted.

Original Tableau With Redundant Eqs. Deletedx1 x2 x3 x4 x5 RHS Eq. no.−1 0 1 1 2 10 1

1 −1 2 −1 1 5 22 0 0 −2 1 25 4

Since we have x1 and x4 as nonbasic variables in the final tableau,this system has an infinite number of solutions. The set of solutionshas dimension 2. From the 1st, 2nd, 3rd rows in the final tableau, weget the expressions for the 1st, 2nd, 3rd basic variables x3, x2, x5 re-spectively in terms of the nonbasic variables, leading to the parametricrepresentation of the general solution of the system as:

x1

x2

x3

x4

x5

=

x1

−60 + 9x1 − 9x4

−40 + 5x1 − 5x4

x4

25 − 2x1 + 2x4

where x1, x4 are arbitrary real valued parameters. For example, makingx1, x4 = 0 leads to the basic solution of the system WRT the basicvector (x3, x2, x5) as (x1, x2, x3, x4, x5)

T = (0,−60,−40, 0, 25)T .

126 Ch. 1. Linear eqs.

Notice also that the original system had six equations in five vari-ables, but after eliminating three redundant equtions one at a time, thefinal tableau was left with only three equations. In the final tableau,the column vectors of the 1st, 2nd, 3rd basic variables x3, x2, x5 are the1st, 2nd, 3rd unit vectors in R

3 in that order.

Example 2:

x1 x2 x3 RHS Eq. no.

1 1 1 5 PR 1−1 3 1 6 2

0 4 2 11 32 3 1 10 4

BV PC↑x3 1 1 1 5 1

−2 2 0 1 PR 2−2 2 0 1 3

1 2 0 5 4PC↑

x3 2 0 1 92

1

x2 −1 1 0 12

2

0 0 0 0 RC 3

3 0 0 4 PR 4PC↑

Final Tableaux3 0 0 1 19

61

x2 0 1 0 116

2

x1 1 0 0 43

4

So, the third constraint is eliminated as a redundant constraint.When we eliminate the corresponding row from the original tableau, we

1.16 GJ Metho d 127

get the following detached coefficient tableau form with the redundantconstraints deleted.

x1 x2 x3 RHS Eq. no.1 1 1 5 1

−1 3 1 6 22 3 1 10 4

All the variables are basic variables, and there are no nonbasic vari-ables. Hence this system has a unique solution which we read from thefinal tableau to be:

x1

x2

x3

=

43

116

196

Example 3:

x1 x2 x3 x4 x5 RHS Eq. no.

2 1 −1 −2 0 4 PR 14 1 5 6 4 9 26 2 4 4 4 12 3

10 3 9 10 8 22 4BV PC↑x2 2 1 −1 −2 0 4 1

2 0 6 8 4 5 PR 22 0 6 8 4 4 34 0 12 16 8 10 4

PC

128 Ch. 1. Linear eqs.

Final Tableau − Inconsistency discoveredBV x1 x2 x3 x4 x5 RHS Eq. no.x2 0 1 −7 −10 −4 −1 1

x1 1 0 3 4 2 52

2

0 0 0 0 0 −1 IE 3

0 0 0 0 0 0 4

Since the 3rd row in the final tableau leads to the inconsistent equa-tion “0 = −1”, we terminate with the conclusion that the originalsystem has no solution.

Optimal Selection of Pivot Columns and Rows inthe GJ Method

Whenever Step 2.3 is carried out, we need to select one of thecolumns with a nonzero coefficient in the pivot row in the presenttableau as the pivot column. How is the choice made?

In the GJ method we perform exactly one pivot step in each rowof the tableau. In the description of the method given above, we haveselected the rows of the tableaus as pivot rows in the order 1 to n; i.e.,if Row r is the pivot row in the present tableau, we are taking Rowr + 1 (or the first row different from the “0=0” equation below r) asthe pivot row in the next tableau. Is this the best order or is there anyadvantage to be gained by using a different order?

We will provide brief answers to these questions now. But theanswer depends on whether the situation under consideration is a smallproblem being solved by hand, or a large problem being solved on adigital computer. First, we review the main differences between thesetwo situations.

When the pivot element is different from 1, division by the pivotelement may generate fractions, and the calculations involved in thepivot step get very tedious to do manually. So, when all the data inthe present tableau is integer, and the system is being solved by hand,

1.16 GJ Method 129

we try to select the pivot column and row so that the pivot element is±1 as often as possible, or at least an integer with the smallest possibleabsolute value so that the fractions generated in the pivot step will notbe too tedious. This is what you should do in solving the exercisesgiven in this book (i.e., for the next pivot step select the pivot rowamong the rows in which a pivot step has not been carried out so far,and the pivot column among the columns in which a pivot step hasnot been carried out so far, so that the pivot element is ±1 as often aspossible, or an integer with the smallest absolute value).

Of course real world problems are always solved using digital com-puters. When writing a computer program for the GJ eliminationmethod, the tediousness of the fractions generated is no longer an im-portant consideration. Other highly critical factors become extremelyimportant. Digital computation is not exact, as it involves finite preci-sion arithmetic. For example, in computing the ratio 1

3, digital compu-

tation would choose the answer to be 0.33 or 0.333 etc. depending onthe number of digits of accuracy maintained. This introduces round-off errors which keep on accumulating as the method progresses, andit may cause the computed solution to be too far away from the un-known actual solution. In writing a computer program for the GJmethod, the pivot row, pivot column choice is handled by very specialrules to keep the accumulation of round-off errors in check as far aspossible. Keeping round-off errors in check is the focus of the subjectNumerical Analysis, which is outside the scope of this book; but wewill give a summary of the most commonly used strategies called fullor complete pivoting that work well.

Full Pivoting: For the next pivot step in the present tableau,select the pivot element to be an element of maximum absolute valueamong all the elements in the present tableau contained in rows inwhich a pivot step has not been carried out so far, and in columnsin which a pivot step has not been carried out so far; breaking tiesarbitrarily. The pivot row, pivot column for this pivot step are therow, column respectively of the pivot element.

In some applications, the GJ method is implemented in a way that

130 Ch. 1. Linear eqs.

in each pivot step, the pivot column is chosen first by some extraneousrule, and then the pivot row has to be selected among the rows in whicha pivot step has not been carried out so far. In such implementations,the strategy that is commonly used is to find a maximum absolutevalue element in the present tableau among the entries in the pivotcolumn contained in rows in which a pivot step has not been carriedout so far, and select its row as the pivot row. This strategy is calledpartial pivoting.

Round-off error accumulation is a serious problem in the GJ methodthat limits its applicability to solve the very large scale systems thatarise sometimes in real world applications. The Gaussian eliminationmethod to be discussed in Section 1.20, implemented with other highlysophisticated numerical analysis techniques, is able to alleviate theround-off error problem somewhat better in such applications.

Obtaining the “0 = 1” and “0 = 0” Equations En-countered in GJ Method As Linear Combinationsof Original Equations

In the description of the GJ method given above, whenever wediscovered the “0 = br” equations, we did not get the “evidence”, i.e.,the vector of coefficients in the expression of this equation as a linearcombination of equations in the original system. In some applicationsit may be necessary to get these expressions. For this, the memorymatrix should be introduced in the original tableau in Step 1 beforebeginning any work on the GJ method, as indicated in Section 1.12,1.13.

Discussion on Some Features of the GJ Method

Here we summarize the main features of the GJ method. Some ofthese have already been mentioned above.

1. When applied on a system of m equations in n variables, themethod performs at most one GJ pivot step in each row for atotal of at most m pivot steps.

1.16 GJ Method 131

2. Whenever a “0 = 0” equation is encountered in the present tableau,a linear dependence relation among the equations in the originalsystem has been identified. The row containing that equationcorresponds to a redundant equation, and is eliminated from thepresent tableau and the original tableau.

Thus the method identifies and eliminates redundant constraintsone by one.

3. Whenever a “0 = br” (for some br �= 0) equation is encounteredin the present tableau, we conclude that the original system isinconsistent, i.e., it has no solution.

If no inconsistency is encountered during the method (as evi-denced by a “0 = br” equation for some br �= 0) the methodterminates with a solution for the system. This proves Result1.10.1 discussed in Section 1.10.

4. Suppose the original system of equations is consistent, and thereare p rows remaining in the final tableau. The method selects abasic variable for each of these rows, The basic variable in theith row is called the ith basic variable, its column vector in thefinal tableau is the ith unit vector in Rp.

Rearrange the variables and their column vectors in the finaltableau so that the basic variables appear first in the tableauin their proper order, followed by nonbasic variables if any (inany order). Also rewrite the original tableau with the variablesarranged in the same order.

For instance, when we do this rearrangement for the systems inExamples 2, 1 above, we get the following:

132 Ch. 1. Linear eqs.

Final tableau in Example 2 afterrearranging variables

Basic var. x3 x2 x1 RHSx3 1 0 0 19

6

x2 0 1 0 116

x1 0 0 1 43

Remaining Originaltableau for Example 2x3 x2 x1 RHS1 1 1 51 3 −1 61 3 2 10

Final tableau in Example 1 after rearrangingvariables

Basic var. x3 x2 x5 x1 x4 RHSx3 1 0 0 −5 5 −40x2 0 1 0 −9 9 −60x5 0 0 1 2 −2 25

Remaining original tableau forExample 1

x3 x2 x5 x1 x4 RHS1 0 2 −1 1 102 −1 1 1 −1 50 0 1 2 −2 25

In the same way, for a general system, after rearrangement ofvariables, the final tableau will be of the following forms. Weassume that the basic variables are xi1 , . . . , xip in that order andthe remaining nonbasic variables are, if any, xj1, . . . , xjn−p.

1.16 GJ Method 133

Then, the vector of variables (xi1 , . . . , xip) is called the ba-sic vector of variables selected by the method for the system.There may be several basic vectors of variables for the system,the one obtained in the method depends on the choice of pivotcolumns selected in Steps 2.3 during the method.

When no nonbasic variables leftFinal tableau after rearrangement

of variablesBasic xi1 xi2 . . . xip

Vars.

xi1 1 0 . . . 0 b1

xi2 0 1 . . . 0 b2...

......

......

xip 0 0 . . . 1 bp

When nonbasic variables existFinal tableau after rearrangement of variables

Basic xi1 xi2 . . . xip xj1 . . . xjn−p

Vars.xi1 1 0 . . . 0 a1,j1 . . . a1,jn−p b1

xi2 0 1 . . . 0 a2,j1 . . . a2,jn−p b2...

......

......

......

xip 0 0 . . . 1 ap,j1 . . . ap,jn−p bp

In both cases, since each row in the final tableau has a basicvariable which only appears in that row and none of the otherrows, the only linear combination of the constraints in the finaltableau that leads to the “0 = 0” equation is the one in which thecoefficients of all the constraints are zero. Hence the system ofconstraints in the final tableau in both cases is linearly indepen-dent. Since the system of constraints in the remaining originaltableau (after the elimination of redundant constraints) is equiv-

134 Ch. 1. Linear eqs.

alent to the system in the final tableau, it is linearly independentas well.

In the first case, (which occurs when all the variables are basic,and there are no nonbasic variables) the final tableau consists ofthe unit columns arranged in their proper order, and the uniquesolution of the system is:

(xi1 , . . . , xip)T = (bi1 , . . . , bip)

T

as can be read from the equations corresponding to the rows inthe final tableau. The solution set consists of a single point andwe say that its dimension is zero. In this case the final sys-tem and the remaining original system at the end (i.e., after theelimination of redundant constraints one by one until the finalsystem has no more redundant constraints) have the same num-ber of equations as variables, which are therefore called squaresystems of linear equations. Also , since they are linearlyindependent, they are called nonsingular square systems oflinear equations.

In the second case, (which occurs when there is at least one non-basic variable in the final tableau) there are an infinite numberof solutions to the system. By writing the equations correspond-ing to the rows in the final tableau, and then transferring allthe terms involving the nonbasic variables to the right in eachequation, we get the general solution to our system as:

xi1

xi2...

xip

xj1...

xjn−p

=

b1 − a1,j1xj1 − . . . − a1,jn−pxjn−p

b2 − a2,j1xj1 − . . . − a2,jn−pxjn−p

...bp − ap,j1xj1 − . . . − ap,jn−pxjn−p

xj1...

xjn−p

where xj1, . . . , xjn−p, the values given to the nonbasic (indepen-dent) variables are arbitrary real parameters. In this represen-

1.16 GJ Method 135

tation for the general solution of the system, the variables maynot be in the usual order of x1 to xn, but the variables can bereordered if desired. Except for the order of the variables, thisrepresentation is exactly the same parametric representation forthe general solution as discussed in Section 1.8. By giving theseparameters xj1 , . . . , xjn−p appropriate real values in this repre-sentation, all possible solutions of the system can be generated.That’s why in this case the dimension of the set of solutionsis defined to be equal to the number of nonbasic (i.e., indepen-dent) variables in the final tableau. Also in this case, the numberof equations in the final system is strictly less than the numberof variables. Hence the final system and the remaining origi-nal system are rectangular systems with more variables thanequations.

A Simple Data Structure

The phrase data structure refers to a strategy for storing infor-mation generated during the algorithm, and using it to improveits efficiency. The idea of recording, after a pivot step is com-pleted, the variable corresponding to the pivot column as thebasic variable in the pivot row, is a very simple data structureintroduced by G. Dantzig in his pioneering 1947 paper on thesimplex method for linear programming. It helps to keep trackof the unit column vectors created by the pivot steps carried outin the algorithm, and makes it easier to directly read a solutionof the system from the final tableau as described in Step 3.

When the system is consistent, the final tableau will have a basicvariable associated with each remaining row in it; and the columnassociated with the rth basic variable will be the rth unit vectorin the final tableau for all r. Dantzig used the term canonicaltableau for the final tableau with a full set of unit vectors in it.

In describing the GJ elimination method, mathematics books onlinear algebra usually state that in each step the pivot columnis selected as the leftmost of the eligible columns. When the

136 Ch. 1. Linear eqs.

system is consistent and the method is carried out this way, thefinal tableau will have a full set of unit vectors occuring in theirproper order all at the left end of the tableau, without the needfor rearranging the variables and column vectors as mentionedabove. The final tableau with this property is said to be in RREF(reduced row echelon form) in mathematical linear algebraliterature. So RREF refers to a specific type of canonical tableauwith all the unit column vectors appearing in their proper orderat the left end of the tableau.

In fact it is not important that all the unit vectors in the finaltableau appear at the left end, or in their proper order. Allowingthe method to generate any canonical tableau (and not just theRREF) introduces the possibility of pivot column choice withouthaving to choose it always as the leftmost of the eligible.

Basic Solution

In linear programming literature the final tableau obtained here(whether the column vectors and the variables are rearranged asdiscussed above or not) is commonly known as the canonicaltableau for the original system WRT the basic vector(xi1 , . . . , xip).

In the general solution for the system obtained above, if we fixall the nonbasic variables at zero, we get the solution

xi1

xi2...

xip

xj1...

xjn−p

=

b1

b2...bp

0...0

known as the basic solution for the system WRT the basicvector (xi1 , . . . , xip).

1.16 GJ Method 137

When we put the column vectors of the basic variables xi1 , . . . , xip

in the remaining original tableau after all the redundant con-straints are deleted, side by side in that order, we get a p×p squarearray of numbers which is known as the basis matrix or basisfor the system corresponding to the basic vector (xi1 , . . . , xip).The general concept of a matrix as a two-dimensional array ofnumbers is given later in Chapter 2.

There is another worked out example after this discussion.

5. One consequence of the results in 4. is that any consistent sys-tem of linear equations is either directly itself or is equivalent toanother in which the number of equations is ≤ the number of vari-ables. If a consistent system has more equations than variables,then some of them can be eliminated as redundant equationsleading to an equivalent system in which the above condition issatisfied. Therefore, when discussing a consistent system of equa-tions, we can always, without loss of generality, assume that thenumber of equations is ≤ the number of variables.

6. On a system of m equations in n variables, in the worst case allthe m(n + 1) entries may change in a pivot step, requiring up to3m(n + 1) multiplication, division, or addition operations. Sincethe method may take up to m pivot steps on this system, it mayrequire at most 3m2(n + 1) arithmetical operations.

The computational complexity of a method is a measure ofthe worst case computational effort needed by the method as afunction of the size of the system as measured by the numberof data elements, or the total number of binary digits of storagespace needed to store all the data, etc. In giving the computa-tional complexity, constant terms like 3 in 3m2(n+1) are usuallyomitted, and then we say that the computational complexity is ofOrder m2n and write it as: O(m2n). It provides the order of anupper bound on the computational effort needed by the methodin terms of the size of the problem. So, we can say that the com-putational complexity of the GJ method to solve a system of mlinear equations in n variables is O(m2n).

138 Ch. 1. Linear eqs.

Example 4: We apply the GJ method to solve the followingsystem of 5 linear equations in 5 variables. The detached coefficienttableau for the original system is the one at the top. PR (pivot row),PC (pivot column), RC (redundant constraint to be eliminated), BV(basic variable selected in this row) have the same meanings as before,and in each tableau the pivot element for the pivot step carried out inthat tableau is boxed.

x1 x2 x3 x4 x5 RHS Eq. no.

0 1 1 −1 −2 3 PR 11 0 1 1 −1 9 21 1 0 −1 1 15 31 1 2 0 −3 12 42 2 2 −1 −2 27 5

PC↑BV x1 x2 x3 x4 x5 RHS Eq. no.x3 0 1 1 −1 −2 3 1

1 −1 0 2 1 6 PR 21 1 0 −1 1 15 31 −1 0 2 2 6 42 0 0 1 2 21 5

PC↑x3 0 1 1 −1 −2 3 1x1 1 −1 0 2 1 6 2

0 2 0 −3 0 9 PR 30 0 0 0 0 0 40 2 0 −3 0 9 5

PC↑x3 0 0 1 1

2−2 −3

21

x1 1 0 0 12

1 212

2

x2 0 1 0 −32

0 92

3

0 0 0 0 0 0 RC 4

0 0 0 0 0 0 RC 5

1.16 GJ Method 139

Final tableaux1 x2 x3 x4 x5 RHS Eq. no.

BV x1 x2 x3 x4 x5 RHS Eq. no.x3 0 0 1 1

2−2 −3

21

x1 1 0 0 12

1 212

2

x2 0 1 0 −32

0 92

3

So, equations 4 and 5 are eliminated as redundant constraints. Theremaining system of original constraints, after these redundant equa-tions are deleted, is:

Remaining Original system after deletion of redundant eqs.x1 x2 x3 x4 x5 RHS Eq. no.0 1 1 −1 −2 3 11 0 1 1 −1 9 21 1 0 −1 1 15 3

The basic vector of variables selected is (x3, x1, x2). The basismatrix corresponding to this basic vector is defined to be the square twodimensional array of numbers obtained by putting the column vectorsof the basic variables x3, x1, x2 in the remaining original system afterthe deletion of redundant constraints, in that order, which is:

B =

1 0 11 1 00 1 1

The general concept of a matrix is discussed in Chapter 2. Fromthe final tableau, we find that the general solution of the system is:

140 Ch. 1. Linear eqs.

x1

x2

x3

x4

x5

=

212−

12 x4 − x5

92

+ 32 x4

− 32−

12 x4 + 2x5

x4

x5

where x4, x5 are real valued parameters representing the values of thefree (nonbasic) variables x4, x5 in the solution. Making x4 = 0, x5 = 0gives the basic solution of the system associated with the basic vectorof variables (x3, x1, x2), which is:

x1

x2

x3

x4

x5

=

212

92

− 32

0

0

By selecting different basic variables in the various pivot steps, wecan get different basic vectors of variables for the system.

Since there are two nonbasic variables in the final tableau in thisexample, the dimension of the set of solutions in this example is 2.

Exercises

In solving these exercises by hand you may want to use pivot ele-ments of ±1 as far as possible to avoid generating fractions that canmake the work messy.

1.16 GJ Method 141

1.16.1: Systems with a unique solution: solve the following systemsby the GJ method and verify that the solution obtained is the uniquesolution of the system.

(a) x1 x2 x3 x4

−1 −2 0 0 12 5 2 2 −30 1 3 0 −41 0 −1 1 32 −2 1 −1 2

(b) x1 x2 x3

3 1 4 5−4 −1 −6 −4−2 −1 0 −10

(c) x1 x2 x3 x4

1 2 3 2 5−1 −3 −5 −1 −4

1 0 4 1 111 2 −2 −2 −13

(d) x1 x2 x3

1 1 1 5−1 3 1 6

0 4 2 112 3 1 10

1.16.2: Systems with multiple solutions, but no redundant con-straints: Solve the following systems by the GJ method. For eachobtain an expression for the general solution of the system, and findthe dimension of the set of all solutions. From the general solutioncheck whether you can obtain a nonnegative solution for the systemby trying to generate nonnegative values to the nonbasic variables thatwill make all basic variables nonnegative using simple trial and error.

142 Ch. 1. Linear eqs.

(a) −x2 + 3x3 + 8x4 − 7x6 = −3

7x1 + x2 − 2x3 − 10x4 + 2x5 − 3x6 = −4

10x1 + 2x3 + 12x4 + 5x5 − 8x6 = 2

(a) x1 x2 x3 x4 x5 x6

1 1 0 −1 −1 1 −3−1 0 1 2 −1 2 −7

0 3 2 3 −5 1 −101 1 0 2 −1 −2 8

(c) x1 x2 x3 x4 x5 x6 x7

2 −1 0 1 2 −3 −2 1−3 1 −1 0 1 −2 −1 −4

2 0 3 −3 −1 0 2 70 2 1 −1 0 2 1 7

(d) x1 x2 x3 x4 x5 x6 x7

1 −2 0 1 0 −1 6 41 −1 2 −2 1 8 −17 −10 2 5 1 −5 3 −4 3

1.16.3: Systems with multiple solutions and redundant constraints:Solve the following systems by the GJ method, and obtain an expres-sion for the general solution, and the dimension of the solution set.

(a) x1 x2 x3 x4 x5 x6

−1 0 1 −1 −1 2 −10 −1 1 0 1 1 2

−2 −3 5 −2 1 7 41 1 −3 2 0 −2 −2

−2 −3 4 −1 1 8 32 2 −4 1 2 −1 −4

1.16 GJ Method 143

(b) x1 x2 x3 x4 x5 x6 x7

2 0 3 0 4 5 −1 −100 3 −2 5 −3 2 1 104 6 2 10 2 14 0 03 −3 2 1 −1 −1 −1 −79 6 5 16 2 20 −1 −75 0 3 6 0 6 −1 −7

(c) x1 x2 x3 x4 x5

−2 7 5 1 4 38−3 2 −2 2 9 84−8 11 1 5 22 206−7 16 8 4 17 160−5 9 3 3 13 122

(d) x11 x12 x13 x21 x22 x23

1 1 1 0 0 0 8000 0 0 1 1 1 3001 0 0 1 0 0 4000 1 0 0 1 0 2000 0 1 0 0 1 500

(e) x1 x2 x3 x4 x5

−1 0 1 1 2 101 −1 2 −1 1 51 −2 5 −1 4 202 0 0 −2 1 253 −2 5 −3 5 455 −2 5 −5 6 70

1.16.4: Infeasible systems: solve the following systems by the GJmethod.

144 Ch. 1. Linear eqs.

(a) x1 x2 x3 x4 x5 x6

1 2 −1 0 3 1 3−1 −3 2 2 −2 −3 4

0 −1 1 2 1 −2 7−1 −4 3 4 −1 −5 9

1 1 0 2 4 −1 7

(b) x11 x12 x13 x21 x22 x23

1 1 1 0 0 0 5000 0 0 1 1 1 10001 0 0 1 0 0 6000 1 0 0 1 0 5000 0 1 0 0 1 500

(c) x1 x2 x3 x4 x5

1 1 0 0 0 30 1 1 0 0 10 0 1 1 0 10 0 0 1 1 41 0 0 0 1 51 1 1 1 1 71 2 1 1 1 9

(d) −5x1 − 2x2 + 10x3 + x4 + 7x5 + 2x6 − 4x7 = 2

−3x1 + 3x2 − 2x3 − x4 + 2x5 + x6 + 3x7 = 3

−8x1 + x2 + 8x3 + 9x5 + 3x6 − x7 = 6

x1 + 2x3 − 3x5 − 4x6 + 2x7 = 5

−7x1 + x2 + 10x3 + 6x5 − x6 + x7 = 8

(e) x1 x2 x3 x4 x5 RHS2 1 −1 −2 0 44 1 5 6 4 96 2 4 4 4 12

10 3 9 10 8 22

1.17 Theorem of Alternatives 145

1.17 A Theorem of Alternatives for Sys-

tems of Linear Equations

Consider a system of linear equations. In the previous section, we haveseen that the GJ method will always find a solution for this systemif no inconsistent equation of the form “0 = α” for some α �= 0 isencountered during the method.

If a “0 = α” equation for some α �= 0 is found during the method,by dividing that equation by α we get the “0 = 1” equation. Thisprovides a proof of Result 1.10.1 of Section 1.10 which states that asystem of linear equations has no solution iff the “0 = 1” equation canbe obtained as a linear combination of equations in the system.

Consider the general system of m equations in n variables x =(x1, . . . , xn)T which is reproduced below in detached coefficient form.

General systemx1 . . . xj . . . xn RHS constantsa11 . . . a1j . . . a1n b1...

......

...ai1 . . . aij . . . ain bi...

......

...am1 . . . amj . . . amn bm

Suppose we take a linear combination of these equations by multi-plying the ith equation in this tableau by πi and summing over i = 1to m. The π = (π1, . . . , πm) is the vector of coefficients in this linearcombination. This results in the following equation:

n∑j=1

(π1aij + . . .+πiaij + . . .+πmamj)xj = (π1b1 + . . .+πibi + . . .+πmbm)

This equation will be the “0 = 1” equation iff the coefficient of xj

in it, π1a1j + . . .+πmamj is 0 for all j = 1 to n, and π1b1 + . . .+πmbm =1. That is, this linear combination is the “0 = 1” equation iff the (πi)satisfy the following system of equations in detached coefficient form,

146 Ch. 1. Linear eqs.

which is known as the alternate system corresponding to the generalsystem given above.

Alternate systemπ1 . . . πi . . . πm RHSa11 . . . ai1 . . . am1 0...

......

...a1j . . . aij . . . amj 0...

......

...a1n . . . ain . . . amn 0b1 . . . bi . . . bm 1

Result 1.10.1 of Section 1.10 is often stated as a mathematical the-orem about systems of linear equations in the following way:

Theorem of Alternatives for Systems of Linear Equations:Consider the general system of m equations in n variables x = (x1, . . . ,xn)T given above. Using the same data in this system, we can con-struct the alternate system of (n + 1) equations in m new variablesπ = (π1, . . . , πm) as shown above, such that the following propertyholds: the original system has no solution x iff the alternate systemhas a solution π.

As explained in Section 1.13, any solution of the alternate systemprovides an evidence for the infeasibility of the original system.

As stated above, if the original system is solved by the GJ methodafter introducing the memory matrix in it, when the method terminateswe will either have a solution of the original system, or of the alternatesystem.

As an example, consider the following system of 4 equations in 5variables.

1.17 Theorem of Alternatives 147

Original systemx1 x2 x3 x4 x5 RHS0 0 1 0 0 31 −2 2 −1 1 −8

−1 0 4 −7 7 160 −2 6 −8 8 6

The alternate system will have 4 variables π1, π2, π3, π4. It is:

Alternate systemπ1 π2 π3 π4 RHS0 1 −1 0 00 −2 0 −2 01 2 4 6 00 −1 −7 −8 00 1 7 8 03 −8 16 6 1

Let us denote the equations in the original system R1 to R4 from topto bottom. We introduce the memory matrix as discussed in Section1.12, and apply the GJ method on it. In each step the PR (pivot row),PC (pivot column) are indicated and the pivot element is boxed; “BV”is “basic variable selected in row”, and “IE” indicates an inconsistentequation discovered.

BV Memory matrix RHSR1 R2 R3 R4 x1 x2 x3 x4 x5

1 0 0 0 0 0 1 0 0 3 PR0 1 0 0 1 −2 2 −1 1 −80 0 1 0 −1 0 4 −7 7 160 0 0 1 0 −2 6 −8 8 6

PC↑x3 1 0 0 0 0 0 1 0 0 3

−2 1 0 0 1 −2 0 −1 1 −14 PR−4 0 1 0 −1 0 0 −7 7 4−6 0 0 1 0 −2 0 −8 8 −12

PC↑

148 Ch. 1. Linear eqs.

BV Memory matrix RHSR1 R2 R3 R4 x1 x2 x3 x4 x5

x3 1 0 0 0 0 0 1 0 0 3x1 −2 1 0 0 1 −2 0 −1 1 −14

−6 1 1 0 0 −2 0 −8 8 −10 PR−6 0 0 1 0 −2 0 −8 8 −12

PC↑x3 1 0 0 0 0 0 1 0 0 3x1 4 0 −1 0 1 0 0 7 −7 −4x2 3 −1/2 −1/2 0 0 1 0 4 −4 5

0 −1 −1 1 0 0 0 0 0 −2 IE

The method terminates with the conclusion that the original systemhas no solution because of the inconsistent equation “0 = −2” in Row 4 ofthe final tableau. From the corresponding row of the memory matrix weconclude that the “0 = −2” equation is the linear combination −R2 −R3 + R4 of equations in the original system. Dividing by −2, we concludethat the linear combination

12R2 +

12R3 − 1

2R4

yields the inconsistent equation “0 = 1”. Hence the coefficient vector of thislinear combination, π = (0, 12 , 12 ,−

12) is a solution of the alternate system,

which can be verified.

In the same way, for any system of linear equations, we can write thealternate system, and obtain a solution for it if the original system is incon-sistent, by applying the GJ method on the original system with the memorymatrix.

Ex ercis es

1.17.1: For each of the following systems do the following: (a) Write thealternate system, (b) Solve the original system by the GJ method using thememory matrix, and obtain a solution for it if it is consistent, or a solutionfor the alternate system if the original system is inconsistent.

1.18 Triangular Systems 149

(i): x1 − 2x2 + 2x3 − x4 + x5 = −8−x1 + 4x3 − 7x4 + 7x5 = 16−2x2 + 6x3 − 8x4 + 8x5 = 6

(ii): x1 + x2 + x3 + x4 = 3−x1 − 2x2 + 2x4 = −4x3 − x4 = 42x1 + 3x2 + x4 = 5

(iii): x1 + x2 − x3 − x4 = −5−2x2 + x3 − x4 = −6x1 − 2x2 + 2x4 = 42x1 − 3x2 = −4

1.18 Special Triangular Systems of Lin-

ear Equations

In this section we consider square nonsingular systems of linear equationswith the special triangular property, and a very efficient method forsolving them.

Consider the general square system of n equations in n unknowns indetached coefficient form.

x1 x2 . . . xn−1 xn RHSa11 a12 . . . a1,n−1 a1n b1

a21 a22 . . . a2,n−1 a2n b2 Abovediagonalpart

Belowdiagonalpart

......

......

...

an1 an2 . . . an,n−1 ann bn

For i = 1 to n, the boxed coefficients aii in Row i and Column i ofthis tableau are known as the diagonal entries in this tableau. Theyconstitute the principal diagonal or the main diagonal of the tableau.

Remember that the concept of diagonal entries and diagonal are only

150 Ch. 1. Linear eqs.

defined for square systems. There is no such concept for rectangular systemswhich are not square.

Ignoring the RHS constants vector, the coefficients of the variables inthis square tableau are partitioned into three parts:

the diagonal part consisting of aii for i = 1 to n

below the diagonal part consisting of aij for i > j

above the diagonal part consisting of aij for i < j.

The square system is said to be an:

Upper triangular system If all diagonal elements nonzero, and be-low diagonal part is all zero; i.e., aii �= 0for all i, and aij = 0 for all i > j.

Lower triangular system If all diagonal elements nonzero, andabove diagonal part is all zero; i.e., aii �= 0for all i, and aij = 0 for all i < j.

Triangular system If the columns (or rows) of the tableau canbe rearranged so that the system becomesan upper triangular system.

Here are some examples.

Upper triangular systemx1 x2 x3 x4

1 1 −1 1 −100 −1 −2 1 −200 0 1 1 200 0 0 −1 10

Lower triangular systemx1 x2 x3 x4 x5

−1 0 0 0 0 51 −1 0 0 0 −5

−1 2 1 0 0 101 1 −1 1 0 10

−1 2 1 2 −1 −10

Triangular systemx1 x2 x3 x4

0 0 0 −1 101 1 −1 1 −100 0 1 1 200 −1 −2 1 −20

1.18 Triangular Systems 151

The last system is triangular, because by arranging the rows in the order:Row 2, Row 4, Row 3, Row 1 it becomes the upper triangular first system.

The Back substitution Method for Solving anUpper Triangular System of Equations

Consider the general upper triangular system of equations given below.Here we know that aii �= 0 for all i, but some of the other coefficients maybe zero.

x1 x2 . . . xn−1 xn RHSa11 a12 . . . a1,n−1 a1n b1

0 a22 . . . a2,n−1 a2n b2...

......

......

0 0 . . . an−1,n−1 an−1,n bn−1

0 0 . . . 0 ann bn

From the equation represented by the last row we get

xn =bn

ann

in the solution. Then from the (n − 1)th equation, we can get the value ofxn−1 in the solution by substituting the value of xn from above, in it.

In the same way, substituting the values of xj+1, . . . , xn in the solutionin the jth equation, we can get the value of xj in the solution, for j =n − 1, . . . , 1, in that order. Hence, using the equations in the tableau fromthe bottom to the top in that order, the variables are evaluated in the orderxn to x1; this is the reason for the name back substitution method.

For an illustration, we will solve the upper triangular system given above,which is reproduced below.

Upper triangular systemx1 x2 x3 x4

1 1 −1 1 −100 −1 −2 1 −200 0 1 1 200 0 0 −1 10

The last equation, “−x4 = 10” yields x4 = −10.

152 Ch. 1. Linear eqs.

The 3rd equation “x3+x4 = 20” yields x3 = 20−x4 = 20−(−10) = 30.The 2nd equation “−x2 −2x3 +x4 = −20” yields x2 = 20−2x3 +x4 =

20 − 2(30) + (−10) = −50.The 1st equation “x1+x2 −x3+x4 = −10” yields x1 = −10−x2+x3+x4

= −10 − (−50) + 30 − (−10) = 80.So the solution of the system is (x1, x2, x3, x4)T = (80,−50, 30,−10)T .In the same way, a lower triangular system of equations can be solved

by using the equations in the tableau from top to bottom in that order, toevaluate the variables in the order x1 to xn. Commonly, this is also calledback substitution method, but some textbooks refer to it as the forwardsubstitution method.

A triangular system must satisfy the following property:

Prop erty of a t riangular syst e m: There exists an equation in whichonly one variable has a nonzero coefficient.

To find the solution of a triangular system, we identify the equationin it satisfying the above property, and use it to get the value of the onlyvariable, xs say, with a nonzero coefficient in it. Eliminate this equation fromthe system, and eliminate the variable xs from the system by substitutingits value in all the other equations.

The remaining system is also triangular, and hence it must also satisfythe above property, and hence the above step can be applied on it. Theprocess continues in the same way until the values of all the variables in thesolution are obtained.

Exercises

1.18.1 The tower of Brahma (or The tower of Hanoi) puzzle:This puzzle consists of 3 pegs, and n rings of different sizes stacked on oneof them (in the traditional puzzle, n = 64). The rings are arranged indecreasing order of size with the largest one at the bottom and the smallestone at the top of the stack. The problem is to move the rings one at a timefrom one peg to another, never putting a larger one on a smaller one, andeventually transferring all 64 rings from one peg to another peg. Let

xn = the minimum number of moves required to arrange the nrings in the same decreasing order of size on another peg, whilenever putting a larger ring on a smaller ring.

1.18 Triangular Systems 153

(x1, . . . , x8)T can be shown to be a solution of the following system oflinear equations.

x1 = 1x2 − 2x1 = 1x3 − 2x2 = 1x4 − 2x3 = 1x5 − 2x4 = 1x6 − 2x5 = 1x7 − 2x6 = 1x8 − 2x7 = 1

and in general xn+1 − 2xn = 1 for all integers n ≥ 1. Write the coefficientmatrix of the system of equations for n = 1 to 8 given above, and verify thatit is lower triangular. Obtain the solution of the system by back substitution.

Show that in general xn = 2n − 1.

1.18.2 Verify whether the following systems of equations are triangular,if so, obtain their solution by back substitution. If not explain why they arenot triangular. (Here (i), (ii) are not triangular, while (iii), (iv) are.)

(i) x1 + x2 + 3x3 − x4 + x5 = 102x3 + x4 = 6x3 + 2x4 − x5 = 72x5 = 10x5 = 5

(ii) 2x1 + 3x2 + 2x3 − 2x4 = 52x3 + 2x4 = 6x1 + 2x4 = 3

(iii) 3x3 − 3x4 + 6x5 = 30x5 = −22x2 − 2x3 − x4 + x5 = −832x4 + 3x5 = 104x1 − 4x4 − 8x5 = 12

154 Ch. 1. Linear eqs.

(iv) x11 + x12 = 3x21 + x23 = 5x32 + x34 = 9x43 = 10x11 + x21 = 4x12 + x32 = 6x23 + x43 = 12

1.19 The Gaussian Pivot Step

Consider the general system of m equations in n variables in detached coef-ficient tableau form. Like the GJ pivot step discussed so far, the G (Gaus-sian) pivot step on this tableau is also specified by a row of the tableauselected as the pivot row, and a column of the tableau selected as the pivotcolumn for it, with the pivot element (element in the pivot row and pivotcolumn) nonzero. The G pivot step consists of a series of row operations onthe tableau to convert the entries in the pivot column below the pivot row tozero. Notice that the pivot row and all the rows above it remain unchangedin a pivot step. Thus a G pivot step may involve a lot less work than aGJ pivot step with the same pivot row and pivot column, particularly ifthe pivot row is well below the top of the tableau. Using abbreviations PR(pivot row), PC (pivot column), we give the operations for a general system.

Original Tableaux1 x2 . . . xs . . . xn RHS constantsa11 a12 . . . a1s . . . a1n b1

a21 a22 . . . a2s . . . a2n b2...

......

......

ar1 ar2 . . . ars . . . arn br PR...

......

......

am1 am2 . . . ams . . . amn bm

PC

The G pivot step involves the following row operations:

for each i > r, Row i - ( aisars

)Row r.

Since the G pivot step involves division by the pivot element, it is possibleonly if the pivot element is nonzero. Denoting the entries in the tableau after

1.19 The Gaussian (G) Pivot Step 155

the G pivot step by aij, bi if they do not change, or by aij , bi if they do, wehave:

Tableau after the G pivot stepx1 x2 . . . xs . . . xn RHS constantsa11 a12 . . . a1s . . . a1n b1

a21 a22 . . . a2s . . . a2n b2...

......

......

ar1 ar2 . . . ars . . . arn br

ar+1,1 ar+1,2 . . . 0 . . . ar+1,n br+1...

......

......

am1 am2 . . . 0 . . . amn bm

We provide a numerical example. The original tableau is at the topwith PR, PC indicated and the pivot element boxed. The row operationsperformed on the original tableau in this G pivot step are: Row 4 −(4

2)Row3, Row 5 −(−6

2 )Row 3. The tableau after the G pivot step is given at thebottom.

Original tableaux1 x2 x3 x4 x5 x6

13 −12 11 0 −8 14 33−70 −80 33 −84 −17 8 −45−2 0 4 −2 2 6 −10 PR

2 1 −2 1 4 3 10−3 −1 1 3 −6 0 8

PCTableau after the G pivot step

13 −12 11 0 −8 14 33−70 −80 33 −84 −17 8 −45−2 0 4 −2 2 6 −10 PR

6 1 −10 5 0 −9 30−9 −1 13 −3 0 18 −22

We leave it to the reader to verify that this G pivot step may involvemuch less work than a GJ pivot step using the same pivot element.

156 Ch. 1. Linear eqs.

Exercises

1.19.1 Perform the G pivot steps indicated by the pivot row (PR), pivotcolumn (PC) indicated on the following tableaues.

PC(i) x1 x2 x3 x4 x5

−2 4 −4 6 2 −12 PR2 1 3 −2 −1 10

−6 3 6 0 2 08 −2 1 −2 3 2

PC(ii) x1 x2 x3 x4

1 −1 −2 3 −40 3 −4 6 73 9 −3 12 6 PR

−2 0 2 1 3

(iii): In the following tableau, a G pivot step needs to be carried outin the pivot column (PC) marked. The pivot row (PR) is determined bythe following procedure: identify the row with entry of maximum absolutevalue in the PC among rows with numbers ≥ 2, and interchange that rowwith row 2. After the row interchange, perform a G pivot step with row 2as the PR.

x1 x2 x3 x4 x5

1 2 0 1 −2 −20−1 3 2 0 −4 4

0 −4 −4 −2 6 −6−18 12 −6 24 6 30

PC

(iv): In the following tableau perform G pivot steps in all the rows inthe order rows 1 to 4, with the diagonal element in that row as the pivotelement; to convert the system into an upper triangular system. Then solvethe resulting system by the back substitution method.

1.20 G Method 157

x1 x2 x3 x4

1 −2 0 1 2−1 1 −2 1 1

0 −1 −1 1 31 2 3 1 0

1.20 The Gaussian (G) Elimination Method

for Solving Linear Equations

This method is based on G pivot steps, and the back substitution methodof Section 1.18 to get the solution from the final tableau if the system turnsout to be consistent.

The GJ method converted the columns of the basic variables into thevarious unit column vectors. The main point in the G method is to convertthe column vectors of the basic variables in their proper order into uppertriangular form so that the values of the basic variables can be computed byback substitution from the final tableau after giving the nonbasic variablesarbitrary values at the user’s choice, and transferring their terms to theright. For this reason, and the nature of the G pivot step, please note thatthe row interchanges mentioned in Steps 2 are necessary depending on themanner in which the next pivot element is choosen.

So, in the G elimination method, we also use the new row operation rowinterchange (interchanging two rows in the tableau) that we have not usedso far in the GJ method.

The G elimination method needs less computational effort than the GJmethod, and hence is the preferred among these two for solving large scaleproblems using computers. It is the most commonly used method for solvinglinear equations in scientific computation.

As in the discussion of the GJ method in Section 1.16, we do not in-troduce here the memory matrix to produce evidences of redundancy andinfeasibility to keep the discussion simple. Also, the version of the methodpresented here is the simplest version. A computationally more efficientversion based on the memory matrix is given in Section 4.11, it has the ad-ditional benefit of producing all evedences without additional computationalburden.

The G Elimination Method

158 Ch. 1. Linear eqs.

BEGIN

Step 1: Put the system in detached coefficient tableau form and obtainthe original tableau. Open a column on the left hand side to record thebasic variables. Give each row its original number (number in the originaltableau) and store this number in a rightmost column. This number will notchange as long as the row remains in the tableau, and moves with the rowif the row is interchanged. This number may be different from the positionof the row in the present tableau because of row interchanges taking placeduring the algorithm, or rows corresponding to redundant constraints beingeliminated.

Set the row index r to 0.

Step 2: Pivot element selection: Among rows in positions (r+1) andbelow (counting from the top) in the present tableau, if there is a “0 = bi”row for some bi �= 0, the original system is inconsistent and has no solution;terminate the method.

Among rows in positions (r+1) and below in the present tableau, if thereis a “0=0” row (redundant constraint) eliminate it and all such rows fromthe present tableau. If all rows in positions (r+1) and below are eliminatedin this process, go to Step 4.

If some rows in positions (r+1) and below remain in the present tableau,each of them has at least one nonzero coefficient. Now select the pivotelement for the next pivot step. One of the following strategies can be usedfor this.

1. Full pivoting: This strategy is used to help minimize accumulationof round-off errors when solving problems on a computer. Select amaximum absolute value coefficient in the present tableau in rows(r + 1) and below, as the pivot element. Suppose this element is ais

in row i and column of variable xs in the present tableau. If i �= r+1,interchange rows r + 1 and i in the present tableau so that the pivotelement is in row r +1 after the interchange. In the resulting tableau,select row r + 1 as the pivot row and the column of xs as the pivotcolumn.

When solving problems by hand, one can use a similar strategy, butselect the pivot element to be an element of value ±1, or one whichis an integer of least absolute value in the same region in order tominimize the fractions generated.——

1.20 G Method 159

2. Partial pivoting: In this strategy, the pivot column is first selected(could be by some other considerations) as a column with at leastone nonzero entry in the present tableau among rows r+1 and below.Suppose it is column s of variable xs. Then we select the pivot elementto be an element of maximum absolute value among rows r + 1 andbelow in the pivot column. If this element is ais in row i and i �= r+1,interchange rows r + 1 and i in the present tableau so that the pivotelement is in row r +1 after the interchange. In the resulting tableau,select row r + 1 as the pivot row, column of xs as the pivot column.

3. Select row r + 1 as the pivot row in the present tableau, and a nozerocoefficient in this row as the pivot element, and its column as thepivot column. This strategy is normally used when solving problemsby hand, in this case we will look for pivot elements which are ±1if possible, or some integer of smallest absolute value to minimizefractions generated in the pivot step.

Step 3: Pivot step: Carry out the G pivot step in the present tableauwith the pivot row, pivot column selected above, and record the variablecorresponding to the pivot column as the basic variable in the pivot row inthe resulting tableau. In this tableau, if row r + 1 is the bottom most row,go with it to Step 4; otherwise increment r by 1 and go back to Step 2.

Step 4: Computing the solution: Let p be the number of rows in thefinal tableau, and xi1 , . . . , xip the basic variables in rows 1 to p in it. Thenxit is the tth basic variable and (xi1 , . . . , xip) is the basic vector selected.

If there are no nonbasic variables at this stage, rearrange the variablesand their columns in the final tableau in the order xi1 , . . . , xip . After thisrearrangement of variables, the final tableau has the following upper trian-gular form:——

When no nonbasic variables leftBasic xi1 xi2 . . . xip−1 xip

Vars.xi1 a11 a12 . . . a1,p−1 a1p b1

xi2 0 a22 . . . a2,p−1 a2p b2...

......

......

...xip−1 0 0 . . . ap−1,p−1 ap−1,p bp−1

xip 0 0 . . . 0 app bp

160 Ch. 1. Linear eqs.

and it can be solved by the back substitution method discussed in Section1.18 to obtain the unique solution of the original system in this case.

If there are some nonbasic variables in the final tableau, rearrange thevariables and their column vectors in the final tableau so that the basic vari-ables appear first in the tableau in their proper order, followed by nonbasicvariables if any (in any order). Then the final tableau assumes the followingform:

When nonbasic variables existFinal tableau after rearrangement

of variablesBasic xi1 xi2 . . . xip xj1 . . . xjn−p

Vars.

xi1 a11 a12 . . . a1p a1,j1 . . . a1,n−p b1

xi2 0 a22 . . . a2p a2,j1 . . . a2,n−p b2...

......

......

......

xip−1 0 0 . . . ap−1,p ap−1,j1 . . . ap−1,n−p bp−1

xip 0 0 . . . app ap,j1 . . . ap,n−p bp

In this case there are an infinite number of solutions to the system, andthe dimension of the solution set is = number of nonbasic variables = n−p.

By giving the nonbasic variables xj1, . . . , xjn−p appropriate real values (0,if you want the basic solution WRT the present basic vector) and transferringall their terms to the right, we are left with an upper triangular system inthe basic variables to compute their values in the corresponding solution byback substitution.

The final tableaus here, after rearrangement of variables as describedabove, are said to be in echelon form or REF (row echelon form) inmathematical literature.

In the original tableau, eliminate all the rows which do not appear amongthe original row numbers for rows in the final tableau (these rows are elim-inated as those corresponding to redundant constraints during the algo-rithm). Now rearrange the remaining rows in the same order as the originalrow numbers in the final tableau. Then rearrange the variables and theircolumn vectors in the same order as those in the final tableaus discussedabove. This gives the remaining original tableau after the deletion of theredundant constraints, corresponding to the final tableau.——

1.20 G Metho d 161

END.

Numerical Examples o f t he G Metho d

We now provide numerical examples. We indicate the pivot row (PR),pivot column (PC), and the boxed pivot element when a G pivot step isperformed on a tableau. RC (redundant constraint to be eliminated), IE(inconsistent equation) have the same meanings as before. We use the ab-breviation “BV” for “basic variable”. The indication “RI to Rt” on a rowmeans that at that stage, that row will be interchanged with Row t in thattableau.

We number all the rows (equations) serially from top to bottom, andrecord these numbers in a rightmost column, to make it easy to keep trackof redundant constraints eliminated.

The original detached coefficient tableau for the original system is alwaysthe first one.

Example 1: In this example, we select the pivot column first, andthen the pivot element in it using the “partial pivot strategy”. Even thoughthe present row may have the maximum absolute value element in the pivotcolumn, we may select the pivot element as another one below it with thesame absolute value to illustrate row interchanges.

BV x1 x2 x3 x4 x5 x6 RHS Eq. no.1 −2 1 1 1 −1 −4 12 2 0 1 −1 −1 10 22 −4 −2 0 −2 −2 −8 RI to R1 30 −6 −2 −1 −1 −1 −18 40 0 0 4 1 1 20 51 4 −1 0 −2 0 14 6

PC↑2 −4 −2 0 −2 −2 −8 PR 32 2 0 1 −1 −1 10 21 −2 1 1 1 −1 −4 10 −6 −2 −1 −1 −1 −18 40 0 0 4 1 1 20 51 4 −1 0 −2 0 14 6

PC↑

162 Ch. 1. Linear eqs.

BV x1 x2 x3 x4 x5 x6 RHS Eq. no.x1 2 −4 −2 0 −2 −2 −8 3

0 6 2 1 1 1 18 20 0 2 1 2 0 0 RI to R2 10 −6 −2 −1 −1 −1 −18 40 0 0 4 1 1 20 50 6 0 0 −1 1 18 6

PC↑x1 2 −4 −2 0 −2 −2 −8 3

0 0 2 1 2 0 0 PR 10 6 2 1 1 1 18 20 −6 −2 −1 −1 −1 −18 40 0 0 4 1 1 20 50 6 0 0 −1 1 18 6

PC↑x1 2 −4 −2 0 −2 −2 −8 3x3 0 0 2 1 2 0 0 1

0 6 0 0 −1 1 18 PR 20 −6 0 0 1 −1 −18 40 0 0 4 1 1 20 50 6 0 0 −1 1 18 6

PCx1 2 −4 −2 0 −2 −2 −8 3x3 0 0 2 1 2 0 0 1x2 0 6 0 0 −1 1 18 2

0 0 0 0 0 0 0 RC 40 0 0 4 1 1 20 50 0 0 0 0 0 0 RC 6

x1 2 −4 −2 0 −2 −2 −8 3x3 0 0 2 1 2 0 0 1x2 0 6 0 0 −1 1 18 2

0 0 0 4 1 1 20 PR 5PC↑

x1 2 −4 −2 0 −2 −2 −8 3x3 0 0 2 1 2 0 0 1x2 0 6 0 0 −1 1 18 2x4 0 0 0 4 1 1 20 5

1.20 G Metho d 163

Final tableau with variables rearrangedBV x1 x3 x2 x4 x5 x6 RHS Eq. no.x1 2 −2 −4 0 −2 −2 −8 3x3 0 2 0 1 2 0 0 1x2 0 0 6 0 −1 1 18 2x4 0 0 0 4 1 1 20 5

Since there are two nonbasic variables in the final tableau, this systemhas an infinite number of solutions, and the dimension of the solution setis two. The remaining original tableau after eliminating the redundant con-straints 4 and 6, and rearranging the rows and the variables in the sameorder as those in the final tableau is given below:

Remaining original tableau with eqs. & variables rearrangedx1 x3 x2 x4 x5 x6 RHS Eq. no.2 −2 −4 0 −2 −2 −8 31 1 −2 1 1 −1 −4 12 0 2 1 −1 −1 10 20 0 0 4 1 1 20 5

After transferring the terms of the nonbasic (free) variables to the right,the final tableau becomes the following upper triangular system in the basicvariables:

Upper triangular system to evaluatebasic variables

x1 x3 x2 x4 RHS2 −2 −4 0 −8 + 2x5 + 2x6

0 2 0 1 −2x5

0 0 6 0 18 + x5 − x6

0 0 0 4 20 − x5 − x6

Giving the nonbasic variables x5, x6 both zero values, leads to the basicsolution WRT present basic vector (x1, x3, x2, x4) which is:

(x1, x2, x3, x4, x5, x6)T = (−12

, 3,−52

, 5, 0, 0)T

Example 2: Here also we use the pivot element choice strategy

164 Ch. 1. Linear eqs.

“partial pivot” as in Example 1 above and the same notation. “IE” indicatesan inconsistent equation obtained in the method. The original tableau isthe one at the top.

BV x1 x2 x3 x4 x5 RHS Eq. no.0 −2 6 −8 8 6 1

−1 0 4 −7 7 16 21 −2 2 −1 1 −8 RI to R1 30 −2 6 −8 8 8 4

PC↑1 −2 2 −1 1 −8 PR 3−1 0 4 −7 7 16 2

0 −2 6 −8 8 6 10 −2 6 −8 8 8 4

PC↑

BV x1 x2 x3 x4 x5 RHS Eq. no.x1 1 −2 2 −1 1 −8 3

0 −2 6 −8 8 8 PR 20 −2 6 −8 8 6 10 −2 6 −8 8 8 4

PC↑x1 1 −2 2 −1 1 −8 3x2 0 −2 6 −8 8 8 2

0 0 0 0 0 −2 IE 10 0 0 0 0 0 RC 4

Since the third row in the final tableau is the inconsistent equation “0= −2” we conclude that the original system is inconsistent, and has nosolution.

Exercises

1.20.1 Solve the numerical exercises in Section 1.16 by the G eliminationmethod.

1.21 Comparing GJ, G Methods 165

1.21 Comparison of the GJ and the G

Elimination Methods

A system of linear equations or the detached coefficient tableau for it is saidto be a:

sparse system if a large proportion of the coefficients in it arezero

dense system otherwise.

For solving a dense system of m equations in n variables, if all thetableaus remain dense:

GJ method needs at most 3m2(n+1) arithmetic operations

G method needs at most 3m(m−1)(n+1)2 + m(m−1)

2 arithmeticoperations.

The G method cuts down the work needed by about half. Hence inpractice, the G elimination method is the method of choice for solving linearequations.

However, for students going on to study linear programming, learningthe details of the GJ method is important, as the original version of thesimplex method for solving linear programs is based on the GJ eliminationmethod and uses GJ pivot steps. But modern computer implementations ofthe simplex method are mostly based on highly sophisticated variants of Gpivot steps for numerical stability.

1.22 Infeasibility Analysis

Consider a mathematical model which is a system of m linear equations inn variables, formulated for a real world application. In such a model, thecoefficient vectors of variables usually come from things like properties ofmaterials which are combined, etc., which are very hard to change. TheRHS constants vector usually comes from requirements that are to be met,or targets to be achieved, etc., which are easier to modify if a need arises.

Suppose it turns out that the model is inconsistent, i.e., it has no solu-tion. Mathematically there is nothing more that can be done on the currentmodel. But the real world problem does not go away; it has to be solved

166 Ch. 1. Linear eqs.

somehow. In this situation, we have to investigate what practically feasiblechanges can be made in the model to modify it into a consistent or feasiblesystem. The study of such changes is known as infeasibility analysis.

Since it is very hard to change the coefficient vectors of the variables,changes in them are rarely considered in applications. In most cases it isthe RHS constants which are changed, and this is what we consider in thissection. We will discuss what changes in the RHS constants vector willmodify an inconsistent system into a consistent one.

Inconsistency of the original system is established when the GJ method isapplied on it and it produces an inconsistent equation of the form “0 = α” forsome α �= 0 and reaches Step 2.2 in the method for the first time (a similarthing occurs if the G method is used instead of the GJ method). In theversions of the GJ (or the G) methods discussed earlier, when an inconsistentequation shows up for the first time, we terminated those methods once andfor all with the conclusion that the original system has no solution.

In order to carry out infeasibility analysis, whenever an inconsistentequation shows up in the current tableau, we leave that inconsistent rowin the tableau, but continue applying the method to perform pivot steps inthe remaining rows. Subsequently, if other instances of inconsistency areencountered , do the same; i.e., leave those inconsistent rows in the tableau,but continue applying the method until pivot steps are carried out in all therows in which they can be.

In the end we have the final tableau, and the remaining original tableauwith the redundant constraints discovered during the method deleted. Sincethe original system is inconsistent, in the final tableau there are one or moreinconsistent equations of the form “0 = α” for some α �= 0. Suppose theseare in rows r1, . . . , rt in the final tableau. Let the updated RHS constantsin these rows in the final tableau be: br1 , . . . , brt , each of which is nonzero.

One possible way to modify the original RHS constants vector to makethe model consistent is: for each k = 1 to t, replace the RHS constant brk

in the rkth row of the remaining original tableau by brk− brk

. The effectof this change is to convert the updated RHS constants in all these rowsin the final tableau to 0, i.e., the changes made convert all the inconsistentequations in the final tableau into redundant constraints.

To get a solution of the modified system, you can now strike off theserows r1, . . . , rt from the final tableau and the remaining original tableau asredundant constraints. With the remaining final tableau, go to the final stepin the method to find a solution of the modified system.

1.22 Infeasibility Analysis 167

Example : Consider the following system of 7 equations in 7 vari-ables in detached coefficient form in the first tableau below. We apply theGJ method to solve it. PR (pivot row), PC (pivot column), RC (redundantconstraint to be eliminated), BV (basic variable selected in this row) havethe same meanings as before, and in each tableau, the pivot element for thepivot step carried out in that tableau is boxed. “IE” indicates an inconsis-tent equation obtained in the method. We number all the rows (equations)serially from top to bottom, and record these numbers in a rightmost col-umn, to make it easy to keep track of redundant constraints eliminated.

BV x1 x2 x3 x4 x5 x6 x7 RHS Eq. no.1 0 1 −1 1 1 0 −7 PR 10 −1 2 1 −1 0 1 8 21 −2 5 1 −1 1 0 9 31 1 0 2 1 0 0 10 43 0 5 5 1 1 2 35 50 0 1 3 1 0 0 15 63 0 7 11 3 1 2 55 7

PC↑x1 1 0 1 −1 1 1 0 −7 1

0 −1 2 1 −1 0 1 8 PR 20 −2 4 2 −2 0 0 16 30 1 −1 3 0 −1 0 17 40 0 2 8 −2 −2 2 56 50 0 1 3 1 0 0 15 60 0 4 14 0 −2 2 76 7

PC↑x1 1 0 1 −1 1 1 0 −7 1x2 0 1 −2 −1 1 0 −1 −8 2

0 0 0 0 0 0 0 0 RC 30 0 1 4 −1 −1 1 25 PR 40 0 2 8 −2 −2 2 56 50 0 1 3 1 0 0 15 60 0 4 14 0 −2 2 76 7

PC↑

168 Ch. 1. Linear eqs.

BV x1 x2 x3 x4 x5 x6 x7 RHS Eq. no.x1 1 0 0 −5 2 2 −1 −32 1x2 0 1 0 7 −1 −2 1 42 2x3 0 0 1 4 −1 −1 1 25 4

0 0 0 0 0 0 0 6 IE 50 0 0 −1 2 1 −1 −10 PR 60 0 0 −2 4 2 −2 −24 7

PC↑Final tableau

BV x1 x2 x3 x4 x5 x6 x7 RHS Eq. no.x1 1 0 0 0 −8 −3 4 18 1x2 0 1 0 0 13 5 −6 −38 2x3 0 0 1 0 7 3 −3 −15 4

0 0 0 0 0 0 0 6 IE 50 0 0 1 −2 −1 1 10 60 0 0 0 0 0 0 −4 IE 7

The original third constraint has been deleted as a redundant constraint.The remaining original tableau is:

x1 x2 x3 x4 x5 x6 x7 RHS Modified Eq. no.RHS

1 0 1 −1 1 1 0 −7 −7 10 −1 2 1 −1 0 1 8 8 21 1 0 2 1 0 0 10 10 43 0 5 5 1 1 2 35 29 50 0 1 3 1 0 0 15 15 63 0 7 11 3 1 2 55 59 7

In the final tableau, the 4th and the 6th rows represent inconsistentequations with updated RHS constants of 6 and −4 respectively. To makethe system feasible we need to subtract 6, and −4 from the correspondingentries in the remaining original tableau, leading to the “modified RHSconstants vector” given on the right in the remaining original tableau.

The effect of this change is to modify the 4th and the 6th RHS constantsin the final tableau to zero. This makes the corresponding constraints re-dundant, which we delete. So, the final tableau for the modified problemis:

1.22 Infeasibility Analysis 169

BV x1 x2 x3 x4 x5 x6 x7 RHSFinal tableau for modified problem

x1 1 0 0 0 −8 −3 4 18x2 0 1 0 0 13 5 −6 −38x3 0 0 1 0 7 3 −3 −15x4 0 0 0 1 −2 −1 1 10

from which we find that the general solution of the modified system is:

x1

x2

x3

x4

x5

x6

x7

=

18 + 8x5 + 3x6 − 4x7

−38 − 13x5 − 5x6 + 6x7

−15 − 7x5 − 3x6 + 3x7

10 + 2x5 + x6 − x7

x5

x6

x7

where x5, x6, x7 are arbitrary real valued parameters.

Discussion on Modifying the RHS Vector to Con-vert an Inconsistent System Into a Feasible System

In the example of an inconsistent system given above, we modified thevalues of two RHS constants to make it consistent. Also we decreased thevalue of one RHS constant from 35 to 29, and increased the value of anotherfrom 55 to 59 for a total change of

|35 − 29| + |55 − 59| = 10 units

Usually, practitioners look for a modification in the RHS constants vectorb = (bi) of an inconsistent system to b′ = (b′i) to make it feasible, with

the smallest number of changes, i.e., one which minimizes thenumber of i for which bi �= b′i, and

the smallest total change, which is∑

i |bi − b′i|.

The number of changes made in the above process is equal to the numberof inconsistent equations “0 = α” for some α �= 0 encountered in the method.Surprisingly, the total number of inconsistent equations in the final tableaudepends on the order in which rows of the tableau are chosen as pivot rows

170 Ch. 1. Linear eqs.

in the method. As an illustration of this, we consider the following examplefrom Murty, Kabadi, Chandrasekaran [2001] (the equations are numberedfor convenience in tracking them).

Eq. no.1 x1 = 12 x2 = 13 x3 = 14 x1 + x2 + x3 = 25 x1 + x2 + x3 = 36 x1 + x2 + x3 = 3

If we apply the GJ method on this system using the equations 1 to 6 inthat order as pivot rows, we get the following final tableau with only oneinconsistent equation:

Final tableau when eqs. usedas PR in order 1 to 6

x1 x2 x3 RHS Eq. no.1 0 0 1 10 1 0 1 20 0 1 1 30 0 0 −1 40 0 0 0 50 0 0 0 6

On the other hand, if we apply the GJ method on this system usingequations as pivot rows in the order 4, 1, 2, 3, 5, 6, we get the followingfinal tableau with three inconsistent equations:

Final tableau when rows 4,1,2,3,5,6 are PRs in orderx1 x2 x3 RHS Eq. no.1 0 0 1 40 1 0 1 10 0 1 0 20 0 0 1 30 0 0 1 50 0 0 1 6

1.23 Homo geneous Systems 171

Given an inconsistent system of equations, to find the order in which rowsin the original tableau have to be choosen as pivot rows to get the smallestnumber of inconsistent equations in the final tableau is a hard combinatorialoptimization problem which belongs to the realm of integer programming,and is outside the scope of this book.

Also, given an inconsistent system of linear equations with the originalRHS constants vector b = (bi), finding the modification of b to b′ = (b′i)that will make the system consistent while minimizing the sum of absolutechanges

∑i |bi − b′i|, can be posed as a linear programming problem and

solved very efficiently using linear programming algorithms. This, again, isoutside the scope of our book.

Exercises

1.22.1 Consider the following system of 6 equations in 7 variables beingsolved by the GJ method. We want to carry out infeasibility analysis onthis system, if it is inconsistent.

Find the orders in which the rows of the tableau have to be choosen aspivot rows, in order to (i) minimize, (ii) maxiimize, the number ofinconsistent equations encountered during the method.

x1 + x2 + x3 + x4 + x5 + x6 + x7 = 3x1 + x2 + x3 + x4 + x5 + x6 + x7 = 3x1 + x2 + x3 + x4 + x5 + x6 + x7 = 3x1 + x2 + x3 + x4 + x5 + x6 + x7 = 2x1 + x2 + x3 + x4 + x5 + x6 + x7 = 3x1 + x2 + x3 + x4 + x5 + x6 + x7 = 3

1.22.2 Do infeasibility analysis for systems of linear equations given inExercise 1.16.4. For each of those systems find the modified RHS constantsvector that would make the system feasible as determined by the methoddiscussed in this section, and a solution of the modified system.

1.23 Homogeneous Systems of Linear Equa-

tions

The general system of linear equations in detached coefficient form is;

172 Ch. 1. Linear eqs.

x1 . . . xj . . . xn RHS constantsa11 . . . a1j . . . a1n b1...

......

...ai1 . . . aij . . . ain bi...

......

...am1 . . . amj . . . amn bm

It is said to be a homogeneous system if the RHS constants bi = 0for all i = 1 to m; nonhomogeneous system if at least one bi �= 0.

For the homogeneous system, x = 0 is always a solution. The interestingquestion that one needs to find about a homogeneous system is whetherx = 0 is its unique solution, or whether it has a nonzero solution (i.e., asolution x �= 0).

Algorithm: How to Check if a Homogeneous System Has aNonzero Solution

BEGIN

Apply the GJ method or the G elimination method on the system. Sincebi = 0 for all i in the original tableau, in all the tableaus obtained underthese methods, all the RHS constants will always be zero. So inconsistentrows will not appear.

At the end, if all the variables have been selected as basic variables (i.e.,there are no nonbasic variables), then x = 0 is the only solution for thehomogeneous system; i.e., it has no nonzero solution.

On the other hand, if there is at least one nonbasic variable at the end,the original homogeneous system has nonzero solutions. A nonzero solutionfor the system can be obtained by giving one or more nonbasic variablesspecific nonzero values, and then finding the corresponding values for thebasic variables from the final tableau. To be specific, suppose the finaltableau has p rows with the basic variables xi1, . . . , xip in that order, and letthe nonbasic variables be xj1, . . . , xjn−p . Then after rearranging the variablesin the tableau so that the basic variables appear first in their proper order,the final tableau has the following form:——

1.23 Homo geneous Systems 173

When nonbasic variables existFinal tableau after rearrangement of variables

Basic xi1 xi2 . . . xip xj 1 . . . xj n−p

Vars.xi1 1 0 . . . 0 a1,j1 . . . a1,jn−p 0xi2 0 1 . . . 0 a2,j1 . . . a2,jn−p 0...

......

......

......

xip 0 0 . . . 1 ap,j1 . . . ap,jn−p 0

This is the final tableau if the GJ method is used. If the G method isused, the portion of the tableau corresponding to basic variables will be anupper triangular tableau.

There are n− p nonbasic variables here. By giving exactly one nonbasicvariable the value 1, and making all other nonbasic variables zero, and com-puting the corresponding values of the basic variables by back substitutionif the G method is used, or reading them directly from the final tableau ifthe GJ method is used, we get n−p distinct nonzero solutions to the originalhomogeneous system. These solutions are of the following form:

xi1

xi2...

xip

xj1

xj2...

xjn−p

=

−a1,j1

−a2,j1...

−ap,j1

10...0

,

−a1,j2

−a2,j2...

−ap,j0

01...0

, . . . ,

−a1,jn−p

−a2,jn−p

...−ap,jn−p

00...1

,

In these solutions of the system, the variables may not be in the usualorder of x1 to xn, but the variables can be reordered if desired. Everysolution of the original homogeneous system is a linear combination of these(n − p) solutions and the dimension of the set of all solutions is (n − p).

END

Example 1: We solve the following homogeneous system using theGJ pivotal method. The original detached coefficient tableau is the oneat the top. We number the equations serially and record this number in

174 Ch. 1. Linear eqs.

the rightmost column. PR (pivot row), PC (pivot column), RC (redundantconstraint to be eliminated), BV (basic variable selected in this row) havethe same meaning as before, and the pivot elements are always boxed.

BV x1 x2 x3 x4 RHS Eq. no.1 2 0 1 0 PR 1−1 −3 1 0 0 2

0 −1 1 1 0 30 −2 3 1 0 40 0 1 1 0 5

PC↑x1 1 2 0 1 0 1

0 −1 1 1 0 PR 20 −1 1 1 0 30 −2 3 1 0 40 0 1 1 0 5

PC↑BV x1 x2 x3 x4 RHS Eq. no.x1 1 0 2 3 0 1x2 0 1 −1 −1 0 2

0 0 0 0 0 RC 30 0 1 −1 0 PR 40 0 1 1 0 5

PC↑x1 1 0 0 5 0 1x2 0 1 0 −2 0 2x3 0 0 1 −1 0 4

0 0 0 2 0 PR 5PC↑

Final Tableaux1 1 0 0 0 0 1x2 0 1 0 0 0 2x3 0 0 1 0 0 4x4 0 0 0 1 0 5

Since all the variables are basic variables (i.e., there are no nonbasicvariables at the end), this homogeneous system has x = 0 as the uniquesolution (i.e., it has no nonzero solution).

1.23 Homo geneous Systems 175

Example 2: We will use the G elimination method to solve thehomogeneous system for which the detached coefficient tableau is the oneat the top. The notation is the same as in Example 1 above.

BV x1 x2 x3 x4 x5 RHS Eq. no.1 0 0 1 1 0 PR 10 1 0 1 0 0 22 1 0 3 2 0 31 1 1 1 0 0 4

PC↑x1 1 0 0 1 1 0 1

0 1 0 1 0 0 PR 20 1 0 1 0 0 30 1 1 0 −1 0 4

PC↑BV x1 x2 x3 x4 x5 RHS Eq. no.x1 1 0 0 1 1 0 1x4 0 1 0 1 0 0 2

0 0 0 0 0 0 RC 30 1 1 0 −1 0 PR 4

PC↑Final Tableau

x1 1 0 0 1 1 0 1x4 0 1 0 1 0 0 2x2 0 1 1 0 −1 0 4

Since there are two nonbasic variables x3, x5 in the final tableau, thishomogeneous system has nonzero solutions. Rearranging the columns of thebasic variables in their proper order, and transferring the columns of thenonbasic variables to the right, the final tableau becomes:

BV x1 x4 x2 RHSx1 1 1 0 −x5

x4 0 1 1 0x2 0 0 1 −x3 + x5

First fixing the nonbasic variables at values x3 = 1, x5 = 0, and thensolving the above upper triangular system by back substitution for the values

176 Ch. 1. Linear eqs.

of the basic variables, we get the first solution given below. Then we fix thenonbasic variables at values x3 = 0, x5 = 1, and get the second solutiongiven below. In writing these solutions, we rearranged the variables in theirnatural order x1 to x5.

x1

x2

x3

x4

x5

=

−1−1

110

,

010

−11

The general solution of the system is a linear combination of these twosolutions. Taking the coefficients of this linear combination to be x3, x5

respectively, the general solution is:

x1

x2

x3

x4

x5

= x3

−1−1

110

+ x5

010

−11

where x3, x5 are arbitrary real parameters representing the values of thenonbasic variables x3, x5 in the solution.

Exercises

1.23.1 In each of the following homogeneous system of linear equations,check whether it has a nonzero solution. If it does, find a nonzero solution,and an expression for the general nonzero solution of the system. Also,determine the dimension of the solution set in each case. ( (i)-(iii) have nononzero solution, the others do).

(i) x1 x2 x3 x4

1 −1 2 1 00 1 1 2 0

−1 0 −3 0 01 0 1 1 00 −1 3 5 0

1.24 Linear Eqs. & Inequalities 177

(ii) x1 x2 x3 x4 x5

1 2 0 −3 −1 0−1 0 −1 2 0 0

0 −3 2 −2 1 0−1 0 0 −1 2 0

0 1 −1 2 1 0−1 0 0 −2 3 0

(iii) x1 x2 x3

1 2 2 02 1 5 02 2 1 0

(iv) x1 x2 x3 x4 x5 x6

1 1 1 0 0 0 00 0 0 1 1 1 01 0 0 1 0 0 00 1 0 0 1 0 00 0 1 0 0 1 0

(v) x1 x2 x3

1 2 2 02 1 5 07 8 16 0

(vi) x1 x2 x3 x4

−1 2 −2 3 02 −5 3 2 0

−1 3 0 −2 0−4 10 −5 −1 0−4 11 −4 −9 0

1.24 Solving Systems of Linear Equations

And/Or Inequalities

Linear algebra techniques, the focus of this book, can directly handle onlysystems of linear equations. When there are also linear inequalities in the

178 Ch. 1. Linear eqs.

system, linear algebra techniques are not able to handle that system directly.To handle systems involving linear inequalities, we need linear programmingtechniques, which are outside the scope of this book.

1.25 Can We Obtain a Nonnegative So-

lution to a System of Linear Equa-

tions by the GJ or the G Methods

Directly?

Nonnegativity restrictions on the variables are actually inequality constraints.If a system of linear equations has a unique solution which happens to be

nonnegative, the GJ or the G methods will of course find it. However, whena system of linear equations has multiple solutions, some of them may satisfythe nonnegativity constraints, others may not. In this case we cannot guar-antee that the GJ or the G methods will find a nonnegative solution to thesystem of equations even if such a solution exists; or determine conclusivelywhether a nonnegative solution exists for the system.

As mentioned earlier, the problem of finding a nonnegative solution toa system of linear equations can be posed as a linear programming problemand solved using linear programming techniques. However, linear program-ming is outside the scope of this book.

1.26 Additional Exercises

1. Toy store problem: A toystore chain has several stores in the midwest.For the coming X-mas season, they need to put orders with their overseassuppliers before the end of May for delivery in time for the X-mas salesperiod.

Since unsold toys at the end of the X-mas season do not contribute muchto the profit of the company, they base their order quantities quite close tothe expected sales volume. From experience over the years they observedthat the X-mas sales volume has a high positive correlation with the DJA= Dow Jones average (a measure of the economic status of the region priorto the sales period), and a high negative correlation with the % unemploy-ment rate in the region. Following table gives data on the DJA during the

1.26 Exercises 179

months of Feb, Mar, Apr (these are independent variables x1, x2, x3), the %unemployment in the region during this period (independent variable x4),and the toy sales volume in the region in $ million during the X-mas salesseason (dependent variable y) between 1990-2001.

From above discussion it is reasonable to assume that the expected valueof y can be approximated by a function a0+a1x1+a2x2+a3x3+a4x4, wherethe parameters a1, a2, a3 are expected to be positive, and a4 is expected tobe negative. But ignore these sign restrictions on the parameters, and derivethe normal equations for finding the best values for them that give the closestfit, by the least squares method.

Year x1 x2 x3 x4 y

2001 10690 10185 10306 4.3 592000 10533 10525 10798 4.0 601999 9356 9550 10307 4.3 541998 8307 8664 8940 4.6 471997 6828 6727 6800 5.2 361996 5435 5527 5579 5.5 281995 3927 4080 4239 5.5 201994 3898 3723 3634 6.5 171993 3344 3405 3434 7.1 141992 3247 3253 3294 7.4 131991 2798 2903 2895 6.6 111990 2607 2665 2673 5.2 10

2. A brother B and his sister S stated the following facts about theirfamily. S has the same number of brothers and sisters. B has no brothers,but has as many sisters as the sum of the number of brothers and sistersof S. Formulate the problem of determining how many sons and daughterstheir parents have using a system of linear equations.

3. The principle behind drain cleaners to unclog a sink is: they containsodium hydroxide (NaOH) and aluminium (Al) which when mixed withwater (H2O) react to produce NaAl(OH)4 and hydrogen gas that bubblesinside the plumbing to unclog the drain. So, NaOH, Al, H2O react toproduce NaAl(OH)4 and H2. Balance this reaction.

4. A test for arsenic (As) poisoning is based on combining hydrochloric

180 Ch. 1. Linear eqs.

acid (HCl), zinc (Zn), with a sample hair of the victim (which is likely tocontain H3AsO4 if the victim suffers from arsenic poisoning). Zn, H3AsO4,and HCl combine to produce ZnCl2, H2O, and AsH3 which is a gas calledarsene gas whose presence can be easily detected. Balance this reaction.

5. A trucking company uses three types of trucks of different capacities.The company plans to use these trucks to haul various numbers of threedifferent machines (machines A, B, C) from the shipyard where they are tobe picked up, to the site of a new plant being set up.

Truck type 1 can haul one machine A, one machine B, and two copiesof machine C in a full load.

Truck type 2 can haul one each of machines A, B, C in a full load.Truck type 3 can haul one of each of machines A, C, and two copies of

machine B in a full load.12 copies of machine A, 17 copies of machine B, and 15 copies of machine

C have to be picked up. Assuming that each truck sent can be fully loadedas mentioned above, find how many trucks of each type should be sent topick up all the machines. Formulate the problem and obtain a solution.

6. There are three fretilizer types available (Fer 1, 2, 3) in bags. eachbag contains the following amounts of the principal fertilizer components N,P, K.

Fertilizer type Lbs. of following per bagN P K

Fer 1 1 3 2Fer 2 3 3 1Fer 3 2 5 5

A farmer wants to apply exactly 25 lbs of N, 42 lbs of P, and 28 lbs of Kon his field. Construct a model to determine how many bags of Fer 1, 2, 3each to apply on the field to get exactly these amounts of N, P, K; and findthe solution of that model.

7. (From C. Rorres and H. Anton, Applications of Linear Algebra, Wiley,1984) Hooke’s law of physics states that the length y of a uniform springis a linear function of the force applied x; i.e., y can be approximated bya + bx, where the coefficient b is called the spring constant. A particular

1.26 Exercises 181

unstretched spring has length 6.1 (i.e., when x = 0, y = 6.1). Differentforces were applied to this spring and it resulted in the following data.

x = force applied y = spring lengthin lbs. under force

0 6.12 7.74 8.66 10.4

Derive the normal equations to find the best values for the parametersa, b that give the best fit for Hooke’s law for this data using the method ofleast squares.

8. Find the value of b3 for which the following system has at least onesolution. Also, show that if the system has one solution, then it has aninfinite number of solutions.

x1 + x2 = 22x2 + x3 = 3

x1 − x2 + (b3 − 6)x3 = −1x1 + 3x2 + x3 = b3

9. Find the values of the parameter a for which the following systemhas no solutions, unique solution, infinite number of solutions.

x1 + x2 = 6x1 + x3 = 7

ax1 + x2 + x3 = 13

10. Develop a system of linear equations for which the general solutionis x = (x1, x2, x3, x4)T = (−17+ 8α− 7β, 28− 13α+ 19β, α, β)T , where α, βare arbitrary real valued parameters.

11. Consider the following system of linear equations. Find the condi-tions that the RHS constants have to satisfy for the system to have at least

182 Ch. 1. Linear eqs.

one solution. Also, show that if the system has a solution, then it has aninfinite number of solutions.

x1 + x2 = b1

x2 + x3 = b2

x1 + 2x2 + x3 = b3

x3 + x4 = b4

x1 + 2x2 + 2x3 + x4 = b5

12. In the following homogeneous system α, β are real valued parame-ters. For which values of these parameters does the system have a nonzerosolution? Explain. When those conditions are satisfied, find an expressionfor the general solution of the system.

x1 + x2 + x3 = 0−x1 + 2x2 − 2x3 = 0

αx2 + βx3 = 0

13. What conditions do the parameters a, b, c, d in the following sys-tem have to satisfy for this system to have at least one solution? If thoseconditions are satisfied, find an expression for the general solution of thesystem.

x1 − x2 + x3 − x4 = a

−x1 + 2x2 + 2x3 + 2x4 = b

x2 + 3x3 + x4 = c

x1 + 4x3 = d

14. Find conditions that α has to satisfy so that the following system hasat least one nonzero solution. When that condition holds, find the generalsolution of the system.

x1 − x2 + 2x3 = 02x1 − 3x2 + x3 = 0

αx2 − 3x3 = 0

1.26 Exercises 183

1.27 References

[1.1] K. G. Murty, S. N. Kabadi, and R. Chandrasekaran, “InfeasibilityAnalysis for Linear Systems, A Survey”, Arabian Journal of Science andEngineering, 25, 1C(June 2000)3-18.

Index

For each index entry we providethe section number where it is de-fined or discussed first.

Additivity assumption 1.7Affine function 1.7Affine hull 1.5Affine span 1.5Algorithm 1.1Al-Khowarizmi 1.1, 1.2Alternate system 1.17Archimedis’ cattle problem 1.2Associative law of addition 1.5

Back substituition metho d 1.18Balancing chemical reactions 1.2Basis, basis matrix 1.16Basic solution 1.8Basic variables 1.8, 1.14Basic vector 1.8, 1.16Bilinear function 1.7

C a nonical t ableau 1.16Central difference approximation 1.3Certificate of infeasibility 1.13Certificate of redundancy 1.13Coefficients 1.2Column vector 1.2Combinatorial optimization 1.2Complete (full) pivoting 1.16

Computational complexity 1.16Constraint function 1.7Coordinate subspaces 1.5Curve fitting problem 1.2

Dant zig 1.2, 1.16Data structures 1.16Decision variables 1.2Dense system 1.21Dependent variables 1.8Detached coefficient tableau 1.4Dimension of solution set 1.16, 1.8Discretization 1.3Dot product 1.5

Echelon form 1.20Edges 1.1Elementary row operations 1.11Elimination approach 1.14Entering variable 1.16Equivalent systems 1.8, 1.13Eratosthenes sieve 1.1Euler circuit 1.1Euler’s algorithm 1.1Evidence of infeasibility 1.13Evidence of redundancy 1.13

Fa rkas’ lemma 1.2Feasible solution 1.2Free variables 1.8

184

Index f or Ch. 1 185

G(aussian) elimination metho d1.20

G(aussian) pivot step 1.19G J pivot method 1.16G J pivot step 1.15General solution (paramertric form)

1.8Grid points 1.3

Homogeneous syst e ms 1.23

I nconsist e nt equat i on 1.10Inconsistent system 1.8Independent variables 1.8Infeasible 1.2Infeasible system 1.8Infeasibility analysis 1.22Inner product 1.5Integer programming 1.2ith unit vector 1.5

j t h comp one nt ( co ordina t e ) 1.2

Linear algebra 1.1, 1.2Least squares estimator 1.2Least squares fit 1.2Linear combinations 1.9Linear dependence relation 1.9Linear diophantine equations 1.2Linear equations 1.2Linear function 1.7Linear inequalities 1.2Linearity assumption 1.7Linearly dependent system 1.9Linear hull (span) 1.5Linearly independent system 1.9Linear Programming 1.2Linear regression problem 1.2

Lower triangular system 1.18

Magic squares 1.2Magical card trick 1.2Mathematical modeling 1.2Memory matrix 1.12Method of least squares 1.2Minimal linear dependence 1.9Model function 1.2Modeling 1.2Monkey & bananas problem 1.2Monkey & coconuts problem 1.2

n dimensional vector space 1.2n-tuple (vector) 1.2Network 1.1Nodes 1.1Nonbasic variables 1.8Nonhomogeneous system 1.23Nonlinear equations 1.7Nonlinear function 1.7Nonsingular square system 1.16Nontrivial linear combination 1.9Normal equations 1.2

O ptimal residual 1.2Order 1.16Ordered set 1.2

Pa rameter estimation problem1.2

Partial pivoting 1.16Pivot column 1.15Pivot row 1.15Prime number 1.1Proprtionality assumption 1.7

RHS const ant s 1.2

186 Ch. 1. Linear eqs.

REF 1.20RREF 1.16Rank 1.8Reduced row echelon form 1.16Redundant constraint 1.9Regression coefficients 1.2Regression line 1.2Round-off errors 1.16Row echelon form 1.20Row interchange 1.11, 1.20Row operations 1.11Row reduction algorithm 1.16Row vector 1.2rth basic variable 1.8

Scalar multiplication 1.5Separability assumption 1.7Simultaneous linear equations 1.2Solution 1.2Solution set 1.8Spanning property 1.5Sparse system 1.21Subspace 1.5Subspace property 1.5Sum of squared errors 1.2Supervisor principle for infeasibil-

ity 1.13

Theorem of a lternatives 1.17Tower of brahma 1.18Tower of Hanoi 1.18Transpose 1.5Triangular system 1.18Trivial combination 1.92-point boundary value problem 1.3

Unique solution 1.8Unknowns 1.2

Upper triangular system 1.18

Va riables 1.2Vector 1.2

0 = 0 equation 1.90 = 1 equation 1.10, 1.13Zero vector 1.5

Index for Ch. 1 187

List of algorithms

1. Eratosthenes Sieve to find all prime numbers ≤ n

2. Euler’s algorithm to check the existence of an Euler circuit in a net-work

3. Algorithm for solving a curve fitting problem

4. Setting up the memory matrix and using it (to express a row in thecurrent tableau as a linear combination of rows in the original tableau)

5. Elimination approach for solving linear equations

6. GJ Pivotal method

7. G elimination method

8. Algorithm for checking if a homogeneous system has a nonzero solu-tion.

Historical note on:

1. Algorithms

2. Algebra, linear algebra

3. Method of least squares

4. Linear inequalities and linear programming

5. GJ pivotal method.

6. Vectors

Discussion on:

1. RHS constants, coefficients

188 Ch. 1. Linear eqs.

2. Rows correspond to constraints, columns to variables

3. Some features of the GJ method

4. Modifying the RHS constants vector to convert an infeasible systeminto a feasible system

List of Results

1. Result 1.7.1: Dot product of two vectors is a bilinear function of thetwo vectors

2. Result 1.9.1: Every solution satisfies all linear combinations of equa-tions in the system

3. Result 1.9.2: Deletion of a redundant constraint leads to an equivalentsystem

4. Result 1.9.3: Any one constraint in a linear dependence relation canbe considered redundant

5. Result: 1.10.1: Fundamental condition for inconsistency

6. Result 1.11.1: Row operations lead to an equivalent system, andnonzero columns remain nonzero.

Theorem

Theorem of alternatives for systems of linear equations.

Contents

2 Matrices, Matrix Arithmetic, Determinants 1892.1 A Matrix as a Rectangular Array of Numbers, Its Rows

and Columns . . . . . . . . . . . . . . . . . . . . . . . 1892.2 Matrix−Vector Multiplication . . . . . . . . . . . . . . 1932.3 Three Ways of Representing the General System of Lin-

ear Equations Through Matrix notation . . . . . . . . 1972.4 Scalar Multiplication and Addition of Matrices, Trans-

position . . . . . . . . . . . . . . . . . . . . . . . . . . 2012.5 Matrix Multiplication & Its Various Interpretations . . 2042.6 Some Special Matrices . . . . . . . . . . . . . . . . . . 2162.7 Row Operations and Pivot Steps on a Matrix . . . . . 2222.8 Determinants of Square Matrices . . . . . . . . . . . . 2242.9 Additional Exercises . . . . . . . . . . . . . . . . . . . 2482.10 References . . . . . . . . . . . . . . . . . . . . . . . . . 260

i

ii

Chapter 2

Matrices, Matrix Arithmetic,Determinants

This is Chapter 2 of “Sophomore Level Self-Teaching Web-book for Computational and Algorithmic Linear Algebra andn-Dimensional Geometry” by Katta G. Murty.

2.1 A Matrix as a Rectangular Array of

Numbers, Its Rows and Columns

A matrix is a rectangular array of real numbers. If it has m rows andn columns, it is said to be an m × n matrix, or a matrix of order orsize m × n. Here is a 3 × 5 matrix with three rows and five columns.

−1 7 8.1 3 −20 −2 0 5 −71 −1 3 0 −8

The size or order of a matrix is always a pair of positive numbers,the first being the number of rows of the matrix, and the second thenumber of columns.

In mathematical literature, matrices are usually written within brack-ets of (...) or [...] type on the left and right.

189

190 Ch. 2. Matrices

History of the name “matrix”: The word “matrix” is derivedfrom the Latin word for “womb” or “a pregnant animal”. In 1848 J.J. Sylvester introduced this name for a rectangular array of numbers,thinking of the large number of subarrays that each such array cangenerate (or “give birth to”). A few years later A. Cayley introducedmatrix multiplication and the basic theory of matrix algebra quicklyfollowed.

Initially, the concept of a matrix was introduced mainly as a tool forstoring the coefficients of variables in a system of linear equations, inthat case it is called the coefficient matrix or matrix of coefficientsof the system. In 1858 A. Cayley wrote a memoir on the theory oflinear transformations and matrices. With arithmetical operations onmatrices defined, we can think of these operations as constituting analgebra of matrices, and the study of matrix algebra by Cayley andSylvester in the 1850’s attracted a lot of attention that led to importantprogress in matrix and determinant theory. ��

Consider the following system of linear equations.

3x2 − 4x4 + x1 − x3 = −10

2x1 + 5x3 − 2x4 + 6x2 = −5

4x4 − x1 + 8x3 − 3x2 = 7

Since this is a system of 3 equations in 4 variables, its coefficientmatrix will be of order 3 × 4. Each row of the coefficient matrix willcorrespond to an equation in the system, and each of its columns corre-sponds to a variable. So, in order to write the coefficient matrix of thissystem, it is necessary to arrange the variables in a particular order,say as in the column vector

x =

x1

x2

x3

x4

and write each equation with the terms of the variables arranged inthis particular order x1, x2, x3, x4. This leads to

2.1. A Matrix 191

x1 + 3x2 − x3 − 4x4 = −10

2x1 + 6x2 + 5x3 − 2x4 = −5

−x1 − 3x2 + 8x3 + 4x4 = 7

Now the coefficient matrix of the system A = (aij : i = 1, . . . , m;j = 1, . . . , n), where aij is the coefficient of xj in the ith equation, canbe easily written. It is

1 3 −1 −42 6 5 −2

−1 −3 8 4

For i = 1 to 3, row i of this matrix A corresponds to the ithequation in the system, and we denote it by the symbol Ai.. For j= 1 to 4, column j of this matrix A, denoted by the symbol A.j ,corresponds to, or is associated with, the variable xj in the system.

Thus the coefficient matrix of a system of linear equations is exactlythe array of entries in the detached coefficient tableau representationof the system without the RHS constants vector. The rows of the coef-ficient matrix are the rows of the detached coefficient tableau withoutthe RHS constant, its columns are the columns of the detached coeffi-cient tableau.

In general, an m×n matrix D can be denoted specifically as Dm×n

to indicate its order; it will have m row vectors denoted by D1., . . . , Dm.;and n column vectors denoted by D.1, . . . , D.n. The matrix D itself canbe viewed as the array obtained by putting its column vectors one afterthe other horizontally as in

Dm×n = (D.1 D.2 . . .D.n)

or putting its row vectors one below the other vertically as

Dm×n =

D1.

D2....

Dm.

192 Ch. 2. Matrices

A row vector by itself can be treated as a matrix with a single row.Thus the row vector

(3,−4, 7, 8, 9)

can be viewed as a matrix of order 1 × 5.In the same way, a column vector can be viewed as a matrix with

a single column. So, the column vector

−321

can be viewed as a matrix of order 3 × 1.The entry in a matrix D in its ith row and jth column is called its

(i, j)th entry, and denoted by a symbol like dij. Then the matrix Ditself can be denoted by the symbol (dij). So, the (2, 3)th entry of thecoefficient matrix A given above, a23 = 5.

Let b denote the column vector of RHS constants in a system oflinear equations in which the coefficient matrix is A of order m × n.Then the matrix A obtained by including b as another column at theright of A, written usually in a partitioned form as in

A = (A|b)is known as the augmented matrix of the system of equations.

A partitioned matrix is a matrix in which the columns and/orthe rows are partitioned into various subsets. The augmented matrixA is written as a partitioned matrix with the columns associated withthe variables in one subset of columns, and the RHS constants columnvector by itself in another subset of columns.

As an example, consider the following system of 3 equations in 4variables.

3x2 − 4x4 − x3 = −10

2x1 − 2x4 + 6x2 = −5

4x4 − x1 + 8x3 = 7

2.2 Matrix − Vector products 193

Arranging the variables in the order x1, x2, x3, x4, the system is

3x2 − x3 − 4x4 = −10

2x1 + 6x2 − 2x4 = −5

−x1 + 8x3 + 4x4 = 7

So, for this system, the coefficient matrix A, and the augmentedmatrix are

A =

0 3 −1 −42 6 0 −2

−1 0 8 4

,A =

0 3 −1 −4 −102 6 0 −2 −5

−1 0 8 4 7

2.2 Matrix−Vector Multiplication

The concept of the product of a matrix times a column vector in thisspecific order, is formulated so that we can write the system of equa-tions with coefficient matrix A, column vector of variables x, and RHSconstants vector b, in matrix notation as Ax = b. We define this con-cept next.

Definitions: Matrix−vector products: Consider a matrix D oforder m × n, and a vector p. The product Dp, written in this specificorder, is only defined if p is a column vector in Rn (i.e., if the numberof column vectors of D is equal to the number of rows in the columnvector p), i.e., if p = (p1, . . . , pn)T . Then the product Dp is itself acolumn vector given by

Dp =

D1.p...

Dm.p

=n∑

j=1

pjD.j

Also, if q is a vector, the product qD, written in this specific order,is only defined if q is a row vector in Rm (i.e., if the number of columns

194 Ch. 2 . Matrices

in q is equal to the number of rows in D), i.e., if q = (q1, . . . , qm)T , andthe product qD is itself a row vector given by

qD = (qD. 1, . . . , qD.n)

=m∑

i=1

qiDi.

We have given above two ways of defining each matrix-vector prod-uct; both ways yield the same result for the product in each case.

In this chapter we use superscripts to number various vectors in aset of vectors (for example, q

1, q2, q3 may denote three different vectorsin R

n). We always use subscripts to indicte various entries in a vector,for example for a row vector p ∈ R

3, the entries in it will be denotedby p1, p2, p3, so, p = (p1, p2, p3).

Example 1: Let

q1 = (1, 2, 1); q2 = (1, 2); q3 = (2, 2, 2, 1)

A =

1 3 −1 −1 −42 6 5 5 −2

−1 −3 8 8 4

, p3 =

(12

), p4 =

011111

p1 =

101

−12

, p2 =

−12101

The products Ap3, Ap4, Aq1, Aq2, Aq3, p1A, p2A, p3A, p4A, q2A, q3Aare not defined. The reader is encouraged to explain the reasons foreach. Here are the row vectors of matrix A.

2.2 Matrix − Vector products 195

A1. = (1, 3,−1,−1,−4)

A2. = (2, 6, 5, 5,−2)

A3. = (−1,−3, 8, 8, 4)

The column vectors of matrix A are:

A.1 =

12

−1

, A.2 =

36

−3

, A.3 =

−158

A.4 =

−158

, A.5 =

−4−2

4

We have

Ap1 =

A1.p1

A2.p1

A3.p1

=

−7−2

7

.

We also have

Ap1 = 1A.1 + 0A.2 + 1A.3 − 1A.4 + 2A.5 =

12

−1

+

−158

−

−158

+ 2

−4−2

4

=

−7−2

7

giving the same result. In the same way verify that both definitionsgive

Ap2 =

0137

Similarly

q1A = (q1A.1, q1A.2, q

1A.3, q1A.4, q

1A.5) = (4, 12, 17, 17,−4)

= 1A1. + 2A2. + A3..

196 Ch. 2. Matrices

Some Properties of Matrix-Vector Products

Let A be an m × n matrix, u1, . . . , ur be all column vectors in Rn,and α1, . . . , αr real numbers. Then

A(u1 + . . . + ur) =r∑

t=1

Aut

A(α1u1 + . . . + αru

r) =r∑

t=1

αtAut

We will provide a proof of the second relation above. (The firstrelation is a special case of the second obtained by taking αt = 1 for allt.) Let u denote a general column vector in Rn. From the definition

Au =

A1.u...

Am.u

From Result 1.7.1 of Section 1.7, Ai.u is a linear function of u, andhence from the definition of linear functions (see Section 1.7)

Ai.(α1u1 + . . . + αru

r) =r∑

t=1

αtAi.ut

Using this in the above equation

Au =

∑rt=1 αtA1.u

t

...∑rt=1 αtAm.u

t

=

r∑t=1

αt

A1.ut

...Am.u

t

=

r∑t=1

αtAut

proving the second relation above.Also, if v1, . . . , vr are all row vectors in Rm, then we get the following

results from similar arguments

(v1 + . . . + vr)A =r∑

t=1

vtA

(α1v1 + . . . + αrv

r)A =r∑

t=1

αtvtA

2 . 3 R epresent ing G enera l System 197

Exercises

2.2.1: Let

A =

(1 0 −1

−2 1 2

), B =

1 1 2−1 0 3−2 2 1

, C =

−2 11 −2

−2 −3

.

p1 =

( −23

), p2 =

032

, p3 =

117−8

9

.

q1 = (−3, 2), q2 = (−2,−1, 2), q3 = (8,−7, 9, 15).

Mention which of Ax, Ax, Cx, yA, yB, yC are defined for x, y ∈{p1, p2, p3, q1, q2, q3} and which are not. Compute each one that isdefined.

2.3 Three Ways of Representing the Gen-

eral System of Linear Equations Through

Matrix notation

Consider the general system of m equations in n variables in detachedcoefficient tableau form.

x1 . . . xj . . . xn RHS constantsa11 . . . a1j . . . a1n b1...

......

...ai1 . . . aij . . . ain bi...

......

...am1 . . . amj . . . amn bm

Let

198 Ch. 2 . Matrices

A = (aij) the m × n coefficient matrix

b = (b1, . . . , bm)T the column vector of RHS constants

x = (x1, . . . , xn)T the column vector of variables

in this system.

Representing the System as a Matrix Equation

Clearly, the above system of equations can be represented by thematrix equation

Ax = b

The number of rows of the coefficient matrix A is the number ofequations in the system, and the number of columns in A is the numberof variables in the system.

From now on, we will use this notation to represent the generalsystem of linear equations whenever it is convenient for us to do so.

Example 2: Consider the following system in detached coeffi-cient form.

x1 x2 x3

1 −2 −1 −5−3 1 2 −7

Denoting the coefficient matrix, RHS constants vector, column vec-tor of variables, respectively by A, b, x, we have

A =

(1 −2 −1

−3 1 2

), b =

( −5−7

), x =

x1

x2

x3

and we verify that this system of equations is: Ax = b in matrix nota-tion.

2.3 Representing General System 199

Representing the System by Equations

The ith row of the coefficient matrix A is Ai. = (ai1, . . . , ain). Hencethe equation by equation representation of the system is

Ai.x = bi i = 1 to m

Representing the System as a Vector Equation

Another way of representing the system of equations Ax = b as avector equation involving the column vectors of the coefficient matrixA and the RHS constants vector b is

x1A.1 + x2A.2 + . . . + xnA.n = b

i.e., the values of the variables in a solution of the system x are thecoefficients in an expression of b as a linear combination of the columnvectors of the coefficient matrix A.

As an example, consider the system of equations Ax = b in Example2, where

A =

(1 −2 −1

−3 1 2

), b =

( −5−7

), x =

x1

x2

x3

One representation of this system as a set of equations that shouldhold simultaneously is

A1.x = b1 which is x1 − 2x2 − x3 = −5

A2.x = b2 which is −3x1 + x2 + 2x3 = −7

Another representation of this system as a vector equation is

x1A.1 + x2A.2 + x3A.3 = b

i.e. x1

(1

−3

)+ x2

( −21

)+ x3

( −12

)=

( −5−7

)

200 Ch. 2 . Matrices

The basic solution of this system corresponding to the basic vec-tor of variables (x1, x3) obtained by the GJ pivotal method is x =(17, 0, 22)T . It satisfies this vector equation because

17

(1

−3

)+ 0

( −21

)+ 22

( −12

)=

( −5−7

)

The general solution of this system is

x1

x2

x3

=

17 − 3x2

x2

22 − 5x2

where x2 is an arbitrary real valued parameter, and it can be verifiedthat this satisfies the above vector equation.

Cautionary Note: Care in writing matrix products: This isfor those students who are still not totally familiar with matrix manip-ulations. In real number arithmetic, the equation

5 × 4 = 20

can be written in an equivalent manner as

5 =20

4

Carrying this practice to equations involving matrix products willmost often result in drastic blunders. For example, writing

Ax = b as A =b

xor as x =

b

Ais a blunder

because division by vectors is not defined, inverse of a rectangular ma-trix is not defined; and even if the matrix A has an inverse to be definedlater, it is not denoted by

1A

but by a different symbol.Therefore deducing that x = b

Afrom Ax = b is always a drastic

blunder.

Ex ercis es

2.4 Matrix Addition & Transposition 201

2.3.1: For A, b given below, write the system of linear equationsAx = b as a vector equation.

(i) A =

1 2 0 9−1 0 1 1

0 3 4 −7

, b =

31

−4

.

(ii) A =

1 1 1 1−1 2 1 2

1 −2 1 −20 −1 −2 0

, b =

9−7

04

.

2.4 Scalar Multiplication and Addition of

Matrices, Transposition

Let A = (aij) be a matrix of order m× n. Scalar multiplication of thismatrix A by a scalar α involves multiplying each element in A by α.Thus

αA = α(aij) = (αaij)

For instance

6

(1 −2 −1

−3 1 2

)=

(6 −12 −6

−18 6 12

).

Given two matrices A and B, their sum is only defined if bothmatrices are of the same order. So, if A = (aij) and B = (bij) areboth of order m×n, then their sum A+B is obtained by summing upcorresponding elements, and is equal to (aij + bij). For instance

(1 −2 −1

−3 1 2

)+

(1 00 1

)is not defined.

(1 −2 −1

−3 1 2

)+

(3 −10 188 −7 20

)=

(4 −12 175 −6 22

)

202 Ch. 2. Matrices

The operation of transposition of a matrix changes its row vectorsinto column vectors and vice versa. Thus if A is an m × n matrix, itstranspose denoted by AT is an n × m matrix obtained by writing therow vectors of A as column vectors in AT in their proper order. Forinstance

(1 −2 −1

−3 1 2

)T

=

1 −3−2 1−1 2

(3 9

18 −33

)T

=

(3 189 −33

).

Earlier in Section 1.4 we defined the transpose of a row vector as acolumn vector and vice versa. That definition tallies with the definitionof transpose for matrices given here when those vectors are looked atas matrices with a single row or column.

Some Properties of Matrix Addition and Trans-position

Let At = (atij) be a matrix of order m × n for t = 1 to r. The

sum, and a linear combination with multipliers α1, . . . , αr respectively,of these r matrices are defined similarly. We have

A1 + . . . + Ar = (r∑

t=1

atij)

α1A1 + . . . + αrA

r = (r∑

t=1

αtatij)

In finding the sum of two or more matrices, these matrices can bewritten in the sum in any arbitrary order, the result does not dependon this order. For instance

(1 −2 30 8 −7

)+

(3 6 −3

−4 −2 −1

)+

( −1 −8 −3−2 −4 13

)

=

( −1 −8 −3−2 −4 13

)+

(1 −2 30 8 −7

)+

(3 6 −3

−4 −2 −1

)

2.4 Matrix Addition & Transp o sition 203

=

(3 −4 −3

−6 2 5

)

Let A be an m×n matrix, u = (u1, . . . , un), v = (v1, . . . vn)T . Thenit can be verified that

(Au)T = u T AT (vA)T = AT v

T

So, the transpose of a matrix vector product is the product of theirtransposes in reverse order. For example, let

A =

(1 −2 30 8 −7

), u =

32

−1

, v = (5, 10).

Then

Au =

(1 −2 30 8 −7

)

32

−1

=

( −423

)

vA = (5, 10)

(1 −2 30 8 −7

)= (5, 70,−55)

u T AT = (3, 2,−1)

1 0−2 8

3 −7

= (−4, 23)

AT v T =

1 0−2 8

3 −7

(

510

)=

570

−55

So, we verify that (Au)T = u TAT and (vA)T = AT v

T .

Ex ercis es

2.4.1: Write the transposes of the following matrices. Also, men-tion for which pair U, V of these matrices is 5U − 3V defined, andcompute the result if it is.

204 Ch. 2. Matrices

A =

(1 −3 0

−2 2 −2

), B =

(1 −1

−2 3

), C =

0 3−1 −2

1 −3

,

D =

(2 31 −2

).

2.5 Matrix Multiplication & Its Various

Interpretations

Rules for matrix multiplication evolved out of a study of transforma-tions of variables in systems of linear equations. Let A = (aij) be thecoefficient matrix of order m × n in a system of linear equations, bthe RHS constants vector, and x = (x1, . . . , xn)T the column vector ofvariables in the system. Then the system of equations is Ax = b inmatrix notation. Here are all the equations in it.

a11x1 + . . . + a1nxn = b1

......

ai1x1 + . . . + ainxn = bi

......

am1x1 + . . . + amnxn = bm

Now suppose we transform the variables in this system using thetransformation x = By where B = (bij) is an n × n matrix, andy = (y1, . . . yn) are the new variables. So the transformation is:

x1 = b11y1 + . . . + b1nyn

......

xj = bj1y1 + . . . + bjnxn

......

xn = bn1y1 + . . . + bnnyn

2.5 Matrix Multiplication 205

Substituting these expressions for x1, . . . , xn in the original systemof equations and regrouping terms will modify it into the transformedsystem in terms of the new variables y. Suppose the transformed sys-tem is Cy = b with C as its coefficient matrix. Alternately,substituting x = By in Ax = b gives ABy = b; hence wedefine the matrix product AB to be C.

We will illustrate this with a numerical example. Let

A =

(1 −2 −1

−3 1 2

), b =

( −5−7

), x = (x1, x2, x3)

T

and consider the system of equations Ax = b, which is

x1 − 2x2 − x3 = −5

−3x1 + x2 + 2x3 = −7

Suppose we wish to transform the variables using the transforma-tion

x1 = 10y1 + 11y2 + 12y3

x2 = 20y1 + 21y2 + 22y3

x3 = 30y1 + 31y2 + 32y3

or x = By where

B =

10 11 1220 21 2230 31 32

, y =

y1

y2

y3

.

Substituting the expressions for x1, x2, x3 in terms of y1, y2, y3 inthe two equations of the system above, we get

1(10y1+11y2+12y3)−2(20y1+21y2+22y3)−1(30y1+31y2+32y3) = −5

−3(10y1+11y2+12y3)+1(20y1+21y2+22y3)+2(30y1+31y2+32y3) = −7

Regrouping the terms, this leads to the system

206 Ch. 2. Matrices

(1,−2,−1)(10, 20, 30)Ty1 + (1,−2,−1)(11, 21, 31)Ty2

+(1,−2,−1)(12, 22, 32)Ty3 = −5

(−3, 1, 2)(10, 20, 30)Ty1 + (−3, 1, 2)(11, 21, 31)Ty2

+(−3, 1, 2)(12, 22, 32)Ty3 = −7

or Cy = b where y = (y1, y2, y3)T and

C =

(A1.B.1 A1.B.2 A1.B.3

A2.B.1 A2.B.2 A2.B.3

)

So, we should define the matrix product AB, in this order, to bethe matrix C defined by the above formula. This gives the reason forthe definition of matrix multiplication given below.

Matrix Multiplication

If A, B are two matrices, the product AB in that order is definedonly if

the number of columns in A is equal to the number of rowsin B

i.e., if A is of order m×n, and B is of order r×s, then for the productAB to be defined n must equal r; otherwise AB is not defined.

If A is of order m × n and B is of order n × s, then AB = C is oforder m × s, and the (i, j)th entry in C is the dot product Ai.B.j.

Therefore matrix multiplication is row by column multiplica-tion, and the (i, j)th entry in the matrix product AB is the dot productof the ith row vector of A and the jth column vector of B.

Examples of Matrix Multiplication

(1 −2 −1

−3 1 2

)(1 00 1

)is not defined

2.5 Matrix Multiplication 207

(1 −2 −1

−3 1 2

)

10 11 1220 21 2230 31 32

=

( −60 −62 −6450 50 50

)

(1 −1 −52 −3 −6

)

−2 −4−6 −4

3 −2

=

( −11 10−4 16

)

The important thing to remember is that matrix multiplication isnot commutative, i.e., the order in which matrices are multiplied isvery important. When the product AB exists, the product BA maynot even be defined, and even if it is defined, AB and BA may be ofdifferent orders; and even when they are of the same order they maynot be equal.

Examples

Let

A =

(1 −2 −1

−3 1 2

), B =

10 11 1220 21 2230 31 32

,

D =

−2 −4−6 −4

3 −2

, E =

0 1 00 0 11 0 0

AB is computed above, and BA is not defined.

AD =

(7 66 4

), DA =

10 0 −66 8 −29 −8 −7

BE =

12 10 1122 20 2132 30 31

, EB =

20 21 2230 31 3210 11 12

So, AD and DA are of different orders. BE and EB are both ofthe same order, but they are not equal.

208 Ch. 2. Matrices

Thus, when writing a matrix product, one should specify the orderof the entries very carefully.

We defined matrix-vector multiplication in Section 2.2. That defi-nition coincides with the definition of matrix multiplication given herewhen the vectors in the product are treated as matrices with a singlerow or column respectively.

In the matrix product AB we say that A premultiplies B; or Bpostmultiplies A.

Column-Row Products

Let u = (u1, . . . , um)T be a column vector in Rm, and v = (v1, . . . , vn)a row vector in Rn. Then the product uv in that order, known as theouter product of the vectors u, v is the matrix product uv whenu, v are treated as matrices with one column and one row respectively,given by

uv =

u1v1 u1v2 . . . u1vn

u2v1 u2v2 . . . u2vn...

...umv1 umv2 . . . umvn

As an example

10

−12

(3, 4) =

3 40 0

−3 −46 8

Notice the difference between the inner (or dot) product of twovectors defined in Section 1.5, and the outer product of two vectorsdefined here. The inner product is always a real number, and it isonly defined for a pair of vectors from the same dimensional space.The outer product is a matrix, and it is defined for any pair of vectors(they may be from spaces of different dimensions).

2.5 Matrix Multiplication 209

Matrix Multiplication Through a Series of Matrix-Vector Products

Let A, B be matrices of orders m × n, n × k respectively. We canwrite the product AB as

AB = A(B.1... . . .

...B.k)

= (AB.1... . . .

...AB.k)

where for each t = 1 to k, B.t is the tth column vector of B, and AB.t

is the column vector that is the result of a matrix-vector product asdefined earlier in Section 2.2. Thus the product AB can be looked atas the matrix consisting of column vectors AB.t in the order t = 1 tok.

Another way of looking at the product AB is

AB =

A1....

Am.

B =

A1.B...

Am.B

From these interpretations of matrix multiplication, we see that the

jth column of the product AB is AB.j

ith row of the product AB is Ai.B

Matrix Product as Sum of a Series of Column-Row Products

Let A = (aij), B = (bij) be two matrices such that the product ABis defined. So

the number of columns in A = number of rows in B = n,say.

210 Ch. 2. Matrices

Let A.1, . . . , A.n be the column vectors of A; and B1., . . . , Bn. be therow vectors in B. For the sake of completeness suppose A is of orderm × n, and B is of order n × k.

From the definition of the matrix product AB we know that its(i, j)th element is

n∑t=1

aitbtj = ai1b1j + . . . + ainbnj

So,

AB =n∑

t=1

(matrix of order m × k whose (i, j)th element is aitbtj)

However, the matrix whose (i, j)th element is aitbtj is the outerproduct A.tBt.. Hence

AB =n∑

t=1

A.tBt. = A.1B1. + . . . + A.nBn.

This is the formula that expresses AB as a sum of n column-rowproducts. As an example let

(1 −2 −1

−3 1 2

), B =

10 11 1220 21 2230 31 32

Then

A.1B1. =

(1

−3

)(10, 11, 12) =

(10 11 12

−30 −33 −36

)

A.2B2. =

( −21

)(20, 21, 22) =

( −40 −42 −4420 21 22

)

A.3B3. =

( −12

)(30, 31, 32) =

( −30 −31 −3260 62 64

)

So

2.5 Matrix Multiplication 211

A. 1B1. + A. 2B2. + A. 3B3. =

( −60 −62 −6450 50 50

)

which is the same as the product AB computed earlier.

Exercises

2.5.1 Let A, B be m × n, n × k matrices respectively. Show that

each column of AB is a linear combination of the columnsof A

each row of AB is a linear combination of the rows of A.

The Product of a Sequence of 3 Or More Matri-ces

Let A1, A2, . . . , At be t ≥ 3 matrices. Consider the matrix product

A1A2 . . . At

in this specific order. This product is defined iff

the product of every consecutive pair of matrices in thisorder is defined,

i.e., for each r = 1 to t − 1, the number of rows in Ar isequal to the number of columns in Ar+1.

When these conditions are satisfied, this product of t matrices iscomputed recursively, i.e., take any consecutive pair of matrices in thisorder, say Ar an Ar+1 for some 1 ≤ r ≤ t−1, and suppose ArAr+1 = D.Then the above product is

A1 . . . Ar−1DAr+2 . . . At

.

212 Ch. 2. Matrices

There are only t − 1 matrices in this product, it can be reduced toa product of t − 2 matrices the same way, and the same procedure iscontinued until the product is obtained as a single matrix.

When it is defined, the product A1A2 . . . At is of order m1×nt where

m1 = number of rows in A1, the leftmost matrix in theproduct

nt = number of columns in At, the rightmost matrix in theproduct.

The product A1A2 . . . At is normally computed by procedures knownas string computation, of which there are two.

The left to right string computation for A1A2 . . . At computesDr = A1 . . . Ar for r = 2 to t in that order using

D2 = A1A2

Dr+1 = DrAr+1, for r = 2 to t − 1

to obtain the final result Dt.The right to left string computation for A1A2 . . . At computes

Er = Ar . . . At for r = t − 1 to 1 in that order using

Et−1 = At−1At

Er−1 = Ar−1Er, for r = t − 1 to 2

to obtain the final result E1, which will be the same as Dt obtained bythe previous recursion.

These facts follow by repeated application of the following result.

Result 2.5.1: Product involving three matrices: Let A, B, Cbe matrices of orders m × n, n × p, p × q respectively. Then A(BC) =(AB)C.

To prove this result, suppose BC = D. Then from one of theinterpretations of two-matrix products discussed earlier we know that

2.5 Matrix Multiplication 213

D = BC = (BC.1... . . .

...BC.q)

So,

A(BC) = AD = A(BC.1... . . .

...BC.q) = (A(BC.1)... . . .

...A(BC.q))

Also, let C = (cij). Then, for each j = 1 to q, BC.j =∑p

i=1 cijB.i.So

A(BC.j) = A(p∑

i=1

cijB.i) =p∑

i=1

cij(AB.i)

So, A(BC) is an m × q matrix whose jth column is∑p

i=1 cij(AB.i)for j = 1 to q.

Also, from another interpretation of two-matrix products discussedearlier we know that

AB = (AB.1... . . .

...AB.p)

So,

(AB)C = (AB.1... . . .

...AB.p)

C1....

Cp.

= (AB.1)C1. + . . . + (AB.p)Cp.

Notice that for each i = 1 to p, (AB.i)Ci. is an outer product (acolumn-row product) that is an m × q matrix whose jth column iscij(AB.i) for j = 1 to q. Hence, (AB)C is an m × q matrix whose jthcolumn is

∑pi=1 cij(AB.i) for j = 1 to q; i.e., it is the same as A(BC)

from the fact established above. Therefore

A(BC) = (AB)C

establishing the result.

214 Ch. 2. Matrices

Properties Satisfied By Matrix Products

Order important: The thing to remember is that the order inwhich the matrices are written in a matrix product is very important.If the order is changed, the product may not be defined, and even if itis defined, the result may be different. As examples, let

A =

( −3 0 1−2 −6 4

), B =

−5 −1020 301 2

, C =

345

Then

AC =

( −4−10

), and CA is not defined

AB =

( −14 −28−106 −152

), BA =

35 60 −45−120 −180 140−7 −12 9

so, AB and BA are not even of the same order.

Associative law of multiplication: If A, B, C are matrices suchthat the product ABC is defined, then A(BC) = (AB)C.

Left and Right distributive laws: Let B, C be matrices of thesame order. If A is a matrix such that the product A(B+C) is defined,it is = AB + AC. If A is a matrix such that the product (B + C)A isdefined, it is = BA + CA. So, matrix multiplication distributes acrossmatrix addition.

If α, β are scalars, and A, B two matrices such that the Product ABis defined, then, (α + β)A = αA + βA; and αAB = (αA)B = A(αB).

The product of two nonzero matrices may be zero (i.e., a matrix inwhich all the entries are zero). For example, for 2 × 2 matrices A, Bboth of which are nonzero, verify that AB = 0.

A =

(1 11 1

), B =

(1 1

−1 −1

)

2.5 Matrix Multiplication 215

If A, B, C are matrices such that the products AB, AC are bothdefined; and AB = AC, we cannot, in general, conclude that B = C.Verify this for the following matrices A, B, C, we have AB = AC, eventhough B �= C.

A =

(1 11 1

), B =

(1 1

−1 −1

), C =

(2 2

−2 −2

)

Matrix multiplication and transposes: If A, B are two matri-ces such that the product AB is defined, then the product B

T AT isdefined and (AB)T = B

T AT . This can be verified directly from thedefinitions.

In the same way if the matrix product A1A2 . . . Ar is defined, then(A1A2 . . . Ar)

T = ATr A

Tr − 1 . . . A

T2 A

T1 .

Ex ercis es

2.5.2: For every pair of matrices U, V among the following, deter-mine whether UV is defined, and compute it if it is.

A =

1 −3 2 0−1 2 3 4

2 1 1 −1

, B =

0 −2 11 0 2

−1 −3 0−2 −7 6

,

C =

4 −5 2−1 1 −2

0 1 −1

, D =

5 −5 0 3−1 −1 −1 −1

1 2 2 10 −2 −2 −1

,

E =

(2 −34 −2

), F =

(1 −2 3 −4

−1 3 −2 2

).

2.5.3: Sometimes when A, B are a pair of matrices, both productsAB, BA may exist. These products may be of different orders, buteven when they are of the same order they may or may not be equal.Verify these with the following pairs.

216 Ch. 2. Matrices

(i): A =

1 7 −7 20 −5 3 −11 1 1 1

, B =

1 1 1−1 1 −1

1 −1 −1−1 −1 1

.

(ii): A =

(1 −11 1

), B =

(2 3

−3 2

).

(iii): A =

(8 3

−4 −2

), B =

(2 96 3

).

2.5.4: Compute the products ADB, DBA, BAD, EFDB wherethe matrices A to F are given in Exercise 2.5.2.

2.6 Some Special Matrices

Square Matrices

A matrix of order m × n is said to be a

square matrix if m = nrectangular matrix that is not square if m �= n.

Hence a square matrix is a matrix of order n× n for some n. Sincethe number of rows and the number of columns in a square matrixare the same, their common value is called the order of the squarematrix. Thus an n × n square matrix is said to be of order n. Hereare examples of square matrices.

(2 −31 −10

),

−3 −4 −56 7 8

−9 10 11

In a square matrix (aij) of order n, the entries a11, a22, . . . , ann con-stitute its diagonal entries or its main diagonal. All the otherentries in this matrix are called off-diagonal entries. Here is a pic-ture

2.6 Special Matrices 217

a11 a12 . . . a1,n−1 a1n

a21 a22 . . . a2,n−1 a2n

......

......

an−1,1 an−1,2 . . . an−1,n−1 an−1,n

an1 an2 . . . an,n−1 ann

A square matrix with its diagonal entries boxed.

Unit (Identity) Matrices

Unit matrices are square matrices in which all diagonal entries are“1” and all off-diagonal entries are “0”. Here are the unit matrices(also called identity matrices) of some orders.

(1 00 1

),

1 0 00 1 00 0 1

,

1 0 0 00 1 0 00 0 1 00 0 0 1

The reason for the name is that when a matrix A is multiplied bythe unit matrix I of appropriate order so that the product is defined,the resulting product IA or AI is A itself. Here are some examples.

(1 00 1

)(1 −2 −1

−3 1 2

)=

(1 −2 −1

−3 1 2

)

(1 −2 −1

−3 1 2

)

1 0 00 1 00 0 1

=

(1 −2 −1

−3 1 2

)

Hence, in the world of matrices, the unit matrices play the samerole as the number “1” does in the world of real numbers. This is thereason for their name.

218 Ch. 2. Matrices

The unit matrix is usually denoted by the symbol I when its ordercan be understood from the context. If the order has to be indicatedspecifically, the unit matrix of order n is usually denoted by the symbolIn.

In Chapter 1 we defined unit vectors as column vectors with a singlenonzero entry of “1”. From this it is clear that the unit vectors arecolumn vectors of the unit matrix. Thus if I is the unit matrix of ordern, then its column vectors I.1, I.2, . . . , I.n are the 1st, 2nd, . . ., nth unitvectors in Rn.

Permutation Matrices

A square matrix of order n in which all the entries are 0 or 1, andthere is exactly a single “1” entry in each row and in each columnis called a permutation matrix or an assignment. A permutationmatrix can always be transformed into the unit matrix by permuting(i.e., rearranging in some order) its rows, or its columns. Here are somepermutation matrices.

(0 11 0

),

0 1 01 0 00 0 1

,

0 1 0 01 0 0 00 0 0 10 0 1 0

Premultiplying any matrix A by a permutation matrix of appro-proate order so that the product is defined, permutes its rows. In thesame way, postmultiplying any matrix A by a permutation matrix ofapproproate order so that the product is defined, permutes its columns.Here are some examples.

(0 11 0

)(1 −2 −1

−3 1 2

)=

( −3 1 21 −2 −1

)

(1 −2 −1

−3 1 2

)

0 0 10 1 01 0 0

=

( −1 −2 12 1 −3

)

The unit matrix is also a permutation matrix. Also, it can beverified that if P is a permutation matrix, then

2.6 Special Matrices 219

PP T = P T P = I, the unit matrix of the same order.

Diagonal Matrices

A square matrix with all its off-diagonal entries zero, and all di-agonal entries nonzero is called a diagonal matrix. Here are someexamples.

(3 00 −2

),

−2 0 00 −1 00 0 −8

,

1 0 0 00 2 0 00 0 3 00 0 0 4

Since all the off-diagonal entries in a diagonal matrix are knownto be zero, if we are given the diagonal entries in it, we can con-struct the diagonal matrix. For this reason, a diagonal matrix oforder n with diagonal entries a11, a22, . . . , ann is denoted by the sym-bol diag{a11, a22, . . . , ann}. In this notation, the three diagonal ma-trices given above will be denoted by diag{3,−2}, diag{−2,−1,−8},diag{1, 2, 3, 4} respectively.

Premultiplying a matrix A by a diagonal matrix of appropriateorder so that the product is defined, multiplies every entry in the ithrow of A by the ith diagonal entry in the diagonal matrix; this operationis called scaling the rows of A. Similarly, postmultiplying a matrixA by a diagonal matrix scales the columns of A.

(3 00 −2

)(1 −2 −1

−3 1 2

)=

(3 −6 −36 −2 −4

)

(1 −2 −1

−3 1 2

)

−2 0 00 −1 00 0 −8

=

( −2 2 86 −1 −16

)

220 Ch. 2. Matrices

Upper and Lower Triangular Matrices, Triangu-lar Matrices

We already discussed upper, lower triangular tableaus; and trian-gular tableaus in Section 1.18. The corresponding matrices are calledupper triangular, lower triangular, or triangular matrices. We giveformal definitions below.

A square matrix is said to be upper triangular if all entries in itbelow the diagonal are zero, and all the diagonal entries are nonzero. Itis said to be lower triangular if all the diagonal entries are nonzero,and all the entries above the diagonal are zero. It is said to be trian-gular if its rows can be rearranged to make it upper triangular. Hereare examples.

1 1 1 10 1 1 10 0 1 10 0 0 1

,

1 0 0 01 1 0 01 1 1 01 1 1 1

0 0 0 10 0 1 11 1 1 10 1 1 1

The first matrix is upper triangular, the second is lower triangular,and the third is triangular.

Symmetric Matrices

A square matrix D = (dij) is said to be symmetric iff DT = D,i.e., dij = dji for all i, j.

A square matrix which is not symmetric is said to be an asymmet-ric matrix.

If D is an asymmetric matrix, (1/2)(D + DT ) is called its sym-metrization.

6 1 −1 −31 0 −1 0

−1 −1 −2 1−3 0 1 3

,

0 2 00 0 22 0 0

,

0 1 11 0 11 1 0

The first matrix is symmetric, the second is asymmetric, and thethird is the symmetrization of the second matrix.

2.6 Special Matrices 221

Submatrices of a Matrix

Let A = (aij) be a given matrix of order m×n. Let R ⊂ {1, . . . , m},C ⊂ {1, . . . , n}, in both of which the entries are arranged in increasingorder. By deleting all the rows of A not in R, and all the columns of Anot in C, we are left with a matrix which is known as the submatrixof A determined by the subsets R, C of rows and columns anddenoted usually by ARC . For example, let

A =

3 −1 1 1 00 1 3 4 04 −2 0 1 15 −3 0 0 2

Let R = {1, 3, 4}, C = {1, 2, 4, 5}. Then the submatrix correspond-ing to these subsets is

ARC =

3 −1 1 04 −2 1 15 −3 0 2

Principal Submatrices of a Square Matrix

Let D = (dij) be a given square matrix of order n. Let Q ⊂{1, . . . , n}, in which the entries are arranged in increasing order. Bydeleting all the rows and columns of D not in Q, we are left with amatrix which is known as the principal submatrix of D determinedby the subset Q of rows and columns and denoted usually by DQQ.For example, let

D =

3 −1 1 1 00 1 3 4 04 −2 0 1 15 −3 0 0 28 −17 −18 9 11

Let Q = {2, 3, 5}. Then the principal submatrix of D determinedby the subset Q is

222 Ch. 2 . Matrices

DQQ =

1 3 0−2 0 1−17 −18 11

Principal submatrices are only defined for square matrices, and themain diagonal of a principal submatrix is always a subset of the maindiagonal of the original matrix.

Ex ercis es

2.6.1: Let

A =

−20 10 30 4 −1718 0 3 2 −190 6 −7 8 12

13 −6 19 33 1412 −9 22 45 51

Write the submatrix of A corresponding to the subset of rows {2, 4}and the subset of columns {1, 3, 5}. Also write the principal submatrixof A determined by the subset of rows and columns {2, 4, 5}.

2.7 Row Operations and Pivot Steps on a

Matrix

Since a matrix is a 2-dimensional array of numbers, it is like the tableauwe discussed earlier. Rows and columns of the matrix are exactly likethe rows and columns of a tableau.

Row operations on a matrix are exactly like row operations on atableau. They involve the following:

1. Scalar Multiplication: Multiply each entry in a rowby the same nonzero scalar

2.7 R ow Op erations 223

2. Add a Scalar Multiple of a Row to Another:Multiply each element in a row by the same nonzero scalarand add the result to the corresponding element of the otherrow.

3. Row interchange: Interchange two rows in the matrix.

Example : Consider the matrix

A =

−1 2 1 30 −3 9 78 6 −4 −5

Multiplying A3. = Row 3 of A, by −2 leads to the matrix

A′ =

−1 2 1 30 −3 9 7

−16 −12 8 10

Performing the row operation A2. − 2A1. leads to the matrix

A′′ =

−1 2 1 32 −7 7 18 6 −4 −5

A GJ (Gauss-Jordan) pivot step on a matrix is exactly like a GJpivot step on a tableau. It is specified by choosing a row of the matrixas the pivot row, and a column of the matrix as the pivot column, withthe element in the pivot row and pivot column called the pivot elementwhich should be nonzero. The GJ pivot step then converts the pivotcolumn into the unit column with a “1” entry in the pivot row and “0”entries in all other rows, by appropriate row operations.

A G (Gaussian) pivot step on a matrix is like the GJ pivot step, withthe exception that it only converts all entries in the pivot column belowthe pivot row into zeros by appropriate row operations, but leaves allthe rows above the pivot row and the pivot row itself unchanged.

224 Ch. 2. Matrices

2.8 Determinants of Square Matrices

History of determinants: Determinants are real valued functions ofsquare matrices, i.e., associated with every square matrix is a uniquereal number called its determinant. Determinants are only definedfor square matrices. There is no such concept for rectangular matri-ces that are not square. The definition of the determinant of a squarearray of numbers goes back to the the end of the 17th century in theworks of Seki Takakazu (also called Seki Kowa in some books) of Japanand Gottfried Leibnitz of Germany. Seki arrived at the notion of a de-terminant while trying to find common roots of algebraic equations.To find common roots of polynomials f(x), g(x) of small degrees Sekigot determinant expressions and published a treatize in 1674. Lebnitzdid not publish the results of his studies related to determinants, butin a letter to l’Hospital in 1693 he wrote down the determinant con-dition of compatiability for a system of three linear equations in twounknowns. In Europe the first publication mentioning determinantswas due to Cramer in 1750 in which he gave a determinant expressionfor the problem of finding a conic through five given points (this leadsto a system of linear equations).

Since then determinants have been studied extensively for their the-oretical properties and their applications in linear algebra theory. Eventhough determinants play a very major role in theory, they have notbeen used that much in computational algorithms. Since the focus ofthis book is computational linear algebra, we will list all the impor-tant properties of determinants without detailed proofs. References forproofs of the results are provided for interested readers to pursue. ��

There are several equivalent ways of defining determinants, but allthese satisfy the following fundamental result.

Result 2.8.1: One set of properties defining a determinant:The determinant of a square matrix of order n is the unique real valuedfunction of the matrix satisfying the following properties.

(a) If A = (aij) of order n is lower or upper triangular,then the determinant of A is the product of the diagonal

2.8 Determinants 225

entries of A.

(b) det(AT ) = det(A) for all square matrices A.

(c) If A, B are two square matrices of order n, then det(AB)= det(A)det(B).

For n = 1, 2, determinants of square matrices of order n can bedefined very easily.

The determinant of a 1 × 1 matrix (a11) is defined to be a11. So,det(0) = 0, det(−4) = −4, det(6) = 6, etc.

For a 2×2 matrix A =

(a11 a12

a21 a22

), det(A) is defined to be a11a22−

a12a21. For example

det

(3 21 4

)= 3 × 4 − 2 × 1 = 12 − 2 = 10

det

(1 43 2

)= 1 × 2 − 4 × 3 = 2 − 12 = −10

det

(1 4

−3 −12

)= 1 × (−12) − (−3) × 4 = −12 + 12 = 0

The original concept of the determinant of a square matrix A = (aij)of order n for n ≥ 2 consists of a sum of n! terms, half with a coefficientof +1, and the other half with a coefficient of −1. We will now explainthis concept.

Some preliminary definitions first. In linear programming litera-ture, each permutation matrix of order n is referred to as an assign-ment of order n. Here, for example, is an assignment of order 4.

0 1 0 00 0 0 11 0 0 00 0 1 0

Hence an assignment of order n is a square matrix of order n thatcontains exactly one nonzero entry of “1” in each row and column.Verify that there are n! distinct assignments of order n.

226 Ch. 2. Matrices

We number the rows and columns of an assignment in natural order,and refer to the (i, j)th position in it as cell (i, j) (this is in row iand column j). We will represent each assignment by the subset ofcells in it corresponding to the “1” entries written in natural orderof rows of the matrix. For example, the assignment of order 4 givenabove corresponds to the set of cells {(1, 2), (2, 4), (3, 1), (4, 3)} in thisrepresentation.

So, in this notation a general assignment or permutation matrix oforder n can be represented by the set of cells {(1, j1), (2, j2), (3, j3), . . . ,(n, jn)} where (j1, j2, . . . , jn) is a permutation of (1, 2, . . . , n), i.e., anarrangement of these integers in some order. We will say that thispermutation matrix and permutation correspond to each other. Forexample, the permutation matrix of order 4 given above correspondsto the permutation (2, 4, 1, 3).

There are 6 = 3! different permutations of (1, 2, 3). These are:(1, 2, 3), (1, 3, 2), (2, 1, 3), (2, 3, 1), (3, 1, 2), (3, 2, 1). In the same waythere are 24 = 4! permutations of (1, 2, 3, 4); and in general n! permu-tations of (1, 2, . . . , n).

Consider the permutation (j1, . . . , jn) corresponding to the permu-tation represented by the set of cells {(1, j1), . . . , (n, jn)}. We will usethe symbol p to represent either the permutation or the correspondingpermutation matrix. The determinant of the square matrix A = (aij)of order n contains one term corresponding to the permutation p, it is

(−1)NI(p)a1j1a2j2 . . . anjn

where NI(p) is a number called the number of inversions in thepermutations p, which we will define below. The determinant of A isthe sum of all the terms corresponding to all the n! permutations.

In the permutation (j1, . . . , jn), consider a pair of distinct entries(jr, js) where s > r. If js < jr [js > jr] we say that this pair contributesone [0] inversions in this permutation. The total number of inversionsin this permutation is obtained by counting the contributions of allpairs of the form (jr, js) for s > r in it. It is equal to

∑n−1r=1 qr where

qr = number of entries in (jr+1, . . . , jn) which are < jr.

2.8 Determinants 227

A permutation is called an even (odd) permutation if the num-ber of inversions in it is an even (odd) integer. As an example considerthe permutation (6, 1, 3, 4, 5, 2). Here

j1 = 6, all of j2, j3, j4, j5, j6 are < j1, so q1 = 5.

j2 = 1, no. entries among (j3, j4, j5, j6) < j2 is 0, so q2 = 0.

j3 = 3, no. entries among (j4, j5, j6) < j3 is 1, so q3 = 1.

j4 = 4, no. entries among (j5, j6) < j4 is 1, so q4 = 1.

j5 = 5, no. entries among (j6) < j5 is 1, so q5 = 1.

So the number of inversions in this permutation is 5 + 0 + 1 + 1 + 1= 8, hence this permutation is an even permutation.

In the same way verify that the number of inversions in the permu-tation (2, 4, 1, 3) is 1 + 2 + 0 = 3, hence this is an odd permutation.

For example, for n = 3, there are 3! = 6 permutations. Thesepermutations, and the formula for the determinant of A = (aij) oforder 3 are given below.

Permutation p NI(p) Term corresponding to p(1, 2, 3) 0 a11a22a33

(1, 3, 2) 1 −a11a23a32

(2, 3, 1) 2 a12a23a31

(2, 1, 3) 1 −a12a21a33

(3, 1, 2) 2 a13a21a32

(3, 2, 1) 3 −a13a22a31

Determinant(A) a11a22a33 −a11a23a32 +a12a23a31

−a12a21a33 +a13a21a32 −a13a22a31

In the same way, the original formula for the determinant of anysquare matrix of any order can be derived. However, since it is in theform of a sum of n! terms, this formula is not used for computing thedeterminant when n > 2. There are much simpler equivalent formulasthat can be used to compute determinants far more efficiently.

228 Ch. 2 . Matrices

Recursive Definition of the Determinant of ann × n Matrix

We will now give a recursive definition of the determinant of ann × n matrix in terms of the determinants of its (n − 1) × (n − 1)submatrices. This definition follows from the one given above. We canuse this definition to compute the determinant of a 3× 3 matrix usingthe values of determinants of its 2 × 2 submatrices which themselvescan be computed by the formula given above. The determinant of amatrix of order 4 × 4 can be computed using the determinants of its3 × 3 submatrices, and so on.

Given a square matrix A = (aij), we will denote by Aij the subma-trix obtained by deleting the ith row and the jth column from A, thissubmatrix is called a minor of A.

Example: Let

A =

1 −2 0 0−1 7 8 5

3 4 −3 −52 9 −4 6

A32 =

1 0 0−1 8 5

2 −4 6

Given the square matrix A = (aij), the (i, j)th cofactor of A,denoted by Cij is given by

Cij = (−1)i+ jdeterminant(Aij)

where Aij is the minor of A obtained by deleting the ith row and thejth column from A.

So, while the minor Aij is a matrix, the cofactor C ij is a real number.

Example: Let A =

3 1 2−2 −1 1

8 1 0

. Then

2 .8 D etermina nt s 229

A11 =

( −1 11 0

), C11 = (−1)1+ 1det

( −1 11 0

)= −1

A12 =

( −2 18 0

), C12 = (−1)1+ 2det

( −2 18 0

)= 8

A13 =

( −2 −18 1

), C13 = (−1)1+ 3det

( −2 −18 1

)= 6

Definition: Cofactor expansion of a determinant: For n ≥ 2,the determinant of the n × n matrix A = (aij) can be computed by aprocess called the cofactor expansion (or also Laplace expansion)across any row or down any column. Using the notation developedabove, this expansion across the ith row is

det(A) = ai1C i1 + ai2C i2 + . . . + ainC in

The cofactor expansion down the jth column is

det(A) = a1jC 1j + a2jC 2j + . . . + anjC nj

Example: Let

A =

2 4 03 3 −21 −1 0

Using cofactor expansion down the third column, we have

det(A) = 0C13 − 2C23 + 0C33

= −2(−1)2+3det

(2 41 −1

)= 2(2 ×−1 - 4 × 1) = −12

Using cofactor expansion across third row, we have

230 Ch. 2 . Matrices

det(A) = 1C31 − 1C 32

= (−1)3+ 1det

(4 03 −2

)−1(−1)3+ 2det

(2 03 −2

)

= −8 − 4 = −12, same value as above.If a row or column has many zero entries, the cofactor expansion

of the determinant using that row or column has many terms that arezero, and the cofactors in those terms need not be calculated. So, tocompute the determinant of a square matrix, it is better to choose arow or column with the maximum number of zero entries, for cofactorexpansion.

Exercises

2.8.1 Find the determinants of the following matrices with cofac-tor expansion using rows or columns that involve the least amount ofcomputation.

(i)

1 13 18 −940 −2 −11 120 0 7 140 0 0 −8

, (ii)

0 0 −4 33 0 0 82 1 −2 0

−1 3 4 0

(iii)

0 −8 0 04 0 0 00 0 0 30 0 −1 0

, (iv)

0 0 −2 −31 3 2 −40 0 0 −50 −1 −2 4

Some Results On Determinants

1. Determinants of upper (lower) triangular matrices: Thedeterminant of an upper triangular, lower triangular, or diagonal ma-trix is the product of its diagonal entries.

Let P be a permutation matrix. Its determinant is (−1)NI(P ) whereNI(P ) is the number of inversions in the permutation correspondingto P .

2.8 Determinants 231

2. Adding a scalar multiple of a row (column) to anotherrow (column) leaves determinant unchanged: If a square matrixB is obtained by adding a multiple of a row to another row (or byadding a multiple of a column to another column) in a square matrixA, then det(B) = det(A).

3. Determinant is 0 if matrix has a 0-row or 0-column: If asquare matrix has a zero row or a zero column, its determinant is 0.

4. Determinant is 0 if a row (column) is a scalar multipleof another row (column): If a row in a square matrix is a multipleof another row, or if a column in this matrix is a multiple of anothercolumn, then the determinant of that matrix is zero.

5. Interchanging a pair of rows (columns) multiplies de-terminant by −1: If a square matrix B is obtained by interchangingany two rows (or interchanging any two columns) of a square matrixA, then det(B) = − det(A).

6. Multiplying a row (column) by a scalar also multipliesdeterminant by that scalar: If a square matrix B is obtained bymultiplying each element in a row or a column of a square matrix A bythe same scalar α, then det(B) = αdet(A).

7. Multiplying all entries in a matrix of order n by scalarα, multiplies determinant by αn: If A is a square matrix of ordern, and α is a scalar, then det(αA) = αndet(A).

8. A matrix and its transpose have the same determinant:For any square matrix A, we have det(AT ) = det(A).

9. Determinant of a product of two matrices is the productof their determinants: If A, B are two square matrices of order n,then det(AB) = (det(A)) (det(B)).

An incomplete proof of this product rule for determinants was given

232 Ch. 2. Matrices

by J. P. M. Binet in 1813. The proof was corrected by A. L. Cauchyin 1815. Hence this result is known as the Cauchy-Binet theorem.

10. The effect of a GJ pivot step on the determinant: Ifthe square matrix A′ is obtained by performing a GJ pivot step onthe square matrix A with ars as the pivot element, then det(A′) =(1/ars)det(A).

11. The effect of a G pivot step on the determinant: Ifthe square matrix A′ is obtained by performing a G pivot step on thesquare matrix A, then

det(A′) = det(A) if no row interchange was performed onA before this G pivot step;

det(A′) = − det(A) if a row interchange was performed onA before this G pivot step.

12. Linearity of the Determinant function in a single col-umn or row of the matrix: The determinant of a square matrixA = (aij) of order n is a linear function of any single row or any sin-gle column of A, when all the other entries in the matrix are fixed atspecific numerical values.

The columns of A are A.j, j = 1 to n. Suppose, for some s, allthe entries in A.1, . . . , A.s−1, A.s+1, . . . , A.n are held fixed, and only theentries in A.s are allowed to vary over all possible real values. Thendet(A) is a linear function of the entries in A.s = (a1s, . . . , ans)

T . Thereason for this is the following. Let Cij denote the (i, j)th cofactor ofA. Then by cofactor expansion down the sth column, we have

det(A) = a1sC1s + a2sC2s + . . . + ansCns

The cofactors C1s, . . . , Cns depend only on entries in A.1, . . . , A.s−1,A.s+1, . . . , A.n which are all fixed, and hence these cofactors are allconstants here. Hence by the above equation and the definition of

2 .8 D etermina nt s 233

linear functions given in Section 1.7, det(A) is a linear function ofa1s, . . . , ans, the entries in A.s, which are the only variables here.

Ai., i = 1 to n are the row vectors of A. A similar argument showsthat if for some r, all the entries in A1., . . . , Ar − 1., Ar +1., . . . , An. areheld fixed, and only the entries in Ar. are allowed to vary over allpossible real values, then det(A) is a linear function of the entries inAr. = (ar 1, . . . , arn).

This leads to the following result. Suppose in the square matrixA of order n, the column vectors A. 1, . . . , A.s− 1, A.s+1, . . . , A.n are allfixed, and

A.s = βb + δd

where b, d are two column vectors in R n, and β, δ are two scalars. So,

the column vectors of A are as given below

A = (A. 1... . . .

...A.s− 1...βb + δd

...A.s+1... . . .

...A.n)

Define two matrices B, D with column vectors as given below.

B = (A. 1... . . .

...A.s− 1...b

...A.s+1... . . .

...A.n), D = (A. 1... . . .

...A.s− 1...d

...A.s+1... . . .

...A.n)

So, the matrices B, D differ from A only in their sth column. Thenby the linearity

det(A) = βdet(B) + δdet(D)

Caution: On the linearity of a determinant: For a squarematrix A, det(A) is a linear function when it is treated as a function ofthe entries in a single column, or a single row of A, while all the otherentries in A are held fixed at specific numerical values. As a functionof all the entries in A, det(A) is definitely not linear. That’s why, ifB, C are square matrices of order n, and A = B + C; these facts donot imply that det(A) = det(B) + det(C).

Example: Let

234 Ch. 2. Matrices

E =

(1 −23 4

), F =

( −1 −2−2 4

), A =

(0 −21 4

).

B =

(1 −13 2

), C =

( −1 −1−2 2

).

Then, A.2 = E.2 = F.2 and A.1 = E.1 + F.1. We have det(E) =10, det(F ) = −8, and det(A) = 2 = det(E) + det(F ), illustrating thelinearity result stated above.

We also have det(B) = 5, det(C) = −4. Verify that even thoughA = B + C, we have det(A) �= det(B) + det(C).

13. The determinant of a square matrix A is a multilinearfunction of A.

Definition of a multilinear function: Consider a real valuedfunction f(x1, . . . , xr) of many variables which are partitioned into vec-tors x1, . . . , xr; i.e., for each k = 1 to r, xk is itself a vector of variablesin Rnk say. f(x1, . . . , xr) is said to be a multilinear function underthis partition of variables if, for each t = 1 to r, the function

f(x1, . . . , xt−1, xt, xt+1, . . . , xr)

obtained by fixing xk = xk where xk is any arbitrary vector in Rnk foreach k = 1, . . . , t− 1, t + 1, . . . , r; is a linear function of xt. Multilinearfunctions are generalizations of bilinear functions defined in Section1.7.

Thus, from the above result we see that the determinant of a squarematrix A is a multilinear function of A, when the entries in A arepartitioned into either the columns of A or the rows of A.

The branch of mathematics dealing with the properties of multi-linear functions is called multilinear algebra. The multilinearityproperty of the determinant plays an important role in many advancedtheoretical studies of the properties of determinants.

14. Singular, nonsingular square matrices:

2.8 Determinants 235

The square matrix A of order n is said to be a

singular square matrix if det(A) = 0

nonsingular square matrix if det(A) �= 0.

The concepts of singularity, nonsingularity are only defined forsquare matrices, but not for rectangular matrices which are not square.

15. The inverse of a square matrix:

Definition of the inverse of a square matrix: Given a squarematrix A of order n, a square matrix B of order n satisfying

BA = AB = I

where I is the unit matrix of order n, is called the inverse of A. Ifthe inverse of A exists, A is said to be invertible and the inverse isdenoted by the symbol A−1. ��

If A is invertible, its inverse is unique for the following reason.Suppose both B, D are inverses of A. Then

BA = AB = I; DA = AD = I.

So, B = BI = BAD = ID = D.Please note that the concept of the inverse of a matrix is defined only

for square matrices, there is no such concept for rectangular matricesthat are not square,

Also, if a square matrix A is singular (i.e., det(A) = 0), then itsinverse does not exist (i.e., A is not invertible) for the following reason.Suppose A is a singular square matrix and the inverse B of A exists.Then, since BA = the unit matrix I, from Result 9 above we have

det(B) × det(A) = 1

and since det(A) = 0, the above equation is impossible. Hence a sin-gular square matrix is not invertible.

236 Ch. 2. Matrices

Every nonsingular square matrix does have an inverse, in fact wewill now derive a formula for its inverse in terms of its cofactors. Thisinverse is the matrix analogue of the reciprocal of a nonzero real numberin real number arithmetic.

Let A = (aij) be a square matrix of order n, and for i, j = 1 to n letCij be the (i, j)th cofactor of A. Then by Laplace expansion we knowthat for all i = 1 to n

ai1Ci1 + . . . + ainCin = det(A)

and for all j

a1jC1j + . . . + anjCnj = det(A)

Now consider the matrix A′ obtained by replacing the first columnof A by its second column, but leaving everything else unchanged. So,by its columns, we have

A′ = (A.2...A.2

...A.3... . . .

...A.n)

By Result 4 above we have det(A′) = 0. Since A′ differs from Aonly in its first column, we know that Ci1 is also the (i, 1)th cofactorof A′ for all i = 1 to n. Therefore by cofactor expansion down its firstcolumn, we have

det(A′) = a12C11 + a22C21 + . . . + an2Cn1 = 0

Using a similar argumant we conclude that for all t �= j

a1tC1j + a2tC2j + . . . + antCnj = 0

and for all i �= t

ai1Ct1 + ai2Ct2 + . . . + ainCtn = 0

Now consider the square matrix of order n obtained by replacingeach element in A by its cofactor in A, and then taking the transpose

2.8 Determinants 237

of the resulting matrix. Hence the (i, j)th element of this matrix isCji. This transposed matrix of cofactors was introduced by the Frenchmathematician A. L. Cauchy in 1815 under the name adjoint of A.However, the term adjoint has acquired another meaning subsequently,hence this matrix is now called adjugate or classical adjoint of A,and denoted by adj(A). Therefore

adj(A) =

C11 C21 C31 . . . Cn1

C12 C22 C32 . . . Cn2

C13 C23 C33 . . . Cn3...

......

...C1n C2n C3n . . . Cnn

The equations derived above imply that adj(A) satisfies the follow-ing important property

adj(A)A = A(adj(A)) = diag{det(A), det(A), . . . , det(A)}

where diag{det(A), det(A), . . . ,det(A)} is the diagonal matrix of ordern with all its diagonal entries equal to det(A). So, if A is nonsingular,det(A) �= 0, and

(1

det(A))adj(A)A = A[(

1

det(A))adj(A)] = I

i.e.,

A−1 = (1

det(A))adj(A) =

1

det(A)

C11 C21 C31 . . . Cn1

C12 C22 C32 . . . Cn2

C13 C23 C33 . . . Cn3...

......

...C1n C2n C3n . . . Cnn

This provides a mathematical formula for the inverse of an invert-ible matrix. This formula is seldom used to compute the inverse of amatrix in practice, as it is very inefficient. An efficient method for com-puting the inverse based on GJ pivot steps is discussed in Chapter 4.

238 Ch. 2. Matrices

This formula for the inverse is only used in theoretical research studiesinvolving matrices.

Cramer’s Rule for the solution of a square nonsingular sys-tem of linear equations:

Consider the system of linear equations

Ax = b.

It is said to be a square nonsingular system of linear equations if thecoefficient matrix A in it is square nonsingular. In this case A−1 exists,and x = A−1b is the unique solution of the system for each b ∈ Rn. Toshow this, we see that x = A−1b is in fact a solution because

Ax = AA−1b = Ib = b

If x is another solution to the system, then Ax = b, and multiplyingthis on both sides by A−1 we have

A−1Ax = A−1b = x, i.e., x = A−1Ax = Ix = x

and hence x = x. So, x = A−1b is the unique solution of the system.Let x = A−1b = (x1, . . . , xn)T . Using

A−1 = (1

det(A))adj(A) =

1

det(A)

C11 C21 C31 . . . Cn1

C12 C22 C32 . . . Cn2

C13 C23 C33 . . . Cn3...

......

...C1n C2n C3n . . . Cnn

where Cij = the (i, j)th cofactor of A, we get, for j = 1 to n

xj = (1

det(A))(b1C1j + b2C2j + . . . + bnCnj)

For j = 1 to n, let Aj(b) denote the matrix obtained by replacingthe jth column in A by b, but leaving all other columns unchanged.So, by columns

2 .8 D etermina nt s 239

Aj(b) = (A. 1...A. 2

... . . ....A.j − 1

...b...A.j +1

... . . ....A.n)

By cofactor expansion down the column b in Aj(b) we see that

det(Aj(b)) = b1C 1j + b2C 2j + . . . + bnC nj

and hence from the above we have, for j = 1 to n

xj =det(Aj(b))

det(A).

This leads to Cramer’s rule, which states that the unique solutionof the square nonsingular system of linear equations Ax = b isx = (x1, . . . , xn), where

xj =det(Aj(b))

det(A)

for j = 1 to n; with Aj(b) being obtained by replacing the jth columnin A by b, but leaving all other columns unchanged.

Historical note on Cramer’s rule: Cramer’s rule which firstappeared in an appendix of the 1750 book Introduction a L’analysedesLignes Courbes Algebriques by the Swiss mathematician Gabriel Cramer,gives the value of each variable in the solution as the ratio of two deter-minants. Since it gives the value of each variable in the solution by anexplicit determinantal ratio formula, Cramer’s rule is used extensivelyin theoretical research studies involving square nonsingular systems oflinear equations. It is not used to actually compute the solutions ofsystems in practice, as the methods discussed in Chapter 1 obtain thesolution much more efficiently.

Example : Consider the following system of equations in de-tached coefficient form.

x1 x2 x3 b1 −1 1 −50 2 −1 10

−2 1 3 −13

240 Ch. 2 . Matrices

Let A denote the coefficient matrix, and for j = 1 to 3, let Aj(b)denote the matrix obtained by replacing A.j in A by b. Then we have

det(A) = det

1 −1 10 2 −1

−2 1 3

= 9;

det(A1(b)) = det

−5 −1 110 2 −1

−13 1 3

= 18;

det(A2(b)) = det

1 −5 10 10 −1

−2 −13 3

= 27;

det(A3(b)) = det

1 −1 −50 2 10

−2 1 −13

= −36

So, by Cramer’s rule, the unique solution of the system is

x = (18

9,27

9,−36

9)T = (2, 3,−4)T .

Example : Consider the following system of equations in de-tached coefficient form.

x1 x2 x3 b1 −1 0 300 2 2 0

−2 1 −1 −60

Here the determinant of the coefficient matrix is 0, hence it is sin-gular. So, Cramer’s rule cannot be applied to find a solution for it.

Example : Consider the following system of equations in de-tached coefficient form.

2.8 Determinants 241

x1 x2 x3 b1 −1 0 20

10 12 −2 −40

This system has three variables but only two constraints, it is nota square system. Hence Cramer’s rule cannot be applied to find asolution for it.

An Algorithm for Computing the Determinantof a General Square Matrix of Order n Efficiently

BEGIN

Step 1: Perforing G pivot steps on the matrix to makeit triangular: Let A be a given matrix of order n. To compute itsdeterminant, perform G pivot steps in each row of A beginning at thetop. Select pivot elements using any of the strategies discussed in Step2 of the G elimination method in Section 1.20. Make row interchangesas indicated there, and a column interchange to bring the pivot elementto the diagonal position (i.e., if the pivot element is in position (p, q),interchange columns p and q before carrying out the pivot step so thatthe pivot element comes to the diagonal (p, p) position). After thepivot step, enclose the pivot element in a box. At any stage, if thepresent matrix contains a row with all its entries zero, then det(A) =0, terminate.

Step 2: Computing the determinant: If a pivot step is per-formed in every row, in the final matrix the diagonal elements are theboxed pivot elements used in the various pivot steps. For t = 1 to n,let att be the boxed pivot element in the final matrix in row t. Let rdenote the total number of row and column interchanges performed.Then

det(A) = (−1)rn∏

t=1

[att]

END.

242 Ch. 2 . Matrices

Example : Consider the matrix

A =

0 1 3 04 −1 −1 26 2 10 2

−8 0 −2 −4

To compute det (A) using G pivot steps, we will use the partial piv-oting strategy discussed in Section 1.20 for selecting the pivot elements,performing pivot steps in Columns 2, 3, 4, 1 in that order. We indicatethe pivot row (PR), pivot column (PC), and put the pivot element ina box after any row and column interchanges needed are performed.The indication “RI to Rt” on a row means that at that stage that rowwill be interchanged with Row t in that tableau. Similarly “CI of Cp &Cq” indicates that at that stage columns p and q will be interchangedin the present tableau.

Carrying out Step 10 1 3 04 −1 −1 26 2 10 2 RI to R1

−8 0 −2 −4PC↑

6 2 10 2 PR CI of C1 & C24 −1 −1 20 1 3 0

−8 0 −2 −4PC↑

2 6 10 2 PR−1 4 −1 2

1 0 3 00 −8 −2 −4

PC↑

2.8 Determinants 243

Step 1 continued

2 6 10 20 7 4 3 CI of C2 & C30 −3 −2 −10 −8 −2 −4

PC↑2 10 6 2

0 4 7 3 PR0 −2 −3 −10 −2 −8 −4

PC↑2 10 6 2

0 4 7 3

0 0 12

12

0 0 −92

−52

RI to R3

PC↑2 10 6 2

0 4 7 3

0 0 −92

−52

PR CI of C3 & C4

0 0 12

12

PC↑

244 Ch. 2 . Matrices

Step 1 continued

2 10 2 6

0 4 3 7

0 0 − 52

− 92

PR

0 0 12

12

PC↑2 10 2 6

0 4 3 7

0 0 − 52

− 92

0 0 0 − 410

Carrying out Step 2: We made a total of 5 row and column inter-changes. So, we have

det(A) = (−1)5[2 × 4 × −5

2× −4

10] = −8.

Example : Consider the matrix

A =

1 1 3 11 −1 3 34 −2 12 104 −1 13 18

To compute det(A) using G pivot steps, we use the same notationas in the example above. We use the same partial pivoting strategy forselecting the pivot element from the pivot column indicated.

2.8 Determinants 245

Carrying out Step 11 1 3 11 −1 3 34 −2 12 10 RI to R14 −1 13 18

PC↑4 −2 12 10 PR CI of C1 & C21 −1 3 31 1 3 14 −1 13 18

PC↑−2 4 12 10 PR−1 1 3 3

1 1 3 1−1 4 13 18

PC↑−2 4 12 10

0 −1 −3 −20 3 9 6 RI to R20 2 7 13

PC↑−2 4 12 10

0 3 9 6 PR0 −1 −3 −20 2 7 13

PC↑−2 4 12 10

0 3 9 60 0 0 00 0 1 9

Since all the entries in the third row in the last tableau are all zero,det(A) = 0 in this example.

246 Ch. 2 . Matrices

Importance of Determinants

For anyone planning to go into research in mathematical sciences,a thorough knowledge of determinants and their properties is essential,as determinants play a prominent role in theoretical research. How-ever, determinants do not appear often nowadays in actual numericalcomputation. So, the study of determinants is very important for un-derstanding the theory and the results, but not so much for actualcomputational purposes.

Ex ercis es :

2.8.2: Let An×n = (aij) where aij = i + j. What is det(A)?

2.8.3: What is the determinant of the following n × n matrix?

1 2 . . . nn + 1 n + 2 . . . 2n

......

...n2 − n + 1 . . . . . . n2

2.8.4: For any real p, q, r prove that the determinant of the follow-ing matrix is 0.

p2 pq prqp q2 qrrp rq r2

2.8.5: The following square matrix of order n called the Vander-monde matrix is encoutered often in research studies.

V (x1, . . . , xn) =

1 x1 x21 . . . xn−1

1...

......

...1 xn x2

n . . . xn−1n

Prove that the determinant of the Vandermonde matrix, called theVandermonde determinant, is =

∏i>j(xi − xj). (Use the results on

2.8 Determinants 247

determinants stated above to simplify the determinant and then applyLaplace’s expansion across the 1st row).

2.8.6: The following square matrix of order n is called the Frobin-ious matrix or the companion matrix of the polynomial p(λ) =λn − an−1λ

n−1 − an−2λn−2 − . . . − a1λ − a0.

A =

0 1 0 . . . 0 00 0 1 . . . 0 0...

.... . .

. . .. . .

...

0 0 0. . . 1 0

0 0 0 . . . 0 1a0 a1 a2 . . . an−2 an−1

Prove that det(λI − A) = p(λ), where I is the identity matrix oforder n.

2.8.7: A square matrix A = (aij) of order n is said to be a skew-symmetric matrix if aij = −aji for all i, j (i.e., aij + aji = 0 for alli, j).

Prove that the determinant of a skew-symmetric matrix of odd orderis 0.

Prove that the determinant of a skew-symmetric matrix of evenorder does not change if a constant α is added to all its elements.

In a skew-symmetric matrix A = (aij) of even order, if aij = 1 forall j > i, find det(A).

2.8.8: Let A = (aij), B = (bij) be two square matrices of the sameorder n, where bij = (−1)i+jaij for all i, j. Prove det(A) = det(B).

2.8.9: Let A = (aij) be a square matrix of order n, and si =∑nj=1 aij for all i. So, si is the sum of all the elements in row i of

A. Prove that the determinant of the following matrix is (−1)n−1(n −1)det(A).

248 Ch. 2. Matrices

s1 − a11 . . . s1 − a1n...

...sn − an1 . . . sn − ann

.

2.8.10: Basic minors of a matrix: There are many instanceswhere we have to consider the determinants of square submatrices of amatrix, such determinants are called minors of the matrix. A minorof order p of a matrix A is called a basic minor of A if it is nonzero,and all minors of A of orders ≥ p + 1 are zero; i.e., a basic minor is anonzero minor of maximal order, and its order is called the rank of A.

If the minor of A defined by rows i1, . . . , ik and columns j1, . . . , jk

is a basic minor, then show that the set of row vectors {Ai1., . . . , Aik.}is linearly independent, and all other rows of A are linear combinationsof rows in this set.

2.8.11: The trace of a square matrix A = (aij) of order n is∑ni=1 aii, the sum of its diagonal elements. If A, B are square matrices

of order n, prove that trace(AB) = trace(BA).

2.8.12: If the sum of all the elements of an invertible square matrixA in every row is s, prove that the sum of all the elements in every rowof A−1 is 1/s.

2.9 Additional Exercises

2.9.1: A, B are two matrices of order m×n. Mention conditions underwhich (A + B)2 is defined. Under these conditions, expand (A + B)2

and write it as a sum of individual terms, remembering that matrixmultiplication is not commutative.

2. Consider the following matrices. Show that FE and EF bothexist but they are not equal.

2.9. Additional Exercises 249

F =

1 0 00 1 00 3 1

, E =

1 0 0−2 1 0

0 0 1

.

3. Is it possible to have a square matrix A �= 0 satisfying theproperty that A × A = A2 = 0? If so find such matrices of orders 2and 3.

4. Show that the product of two lower triangular matrices is lowertriangular, and that the product of upper triangular matrices is uppertriangular.

Show that the inverse of a lower triangular matrix is lower trian-gular, and that the inverse of an upper triangular matrix is uppertriangular.

Show that the sum of symmetric matrices is symmetric. Show thatany linear combination of symmetric matrices of the same order is alsosymmetric.

Construct numerical examples of symmetric matrices A, B of thesame order such that the product AB is not symmetric. Constructnumerical examples of symmetric matrices whose product is symmetric.

If A, B are symmetric matrices of the same order, prove that theproduct AB is symmetric iff AB = BA.

Show that the inverse of a symmetric matrix is also symmetric.For any matrix A show that both the products AAT , AT A always

exist and are symmetric.Prove that the product of two diagonal matrices is also diagonal.

5. Let C = AB where A, B are of orders m× n, n× p respectively.Show that each column vector of C is a linear combination of the col-umn vectors of A, and that each row vector of C is a linear combinationof row vectors of B.

6. If both the products AB, BA of two matrices A, B are definedthen show that each of these products is a square matrix. Also, if thesum of these products is also defined show that A, B must themselvesbe square matrices of the same order.

250 Ch. 2. Matrices

Construct numerical examples of matrices A, B where both prod-ucts AB, BA are defined but are unequal square matrices of the sameorder.

Construct numerical examples of matrices A, B both nonzero suchthat AB = 0.

Construct numerical examples of matrices A, B, C where B �= Cand yet AB = AC.

7. Find the matrix X which satisfies the following matrix equation.

1 1 −1−1 1 1

1 −1 1

X =

11 −5 −7−5 −7 11−7 11 −5

.

8. Consider the following square symmetric matrix A = (aij) oforder n in which a11 �= 0. Perform a G pivot step on A with a11 asthe pivot element leading to A. Show that the matrix A of order n− 1obtained from A by striking off its first row and first column is alsosymmetric.

A =

a11 a12 . . . a1n

a21 a22 . . . a2n...

... . . ....

an1 an2 . . . ann

, A =

a11 a12 . . . a1n

0 a22 . . . a2n...

... . . ....

0 an2 . . . ann

.

9. A square matrix A is said to be skew symmetric if AT = −A.Prove that if A is skew symmetric and A−1 exists then A−1 is also skewsymmetric.

If A, B are skew symmetric matrices of the same order show that A+B, A − B are also skew symmetric. Show that any linear combinationof skew matrices of the same order is also skew symmetric.

Can the product of skew symmetric matrices be skew symmetric?

Using the fact that A = (1/2)((A+AT )+(A−AT )) show that everysquare matrix is the sum of a symmetric matrix and a skew symmetricmatrix.

If A is a skew symmetric matrix, show that I+A must be invertible.

2.9. Additional Exercises 251

10. List all the 24 permutations of {1, 2, 3, 4} and classify theminto even, odd classes.

Find the number of inversions in the permutation (7, 4, 2, 1, 6, 3, 5).

11. Show that the determinant of the elementary matrix corre-sponding to the operation of multiplying a row of a matrix by α isα.

Show that the determinant of the elementary matrix correspondingto the operation of interchanging two rows in a matrix is −1.

Show that the determinant of the elementary matrix correspondingto the operation of adding a scalar multiple of a row of a matrix toanother is 1.

Show that the determinant of the pivot matrix corresponding to aGJ pivot step with ars as the pivot element is 1/ars.

12. Construct a numerical example to show that det(A+B) is notnecessarily equal to det(A) + det(B).

By performing a few row operations show that the following deter-minants are 0

∣∣∣∣∣∣∣

5 4 31 2 31 1 1

∣∣∣∣∣∣∣,

∣∣∣∣∣∣∣∣∣

2 3 0 03 2 2 0

−5 −5 0 20 0 1 1

∣∣∣∣∣∣∣∣∣.

If A is an integer square matrix and det(A) = ±1, then show thatA−1 is also an integer matrix.

Show that the following determinant is 0 irrespective of what valuesthe aijs have.

∣∣∣∣∣∣∣∣∣∣∣∣∣∣

a11 a12 a13 0 0 0a21 a22 a23 0 0 0a31 a32 a33 0 0 0a41 a42 a43 0 0 0a51 a52 a53 0 a55 a56

a61 a62 a63 a64 a65 a66

∣∣∣∣∣∣∣∣∣∣∣∣∣∣

.

252 Ch. 2. Matrices

13. In a small American town there are three newspapers thatpeople buy, NY, LP, CP. These people make highly predictable transi-tions from one newspaper to another from one quarter to the next asdescribed below.

Among subscribers to NY: 80% continue with NY, 7% switch toCP, and 13% switch to LP.

Among subscribers to LP: 78% continue with LP, 12% switch toNY, 10% switch to CP.

Among subscribers to CP: 83% continue with CP, 8% switch to NY,and 9% switch to LP.

Let x(n) = (x1(n), x2(n), x3(n))T denote the vector of the numberof subscribers to NY, LP, CP in the nth quarter. Treating these ascontinuous variables, find the matrix A satisfying: x(n + 1) = Ax(n).

14. There are two boxes each containing different numbers of bagsof three different colors as represented in the following matrix A (rowsof A corespond to boxes, columns of A correspond to colors of bags)given in tableau form.

White Red BlackBox 1 3 4 2Box 2 2 2 5

Each bag contains different numbers of three different fruit as inthe following matrix B given in tabular form.

Mangoes Sapotas BananasWhite 3 6 1Red 2 4 4

Black 4 3 3

Show that rows, columns of the matrix product C = AB correspondto boxes, fruit respectively, and that the entries in C are the totalnumber of various fruit contained in each box.

15. Express the row vector (a11b11 + a12b21, a11b21 + a12b22) as a

2.9. Additional Exercises 253

product of two matrices.

16. A, B are two square matrices of order n. Is (AB)2 = A2B2?

17. Plant 1, 2, each produce both fertilizers Hi-ph, Lo-ph simulta-neously each day with daily production rates (in tons) as in the follow-ing matrix A represented in tableau form.

Plant 1 Plant 2Hi-ph 100 200Lo-ph 200 300

In a certain week, each plant worked for different numbers of days,and had total output in tons of (Hi-ph, Lo-ph)T = b = (1100, 1800)T .

What is the physical interpretation of the vector x which is thesolution of the system Ax = b?

18. The % of N, P, K in four different fertilizers is given below.

N P KFertilizer 1 10 10 10

2 25 10 53 30 5 104 10 20 20

Two different mixtures are prepared by combining various quanti-ties in the following way.

Lbs in mixture, ofFertilizer 1 2 3 4

Mixture 1 100 200 300 4002 300 500 100 100

Derive a matrix product that expresses the % of N, P, K in the twomixtures.

254 Ch. 2. Matrices

19. Bacterial growth model: A species of bacteria has thisproperty: after the day an individual is born, it requires a maturationperiod of two complete days, then on 3rd days it divides into 4 newindividuals (after division the original individual no longer exists).

Let xk = (xk1, x

k2, x

k3)

T denote number of individuals for whom dayk is 1st, 2nd, 3rd complete day after the day of their birth.

Find a square matrix A of order 3 which expresses xk+1 as Axk. Forany k express xk in terms of A and x1.

20. Bactterial growth model with births and deaths: Con-sider bacterial population described in Exercise 19 above with two ex-ceptions.

1. Let xki be continuous variables here.

2. In 1st full day after their birth day each bacteria either dies withprobability 0.25 or lives to 2nd day with probability 0.75.

Each bacteria that survives 1st full day of existence after birth,will die on 2nd full day with probability 0.2, or lives to 3rd day withprobability 0.8.

The scenario on 3rd day is the same as described in Excercise 19,i.e., no death on 3rd day but only division into new individuals.

Again find matrix A that expresses xk+1 as Axk here, and give anexpression for xk in terms of A and x1.

21: A 4-year community college has 1st, 2nd, 3rd, 4th year stu-dents. From past data they found that every year, among the studentsin each year’s class, 80% move on to the next year’s class (or graduatefor 4th year students); 10% drop out of school; 5% are forced to repeatthat year; and the remaining 5% are asked to take the next year offfrom school to work and get strong motivation to continue studies, andreturn to college into the same year’s class after that work period. Letx(n) = (x1(n), x2(n), x3(n), x4(n))T denote the vector of the numberof 1st year, 2nd year, 3rd year, 4th year students in the college in thenth year, treat each of these as continuous variables. Assume that the

2.9. Additional Exercises 255

college admits 100 first year students, and 20 transfer students fromother colleges into the 2nd year class each year. Find matrices A, b thatexpresses x(n + 1) as Ax(n) + b.

22: Olive harvest: Harvested olives are classified into sizes 1 to 5.Harvest season usually lasts 15 days, with harvesting carried out onlyon days 1, 8, and 15 of the season. Of olives of sizes 1 to 4 on trees notharvested on days 1, 8 of the season, 10% grow to the next size, and5% get rotten and drop off, in a week.

At the beginning of the harvest season an olive farm estimated thatthey have 400, 400, 300, 300, 40 kg of sizes 1 to 5 respectively on theirtrees at that time. On days 1, 8 of the harvest season they plan toharvest all the size 5 olives on the trees, and 100, 100, 50, 40 kg ofsizes 1 to 4 respectively; and on the final 15th day of the season, allthe remaining olives will be harvested.

Let xr = (xr1, x

r2, x

r3, x

r4, x

r5)

T denote the vector of kgs of olives ofsizes 1 to 5 on the trees, before harvest commences, on the rth harvestday, r = 1, 2, 3.

Find matrices A1, A2, b1, b2 such that x2 = A1(x1−b1), x3 = A2(x2−b2). Also determine the total quantity of olives of various sizes that thefarm will harvest that year.

23. A population model: (From A. R. Majid, Applied MatrixModels, Wiley, 1985) (i): An animal herd consists of three cohorts:newborn (age 0 to 1 year), juveniles (age 1 to 2 years), and adults (age2 or more years.

Only adults reproduce to produce 0.4 newborn per adult per year(a herd with 50 male and 50 female adults will produce 40 newborncalves annually, half of each sex). 65% of newborn survive to becomejuveniles, 78% of juveniles survive to become adults, and 92% of adultssurvive to live another year.

Let xk = (xk1, x

k2, x

k3)

T denote the vector of the number of newborn,juvenile, and adults in the kth year. Find a matrix k that expressesxk+1 as Axk for all k.

A system like this is called a discrete time linear system with A asthe transition matrix from k to k + 1. Eigenvalues and eigenvectors

256 Ch. 2. Matrices

of square matrices discussed in Chapter 6 are very useful to determinethe solutions of such systems, and to study their limiting behavior.

(ii): Suppose human predators enter and colonize this animal habi-tat. Let xk

4 denote the new variable representing the number of humansin the colony in year k. Human population grows at the rate of 2.8%per year, and each human eats 2.7 adult animals in the herd per year.Let Xk = (xk

1, xk2, x

k3, x

k4)

T . Find the matrix B such that Xk+1 = BXk.

24. Fibonacci numbers: Consider the following system of equa-tions: x1 = 1, x2 = 1, and xn+1 = xn + xn−1 for n ≥ 2.

Write the 3rd equation for n = 2, 3, 4; and express this system of5 equations in (x1, . . . , x5)

T in matrix form.

For general n, find a matrix B such that (xn+1, xn+2)T = B(xn, xn+1)

T .Then show that (xn+1, xn+2)

T = Bn(x1, x2)T .

Also for general n find a matrix A such that (xn+1, xn+2, xn+3)T =

A(xn, xn+1, xn+2)T Hence show that (xn+1, xn+2, xn+3)

T = An(x1, x2, x3)T .

xn is known as the nth Fibonacci number, all these are positiveintegers. Find xn for n ≤ 12, and verify that xn grows very rapidly.The growth of xn is similar to the growth of populations that seem tohave no limiting factors.

25. For any u, v between 1 to n define Euv to be the square matrixof order n with the (u, v)th entry = 1, and every other entry = 0.

Let (j1, . . . , jn) be a permutation of (1, . . . , n), and let P = (pr,s)be the permutation matrix of order n corresponding to it (i.e., prjr =1 for all r, and all other entries in P are 0).

Then show that PEuvP−1 = Ejujv . (Remember that for a permu-

tation matrix P , P−1 = P T .)

26. Find conditions on b1, b2, b3 so that the following system hasat least one solution. Does b1 = 5, b2 = −4, b3 = −2 satisfy thiscondition? If it does find the general solution of the system in this casein parametric form.

2.9. Additional Exercises 257

x1 x2 x3 x4

1 −1 −1 1 b1

2 −3 0 1 b2

8 −11 −2 5 b3

27. Fill in the missing entries in the following matrix product.

1 0 12 −1

1 3

101 0 2

=

0 −1 31 0 02 −2 9

28. If A, B are square matrices of order n, is it true that (A −B)(A + B) = A2 − B2? If not under what conditions will it be true?

Expand (A + B)3.

29. Determine the range of values of a, b, c for which the followingmatrix is triangular.

5 6 b 00 a 2 −1c 0 −1 20 0 0 3

30. In each case below, you are given a matrix A and its productAx with an unknown vector x. In each case determine whether youcan identify the vector x unambiguously.

(i.) A =

(1 −11 2

), Ax =

( −111

)

(ii.) A =

1 1 −1−1 1 1−1 5 1

, Ax =

42

14

(iii.) A =

1 1 −1−1 1 1

0 3 1

, Ax =

000

258 Ch. 2. Matrices

(iv.) A =

1 1 2−1 0 1

1 3 4

, Ax =

000

31. Inverses of partitioned matrices: Given two square matri-ces of order n, to show that B is the inverse of A, all you need to do isshow that AB = BA = the unit matris of order n.

Consider the partitioned square matrix M of order n, where A, Dare square submatrices of orders r, n− r respectively. In the followingquestions, you are asked to show that M−1 is of the following form,with the entries as specified in the question.

M =

(A BC D

)

M−1 =

(E FG H

)

(i.) If M, A are nonsingular, then E = A−1 − FCA−1, F =−A−1BH , G = −HCA−1, and H = (D − CA−1B)−1.

(ii.) If M, D are nonsingular, then E = (A − BD−1C)−1, F =−EBD−1, G = −D−1CE, H = D−1 − GBD−1.

(iii.) If B = 0 , r = n − 1, and D = α where α = ±1, and A isinvertible, then E = A−1, F = 0, G = −αCA−1, H = α.

32. Find the determinant of the following matrices.

0 1 0 . . . 00 0 1 . . . 0...

......

. . ....

0 0 0 . . . 1p1 p2 p3 . . . pn

,

a + b c 1b + c a 1c + a b 1

2.9. Additional Exercises 259

1 + α1 α2 . . . αn

α1 1 + α2 . . . αn...

.... . .

...α1 α2 . . . 1 + αn

33. Consider following matrices:

An = (aij) of order n with all diagonal entries aii = n, and alloff-diagonal entries aij = 1 for all i �= j.

Bn = (bij of order n with all diagonal entries bii = 2; bi+1,i = bi,i+1 =−1 for all i= 1 to n − 1; and all other entries 0.

Cn = (cij) of order n with ci,i+1 = 1 for all i = 1 to n − 1; and allother entries 0.

Show that Cn + CTn is nonsingular iff n is even.

Develop a formula for determinant(An) in terms of n.

Show that determinant(Bn) = 2 determinant(Bn−1) - determinant(Bn−2).

34. If A, B, C are all square matrices of the same order, C is non-singular, and B = C−1AC, show that Bk = C−1AkC for all positiveintegers k.

If A is a square matrix and AAT is singular, then show that A mustbe singular too.

If u = (u1, . . . , un)T is a column vector satisfying uT u = 1, showthat the matrix (I − 2uuT ) is symmetric and that its square is I.

If A, B are square matrices of the same order, and A is nonsingular,show that determinant(ABA−1) = determinant(B).

35. Consider the following system of equations. Without comput-ing the whole solution for this system, find the value of the variable x2

only in that solution.

x1 + 2x2 + x3 = 13

x1 + x2 + 2x2 = −9

2x1 + x2 + x3 = −12.

260 Ch. 2. Matrices

2.10 References

[2.1]. M. Golubitsky and M. Dellnitz, Linear Algebra and DifferentialEquations Using MATLAB, Brooks/Cole, Pacific Grove, CA, 1999.

Index

For each index entry we providethe section number where it isdefined or discussed first. Click-ing on the light blue section num-ber takes you to the page wherethis entry is defined or discussed.

Adjugate 2.8Assignment 2.8Associative law 2.5Asymmetric matrix 2.7Augmented matrix 2.1

Basic minors 2.8

Cauchy-Binet theorem 2.8Classical adjoint 2.8Coefficient matrix 2.1Cofactor expansion 2.8Commutative 2.5Companion matrix 2.8Cramer’s rule 2.8

Determinant 2.8Diag{a11, . . . , ann } 2.6Diagonal entries 2.6Diagonal matrix 2.6

Frobinious matrix 2.8

Identity matrix 2.6

Inverse 2.8

Inversions 2.8

Laplace expansion 2.8

Left & right distributive laws 2.5

Lower triangular matrix 2.6

Main diagonal 2.6

Matrix 2.1

Algebra 2.1

Equation 2.3

History 2.1

Of coefficients 2.1

Order 2.1

Size 2.1

Multilinear algebra 2.8

Multilinear function 2.8

Nonsingular matrix 2.8

Off-diagonal entries 2.6

Outer product 2.5

Partitioned matrix 2.1

Permutation 2.8

Permutation matrix 2.6

261

262 Ch. 2. Matrices

Premultiplication, postmultipli-cation 2.5

Principal submatrix 2.7

Rectangular matrices 2.6Row by column multiplication

2.5

Singular matrix 2.8Square matrices 2.6String computation 2.5

Right to left 2.5Left to right 2.5

Submatrix 2.7Symmetric matrix 2.7Symmetrization 2.7

Triangular matrix 2.6Trace 2.8

Unit matrix 2.6Upper triangular 2.6

Vandermonde matrix 2.8

Index for Ch. 2 263

List of algorithms

1. Algorithm for computing the determinant of a square matrixusing G pivot steps.

Historical note on:

1. The word “matrix”

2. Determinants

3. Cramer’s rule.

Definition of:

1. Matrix−vector product

2. Cofactor expansion of a determinant

3. Multilinear function

4. Inverse of a square matrix.

Cautionary note on:

1. Care in writing matrix products

2. The linearity of a determinant.

264 Ch. 2. Matrices

List of Results

1. Result 2.5.1: Product involving three matrices2. Result 2.8.1: One set of properties defining a determinant3. Results on determinants.

Contents

3 n-Dimensional Geometry 2653.1 Viewing n-tuple Vectors As Points of n-Dimensional Space

R n By Setting Up Cartesian Coordinate Axes . . . . . 265

3.2 Translating the Origin to the Point a, Translation of aSet of Points . . . . . . . . . . . . . . . . . . . . . . . . 272

3.3 Parametric Representation of Subsets of R n . . . . . . 276

3.4 Straight Lines in R n . . . . . . . . . . . . . . . . . . . 277

3.5 Half-Lines and Rays in R n . . . . . . . . . . . . . . . . 283

3.6 Line Segments in R n . . . . . . . . . . . . . . . . . . . 286

3.7 Straight Lines Through the Origin . . . . . . . . . . . . 2903.8 Geometric Interpretation of Addition of Two Vectors,

Parallelogram Law . . . . . . . . . . . . . . . . . . . . 2923.9 Linear Combinations, Linear Hulls or Spans, Subspaces 2933.10 Column Space, Row Space, and Null Space of a Matrix 3073.11 Affine Combinations, Affine Hulls or Spans, Affinespaces 3083.12 Convex Combinations, Convex Hulls . . . . . . . . . . 3243.13 Nonnegative Combinations, Pos Cones . . . . . . . . . 3323.14 Hyperplanes . . . . . . . . . . . . . . . . . . . . . . . 3353.15 Halfspaces . . . . . . . . . . . . . . . . . . . . . . . . 3383.16 Convex Polyhedra . . . . . . . . . . . . . . . . . . . . 3423.17 Convex and Nonconvex Sets . . . . . . . . . . . . . . . 3463.18 Euclidean Distance, and Angles . . . . . . . . . . . . . 3473.19 The Determinant Interpreted As A Volume Function . 3533.20 Orthogonal Projections (Nearest Points) . . . . . . . . 3583.21 Procedures for Some Geometric Problems . . . . . . . . 3653.22 Additional Exercises . . . . . . . . . . . . . . . . . . . 373

i

ii

3.23 References . . . . . . . . . . . . . . . . . . . . . . . . 374

Chapter 3

n-Dimensional Geometry

This is Chapter 3 of “Sophomore Level Self-Teaching Web-book for Computational and Algorithmic Linear Algebra andn-Dimensional Geometry” by Katta G. Murty.

3.1 Viewing n-tuple Vectors As Points of

n-Dimensional Space Rn By Setting

Up Cartesian Coordinate Axes

In 1637 the French philosopher and mathematician Rene’ Descartesoutlined a very simple idea for viewing n-tuple vectors as points inn-dimensional space using a frame of reference which is now calledthe Cartesian coordinate system. It is a very revolutionary ideathat helped unify the two main branches of mathematics, algebra andgeometry. We will now explain this idea for n = 1, 2, 3, and highervalues.

The Case n = 1

The case n = 1 is very direct. Each vector of dimension 1 is a realnumber, x, say. Draw a straight line (the one dimensional real line)and select any point on it as the origin which will be represented by(or corresponds to) 0. Select a unit of length for measurement, say

265

266 Ch. 3. Geometry of Rn

1 inch = 1 unit. The origin divides the real line into two half lines(on the right and left hand sides of the origin), select one of these halflines to represent positive numbers, and the other to represent negativenumbers. By the universally accepted convention, if x > 0, then it isrepresented by the point exactly x inches to the right of the origin onthe real line; and if x < 0, then it is represented by the point exactly|x| inches to the left of the origin. This way, there is a one-to-onecorrespondence between points on the real line and real numbers, i.e.,one dimensional vectors. See Figure 3.1.

-3 -2 -1 0 1 2 3. . .. ...

Figure 3.1: One dimensional vectors represented as points on the realline.

The Case n = 2

We will now explain the concept for n = 2. In this case also, it ispossible to draw the axes of coordinates on a two dimensional sheetof paper and verify the one-to-one correspondence between the vectorsand points in the space.

Select a point on the two dimensional sheet of paper as the origincorresponding to the 0-vector, (0, 0). Draw two perpendicular straightlines through the origin, one horizontal stretching from left to right, andthe other vertical stretching from top to bottom. These two perpendic-ular straight lines constitute the Cartesian coordinate axes system(also called rectangular coordinate system) for the 2-dimensionalspace represented by the sheet of paper; the lines are called axes orcoordinate axes, the horizontal and the vertical axes. Now assign

3.1. Cartesian Coordinate Axes 267

(-2,0) (3,0)

(1,2)

-1

-2

-3

-1-2-3

1

2

3

1 2 3

(2,-1)

(0,-3)

Q(x , x )1 2

1x -axis

2x -axis

x1

x2

(0,2)

0

Figure 3.2: The Cartesian (rectangular) coordinate system for the2-dimensional plane. The angles are all 900 (π/2 in radians). 2-dimensional vectors x = (x1, x2) are represented as points in the 2-dimensional plane.

each component of the vector (x1, x2) to an axis. Suppose we assign x1

to the horizontal axis, and x2 to the vertical axis. Then the horizontalaxis is called the x1-axis, and the vertical axis called the x2-axis inthe coordinate system.

Select a unit of length for measurement, say 1 inch = 1 unit, on eachof the axes (the units for the two axes may be different). By convention,

268 Ch. 3. Geometry of Rn

on the horizontal axis, points to the right of the origin have x1 > 0;and those to the left have x1 < 0. Similarly, on the vertical axis, pointsabove the origin have x2 > 0, and those below the origin correspond tox2 < 0. On each axis mark out the points which are an integer numberof inches from the origin and write down the number corresponding tothem as in Figure 3.2.

To get the unique point corresponding to the vector x = (x1, x2),do the following:

if x1 = 0 from the origin, go to the point corresponding tox2 on the x2-axis itself (i.e., if x2 > 0 it is x2 inchesabove the origin, if x2 < 0 it is |x2| inches belowthe origin), this is the point corresponding to thevector x

if x1 �= 0, go to the point corresponding to x1 on the x1-axis, P say. From P move x2 inches above P ifx2 > 0, or |x2| inches below P if x2 < 0, alongthe straight line parallel to the x2-axis throughP . The resulting point is the one correspondingto the vector x.

Now to find the 2-dimensional vector corresponding to a point Q inthe 2-dimensional plane under this frame of reference, do the following:drop perpendiculars from Q to each of the coordinate axes, and measurethe length of each perpendicular in the unit choosen for the axis towhich it is parallel (inches here). Then, this point Q corresponds tothe vector

x = (x1 = length of the perpendicular parallel to the x1-axis,x2 = length of the perpendicular parallel to the x2-axis)

this vector x is called the vector of Cartesian coordinates or thecoordinate vector of the point Q.

Clearly each 2-dimensional vector corresponds to a unique point inthe 2-dimensional space and vice versa, once the rectangular coordinatesystem is choosen.

3.1. Cartesian Coordinate Axes 269

The Case n = 3

For n = 3, to represent 3-dimensional vectors x = (x1, x2, x3) aspoints in 3-dimensional space, the procedure is very similar. We selecta point in 3-dimensional space as the origin representing the vector 0 =(0, 0, 0), and draw three mutually perpendicular axes, the x1-axis, x2-axis, x3-axis through it, and select a scale on each axis (1 inch = 1 unit,say). This gives the Cartesian coordinate system for 3-dimensionalspace. In this, the origin divides each axis into two halfs. On eachaxis, designate one of the halfs to represent nonnegative values of thecorrresponding component, and the other half to represent nonpositivevalues of that component.

To find the point corresponding to the vector x = (x1, x2, x3),first find the point P corresponding to the subvector (x1, x2) in the2-dimensional plane determined by the x1 and x2 axes (called the x1

x2-coordinate plane) just as in the 2-dimensional case discussed ear-lier. If x3 = 0, P is the point corresponding to the vector x. If x3 �= 0,at P draw the line parallel to the x3-axis and move a distance of |x3|in x3-axis units along this line in the positive or negative directionaccording to whether x3 is positive or negative, leading to the pointcorresponding to the vector x. See Figure 3.3.

Now to find the vector (x1, x2, x3) corresponding to a point Q inspace: if Q is already a point in the x1 x2-coordinate plane, x3 = 0,and find the vector (x1, x2) by applying the 2-dimensional procedurediscussed earlier in the x1 x2-coordinate plane. If Q is not in thex1 x2-coordinate plane, drop the perpendicular from Q to the x1 x2-coordinate plane, and measure its length in x3-axis units. x3 is thislength multiplied by ±1 depending on whether this perpendicular ison the positive or negative side of the x3-axis in relation to the x1 x2-coordinate plane. If the foot of this perpendicular is P , then find thesubvector (x1, x2) as the coordinate vector of P in the x1 x2-coordinateplane as in the 2-dimensional case discussed earlier. See Figure 3.4.

Here again there is a one-to-one correspondence between 3-dimensionalvectors and points in 3-dimensional space. The vector coorespondingto a point in 3-space is called its coordinate vector under the coordinateaxes system set up.

270 Ch. 3. Geometry of Rn

The Case n > 3

For n > 3, the procedure can be visualized to be the same, eventhough we cannot carry it out physically, since we only live in a 3-dimensional world. We select the coordinate axes to be n mutuallyperpendicular lines through a point in space selected as the origin rep-resenting the 0-vector, and select the positive and negative sides of theorigin on each axis, and a scale for measuring lengths on it.

x -axis1

2x -axis

3x -axis

(x , 0, 0)1

(x , x , x )1 2 3

P = (x , x , 0)1 2

.

.

.

0

Figure 3.3: The Cartesian (rectangular) coordinate system for 3-dimensional space. All the angles are 900 (π/2 measured in radians).3-dimensional vectors (x1, x2, x3) are represented as points in the 3-dimensional space.

For each j = 1 to n, the x1x2 . . . xj−1xj+1 . . . xn-coordinate planedetermined by the corresponding axes partitions the space into two

3.1. Cartesian Coordinate Axes 271

half-spaces, one containing the positive half of the xj-axis, and theother containing the negative half.

To get the point in space corresponding to the vector x = (x1, . . . ,xn−1, xn), first obtain the point P in the x1x2 . . . xn−1-coordinate planecorresponding to the subvector (x1, . . . , xn−1); if xn = 0, this P is thepoint corresponding to x. If xn �= 0, to get the point corresponding tox, move a distance of |xn| in xn-axis units in the positive or negativedirection depending on the sign of xn, along the line parallel to thexn-axis through P .

To find the coordinate vector (x1, . . . , xn) corresponding to a pointQ in space, if Q is in the x1x2 . . . xn−1-coordinate plane, xn = 0 and thesubvector (x1, . . . , xn−1) is found exactly as in the (n − 1) dimensionalcase in the x1 . . . xn−1-coordinate plane. If Q is not in the x1 . . . xn−1-

x1

x2

x3

1___

Q = (x , x , x )2 3

x_1

__P = (x , x , 0)1 2

_x2

0

.

.

.

Figure 3.4: P is the foot of the perpendicular from Q to the x1 x2-coordinate plane, and x3 is the length of QP multiplied by ±1. Allangles are 900 (π/2 in radians).

272 Ch. 3. Geometry of R n

coordinate plane, drop the perpendicular from Q to the x1 . . . xn− 1-coordinate plane, and take xn to be its length in the xn-axis units, mul-tiplied by ±1 depending on whether Q is on the positive or the negativeside of this coordinate plane. The remaining subvector (x1, . . . , xn − 1)is the coordinate vector of the foot of this perpendicular P in thex1 . . . xn − 1-coordinate plane found as in the (n − 1)-dimensional case.

With this, the one-to-one correspondence between n-dimensionalvectors and points in n-dimensional space holds.

Setting up the coordinate axes and representing each point in spaceby its coordinate vector makes it possible to use algebraic notation andprocedures for the description and study of geometric objects. Thisarea of mathematics is called analytical geometry.

So, in the sequel, we will refer to vectors x ∈ R n as also points in

R n; and the various components in the vector x = (x1, . . . , xn) as its

coordinates.In the geometric representation of an n-dimensional vector by a

point in n-dimensional space, there is no distinction made between rowvectors and column vectors. We will assume that the vectors corre-sponding to points in n-dimensional space are either all row vectors, orall column vectors, which will be clear from the context.

Ex ercis es

3.1.1: Set up the 2-dimensional Cartesian coordinate system andplot the points: (1, 2), (−1,−2), (1,−2), (−1, 2), (0, 3), (−3, 0), (0,2), on it.

3.2 Translating the Origin to the Point a,

Translation of a Set of Points

Suppose we have set up the Cartesian coordinate system for Rn witha point O selected as the origin represented by the 0-vector. Letx = (x1, . . . , xn)T , a = (a1, . . . , an)T be the coordinate vectors cor-

3.2. Translation 273

responding to points P, Q respectively in space WRT this coordinatesystem.

Now if we change the origin from O to Q, moving each coordinateaxis from the present position to the new position parallel to itself,we get a different Cartesian coordinate system which is parallel to theold. This operation is called translating the origin to the point Qrepresented by the coordinate vector a.

The coordinate vector cooresponding to the point P WRT the newcoordinate system will be x − a.

Thus if we translate the origin to a, it has the effect of adding thevector −a to the coordinate vector of every point.

x -axis1

2x -axis

. .0

..

.

. .

.1

2

3

1 2 3

1

21

2

P = (x , x ) becomes

(x -1, x -2 ) after translation.

(1, 2)

Figure 3.5: Translating the origin from 0 to (1, 2). The new coordinatesystem is shown in dotted lines. This changes (x1, x2) to (x1−1, x2−2).

Let S ⊂ Rn be a set of points, and x ∈ Rn a point. Then thetranslate of the set S to the point x is the set of points {y+x : y ∈ S};

274 Ch. 3. Geometry of Rn

which is denoted by the symbol x+S sometimes. It is the set obtainedby adding x to every point in S. The process of obtaining x + S fromS is called translation of S to x.

Example 1: Let

S =

{(12

),

( −11

),

( −12

)}⊂ R2; x =

( −1−1

).

Then x + S = translate of S to x is:{(

12

)+

( −1−1

),

( −11

)+

( −1−1

),

( −12

)+

( −1−1

)}

=

{(01

),

( −20

),

( −21

)}.

See Figure 3.6

x -axis1

2x -axis

.

..

0...

.

.

1 2

1

2

-1

-2

-1-2

**

*

(-1, -1)x_

=

+

+ +

.

.3

Figure 3.6: The set S consists of all the points marked with “*”. x isthe point marked with “.”. x + S is the set of all points marked with“+”.

3.2. Translation 275

Exercises

3.2.1 We are given the following sets of vectors. Write the set ofcoordinate vectors of points in these sets when the origin is translatedto (1, 2,−3)T .

S1 =

398

,

398

,

−200

,

08

−18

S2 =

000

,

12

−3

,

−1−2

3

,

24

−6

,

111

.

3.2.2: Consider the following sets: S = {x = (x1, x2)T : 0 ≤ x1 ≤

1, 0 ≤ x2 ≤ 1}, T = {x + (−1, 2)T : x ∈ S}. To which point should theorigin be translated to make the set S into the set T ?

3.2.3: In the following pairs S, T check whether T is a translate ofS.

S =

112

,

24

−7

,

038

,

222

; T =

13

−9

,

110

000

,

−126

S =

{(2

−3

),

(46

),

( −7−8

),

(84

)}; T =

{(0

−5

),

(24

)

( −9−10

),

(63

)}

276 Ch. 3. Geometry of Rn

3.3 Parametric Representation of Subsets

of Rn

For any n ≥ 2, one way of representing a subset of points in Rn is theparametric representation.

In parametric representation of any subset S of points in Rn, thecoordinates of a general point in the subset S are expressed as functionsof one or more parameters in the form

x =

x1

x2...xj...

xn

=

f1(α1, . . . , αr)f2(α1, . . . , αr)

...fj(α1, . . . , αr)

...fn(α1, . . . , αr)

(3.1)

In this representation, the jth coordinate of a general point in S isexpressed as the function fj(α1, . . . , αr) of r parameters α1, . . . , αr, forj = 1 to n; where each parameter can take a real value subject to someconstraints which will be specified.

If there are no constraints on the parameters α1, . . . , αr, by givingall possible real values to each of them in (3.1), we should get all thepoints in the set S. Thus in this case

S = {(f1(α1, . . . , αr), . . . , fn(α1, . . . , αr))T :

α1, . . . , αr take all possible real values}.

Constraints on the parameters α1, . . . , αr may be simple, such as:αt ≥ 0 for all t = 1 to r; or 0 ≤ αt ≤ 1 for all t = 1 to r, etc. By givingall possible real values to the parameters satisfying these constraints,we should get all points in the set S. Then in this case,

S = {(f1(α1, . . . , αr), . . . , fn(α1, . . . , αr)) : α1, . . . , αr

take all possible real values subject to the constraints on them}.

3.4. St. Lines 277

In (3.1), if r = 1, i.e., if there is only one parameter in the represen-tation, then the set S usually corresponds to the set of points on a onedimensional curve in Rn. If r = 2 (two parameters), then the set Susually corresponds to a set of points on a two dimensional surface inRn.

We will see examples of parametric representations of some subsetsin Rn in the following sections. For us, the most important subsets areof course straight lines, half-lines, and line segments, for which we givethe definitions here.

The expression in (3.1) represents a straight line in Rn for anyn ≥ 2, if r = 1 and all the functions f1, . . . , fn are affine functions (i.e.,polynomials of degree one or constants) with the parameter appearingwith a nonzero coefficient in the expression for at least one xj , andwith no constraints on the value of the parameter. In addition, if theparameter, α1 say, is required to satisfy an inequality constraint (i.e.,either α1 ≥ k or α1 ≤ k for some constant k, this type of constraintis usually transformed into and stated in the form of nonnegativityrestriction on the parameter) it represents a half-line in Rn. On theother hand if the parameter α1 is required to lie between two specifiedbounds (i.e., in the form k1 ≤ α1 ≤ k2 for specified k1 < k2, this typeof restriction is usually transformed into and stated in the form 0 ≤parameter ≤ 1) it represents a line segment in Rn. We discuss eachof these subsets in detail in the following sections.

3.4 Straight Lines in Rn

The general parametric representation of a straight line in Rn involvesonly one parameter, and is of the form

{x = (a + αb) : parameter α taking all real values}

where a, b are given vectors in Rn and b �= 0.Notice that if b = 0, x = a for all values of α, the above set

contains only one point a and not a straight line. That’s why in theabove representation b is required to be nonzero.

α = 0 gives x = a, and α = 1 gives x = a + b. So, both a, a + b are

278 Ch. 3. Geometry of R n

on the straight line. Since two points in any space uniquely determinea straight line, the above straight line is the one joining a and a + b.

Here are examples of parametric representations of straight linesin spaces of various dimensions (α is the only parameter in all theseexamples).

L1 =

{(x1

x2

)=

(3 − 4α8 + 2α

): α takes all real values

}

L2 =

{(x1

x2

)=

(2 − α−3

): α takes all real values

}

L3 =

x1

x2

x3

x4

x5

=

−6 − 4α3α03

8 − 7α

: α takes all real values

By giving the parameter the values 0 and 1 respectively, we get twodistinct points on the straight line, and the straight line itself is theunique one joining these two points. For example on the straight lineL3 in R

5 given above, these two points are

(−6, 0, 0, 3, 8)T and (−10, 3, 0, 3, 1)T

and L3 is the unique straight line joining these two points.When a parametric representation of a straight line L in R

n is givenin terms of the parameter α, using it, it is very easy to check whethera given point x ∈ R

n is contained on L or not. All we need to do ischeck whether the general point on L expressed in terms of α becomesx for some real value of α. If it does, then x ∈ L, otherwise x �∈ L.Here are some examples.

Example 1: Check whether x = (114,−90, 0, 3, 218)T is con-tained on the straight line L3 in R5 given above. For this we need tocheck whether there is a value of α which will make

3.4. St. Lines 279

−6 − 4α3α03

8 − 7α

=

114−9003

218

From the first component of these two vectors, we see that −6−4α =114 iff α = −30. We verify that when we substitute α = −30, theequation holds for all components. Hence x ∈ L3.

Now consider the point x2 = (−10, 3, 0, 5, 1)T . We see that theequation

−6 − 4α3α03

8 − 7α

=

−103051

can never be satisfied for any α, since the fourth components in the twovectors are not equal. So, x2 �∈ L3. Similarly x3 = (−10, 3, 0, 3, 15)T �∈L3 as there is no value of α which will satisfy

−6 − 4α3α03

8 − 7α

=

−1030315

Example 2: Consider the straight line {x = x(α) = (−1 +2α, 2 − α)T : α real}. This is the straight line joining the pointsx(0) = (−1, 2)T and x(1) = (1, 1)T in R2. See Figure 3.7

280 Ch. 3. Geometry of R n

x -axis1

2x -axis

.

..

0...

.

.

1 2

1

2

-1

-2

-1-2

..3

.(1, 1)T(-1, 2)T

Figure 3.7: A straight line in R 2.

Given Two Distinct Points in R n, How to Obtain

the Parametric Representation of the Straight LineJoining Them?

Suppose a = (a1, . . . , an)T and c = (c1, . . . , cn)T are two distinctpoints in R

n. The parametric representation of the unique straight linejoining them, with parameter α, is

x = x(α) = a + αb

where b = c − a. Since a, c are distinct, b = c − a �= 0. So the aboveparametric representation does represent a straight line. Also, verifyx(0) = a, and x(1) = a + b = a + c − a = c. So both a, c are on thisstraight line, hence it is the straight line joining them.

Example 3: Let

a =

( −13

), c =

(2

−1

), b = c − a =

(3

−4

).

Hence the straight line in R2 joining a, c is given by

3.4. St. Lines 281

{x = x(α) =

( −1 + 3α3 − 4α

): α takes all real values

}

How to Check if Two Different Parametric Rep-resentations Correspond to the Same Straight Line?

Consider the following three different parametric representations ofstraight lines:

L1 = {(2, 3)T + α(−1, 1)T : α real}L2 = {(2, 3)T + β(−2, 2)T : β real}L3 = {(−1, 6)T + γ(−3, 3)T : γ real}

Outwardly they seem to be three different lines. But actually if wesubstitute β = α/2 in L2 we see that L2 is the same set as L1, so L1

and L2 are the same line. Also, in L3 if we substitute γ = −1 + (α/3)we get the representation for L1, so L1 and L3 are the same line. So,all of the representations above correspond to the same straight line inR2.

In the same way, given two parametric representations in Rn:

U = {x = a + αb : α real}V = {x = c + βd : β real}

they both correspond to the same straight line in Rn if the followingconditions are satisfied:

(i) d is equal to a scalar multiple of b

(ii) a − c is a scalar multiple of b.

If either of these conditions does not hold, the straight lines U, Vare different.

282 Ch. 3. Geometry of R n

Example 4: Consider the representations for L1, L2, L3 givenabove. We verify that every pair of these representations correspondto the same straight line in R

2.

Example 5: Consider the representations

L4 = {x = (2, 3,−7)T + α(0, 1,−1)T : α real}L5 = {x = (2, 3,−7)T + β(0, 6,−1)T : β real}

Since the coefficient vectors of the parameters in these two represen-tations are (0, 1,−1)T , (0, 6,−1)T and these two vectors are not scalarmultiples of each other, L4, L5 are different staright lines.

Example 6: Consider the representations

L6 = {x = (4, 1, 5)T + α(1, 2, 3)T : α real}L7 = {x = (5, 3, 7)T + β(2, 4, 6)T : β real}

Here the coefficient vectors of the parameters in the two representa-tions (1, 2, 3)T , (2, 4, 6)T are scalar multiples of each other, so condition(i) is satisfied. When we put α, β = 0, the points obtained on L6, L7

are (4, 1, 5)T , (5, 3, 7)T and their difference (1, 2, 2)T is not a scalar mul-tiple of the coefficient vector of the parameter. Hence condition (ii) isviolated, so L6, L7 are different lines.

How to Obtain a Linear Equation Representa-tion of a Straight Line in R

n Given Its ParametricRepresentation

A straight line in R n is the set of solutions of a linearly independent

system of n − 1 linear equations in n variables. It is the affine hull ofany two distinct points in it. Its linear equation representation canbe obtained from its parametric representation using the algorithmdiscussed in Section 3.11.

Exercises

3.5. Half-Lines 283

3.4.1 Check whether the straight lines represented by the followingpairs of representations are the same or different (in (i), (ii) they arethe same; in (iii), (iv) they are different).

(i) {x = (0,−7, 2,−6)+α(−2, 1, 2, 4) : α real}; {x = (−6,−4, 8, 6)+β(−6, 3, 6, 12) : β real}.

(ii) {x = (18,−12, 3, 0) + α(0, 2, 1,−7) : α real}; {x = (18,−12, 3,0) + β(0, 1, 1/2,−7/2) : β real}.

(iii) {x = (8, 9−5, 4)+α(12, 18,−21, 9) : α real}; {x = (20, 27,−26,12) + β(4, 6,−7, 3) : β real}.

(iv) {x = (1, 1, 1, 1, 1) + α(10, 3,−7, 4, 2) : α real}; {x = (1, 1, 1, 1,1) + β(30, 9,−21, 11, 6) : β real}.

3.5 Half-Lines and Rays in Rn

The general parametric representation of a half-line is of the form {x =a + αb : α ≥ k} or {x = a + βb : β ≤ k} for some given vectorsa, b in R

n with b �= 0 and constant k. The first can be expressed as{x = (a + kb) + γb : γ ≥ 0}, and the second can be expressed as{x = (a + kb) + δ(−b) : δ ≥ 0}. For this reason, we take the generalparametric representation for a half-line to be

{x = x(α) = (x1(α), . . . , xn(α))T : α ≥ 0}where x(α) = a + αb for some a, b ∈ R

n with b �= 0. It is the subset ofthe straight line {x = x(α) : α takes all real values} corresponding tononnegative values of the partameter, that’s the reason for the name.This half-line is said to begin at a = (a1, . . . , an)T corresponding toα = 0, and said to have the direction vector b = (b1, . . . , bn)T . Thedirection vector for any half-line is always a nonzero vector.

Example 1: Consider the half-line in R 4

L+ = {x = x(α) = (6 − α,−3 + α, 4, 7 − 2α)T : α ≥ 0}

284 Ch. 3. Geometry of R n

Let

x1 = (4,−1, 4, 3)T , x2 = (8,−5, 4, 11)T , x3 = (5, 2, 4, 9)T

Determine which of these points are in L+. For this we need to finda nonnegative value for the parameter α that will make x(α) equal toeach of these points. For x1 this equation is

6 − α−3 + α

47 − 2α

=

4−143

and we verify that α = 2 > 0 satisfies this equation. So, x1 ∈ L+.For x2, the equation is

6 − α−3 + α

47 − 2α

=

8−5411

and α = −2 satisfies this equation. Since −2 �≥ 0, x2 �∈ L+.For x3, we have the equation

6 − α−3 + α

47 − 2α

=

5249

.

This cannot be satisfied by any α, hence x3 �∈ L+.

Example 2: Consider the half-line {x = x(α) = (1 + 2α,−2 +3α)T : α ≥ 0} in R2. It begins at x(0) = (1,−2)T , and x(1) = (3, 1)T

is a point on it. See Figure 3.8. Its direction vector is x(1) − x(0) =(2, 3)T .

A half-line which begins at 0 is called a ray. In Rn, for everya ∈ Rn, a �= 0, there is a unique ray, {x = αa : α ≥ 0}. a is called thedirection or the direction vector of this ray.

3.5. Half-Lines 285

Example 3: Let a = (−1,−2)T ∈ R 2. Its ray is shown in Figure

3.8 as the dotted half-line. Since (−10,−20)T = 10(−1,−2)T and thescalar multiple 10 > 0, the point (−10,−20)T is contained on this ray.

(10, 20)T = −10(−1,−2)T , and since the scalar multiplier −10 �≥ 0,the point (10, 20)T is not contained on the ray of (−1,−2)T .

For a ∈ Rn, a �= 0, verify that the rays of a, and γa for any γ > 0,are the same. So, the points a, γa for any γ > 0, correspond to thesame direction.

In R1, there are only two directions (positive direction, negativedirection). For any n ≥ 2, there are an infinite number of directions inRn.

Let a, b ∈ Rn, b �= 0. Then the half-line {x = a + αb : α ≥ 0} inRn beginning at a, is parallel to the ray of b. It is the translate of theray of b to a. The point a + αb on this half-line is known as the pointobtained by moving from a in the direction of b a step length of α.See Figure 3.9.

Two parametric representations

x -axis1

2x -axis

.

..

0...

.

.

1 2

1

2

-1

-2

-1-2

.

.3 .

.

.

.

(3, 1)

(2, 3)

(-1, -2)

3

(1, -2)

Figure 3.8: The half-line in R2 from (1,−2)T in the direction of (2, 3)T isthe solid half-line parallel to the ray of (2, 3)T which is the dashed half-line through the origin in the positive quadrant. The ray of (−1,−2)T

is the dotted half-line through the origin in the negative quadrant.

286 Ch. 3. Geometry of Rn

L1 = {x = a + αb : α ≥ 0}L2 = {x = c + βd : β ≥ 0}

represent the same half-line iff (i) starting points are the same, i.e.,a = c, and (ii) the directions b, d are positive scalar multiples of eachother.

For example {(2, 1, 3)+α(1, 2, 4) : α ≥ 0} and {(3, 3, 7)+β(1, 2, 4) :β ≥ 0} are not the same half-line because their starting points aredifferent. In fact the the 2nd half-line is a subset of the first.

The half-lines L1 = {(−1, 2, 1) + α(−2, 4, 2) : α ≥ 0} and L2 ={(−1, 2, 1) + β(−1, 2, 1) : β ≥ 0} are the same half line. However, asthey are represented, the points obtained by moving from the startingpoint a fixed step length (say 1 unit) in L1, L2 are not the same becausethe direction vector of L2 is half the direction vector of L1.

The half-lines {(10, 3)+α(−1, 2) : α ≥ 0}, {(10, 3)+β(1,−2) : β ≥0} are not the same because their direction vectors are not positive

multiples of each other. In fact they form the two half-lines of a straightline broken at the point (10, 3) which is the starting point of both thesehalf-lines.

3.6 Line Segments in Rn

A general line segment in Rn is of the form {x = a′ + βb′ : � ≤ β ≤ u}where a′, b′ are given vectors in Rn with b′ �= 0, and the bounds on theparameter satisfy � < u. By defining a = a′ + �b′, b = (u− �)b′ this canbe expressed as {x = x(α) = a + αb : 0 ≤ α ≤ 1}, where b �= 0. Forthis reason, the parametric representation for a line segment is usuallystated in this form.

3.6 Line Segments 287

x -axis1

2x -axis

.

..

0...

.

.

1 2

1

2

-1

-2

-1-2

.3

.

.

.

.3

(2, -1) = a

(3, 1) = a + b

(1, 2) = b

Half-line

Ray

Figure 3.9: A half-line through (2,−1)T parallel to the ray of (1, 2)T inR

2. The point (3, 1)T = (2,−1)T + (1, 2)T is obtained by moving from(2,−1)T a step length of 1 in the direction of (1, 2)T .

This line segment joins x(0) = a to x(1) = a + b, and is the portionof the straight line joining these two points that lies between them.x(0), x(1) are the two end points of this line segment.

Example 1: Consider the line segment {x = x(α) = (6−α,−3+α, 4 + 2α, 7 − 2α)T : 0 ≤ α ≤ 1}. Let x1 = (5.5,−2.5, 5, 6)T , x2 =(4,−1, 8, 3)T .

This line segment joins x(0) = (6,−3, 4, 7)T and x(1) = (5,−2, 6, 5)T

in R4. To check whether x1 is on this line segment, we consider theequation

288 Ch. 3. Geometry of R n

6 − α−3 + α4 + 2α7 − 2α

=

5.5−2.5

56

which has the unique solution α = 0.5, and since 0 < 0.5 < 1, x1 is onthis line segment.

Similarly, to check whether x2 is on this line segment, we considerthe equation

6 − α−3 + α4 + 2α7 − 2α

=

4−1

83

which has a solution α = 2. Since 2 is not between 0 and 1, x2 is noton this line segment.

Example 2: Consider the line segment {x = x(α) = (1 +2α,−2 + 3α)T : 0 ≤ α ≤ 1} in R

2. This line segment joins x(0) =(1,−2)T and x(1) = (3, 1)T . See Figure 3.10.

Suppose a = (a1, . . . , an)T and c = (c1, . . . , cn)T are two distinctpoints in R

n. The parametric representation (with parameter α) ofthe line segment joining them is {x = x(α) = a + αb : 0 ≤ α ≤ 1},where b = c − a �= 0 because a �= c. We verify that x(0) = a, andx(1) = a + b = a + c − a = c.

Example 3: Let a = (1,−2)T and c = (3, 1)T . Then b =c − a = (2, 3)T . Then the line segment joining a, c is {x = x(α) =(1 + 2α,−2 + 3α)T : 0 ≤ α ≤ 1} pictured in Figure 3.10 above.

Given two parametric representations for line segments, they repre-sent the same line segment iff the sets of end points for the line segmentsin the two representations are the same.

3.6 Line Segments 289

x -axis1

2x -axis

.

..

0...

.

.

1 2

1

2

-1

-2

-1-2

.3

.

.

.

3

(1, -2)T

(3, 1)T

Figure 3.10: The line segment joining (1,−2)T and (3, 1)T in R 2.

Exercises

3.6.1 (i): Let L be the straight line {x = x(α) = (7 − 2α, 8 +3α,−4,−2−3α) : α takes all real values}. Check whether the followingpoints are on L or not. (a) (Yes) (−13, 38,−4,−32), (19,−10,−4, 16),(7, 8,−4,−2); (b) (No) (5, 14,−4,−8), (1, 17,−4,−5).

(ii): Express the straight line joining the following pairs of points inparametric representation: {(1, 8,−6), (2, 6,−10)}, {(2, 1, 4, 0), (−3, 0,−2,−8)}, {(2, 2, 2), (1, 1, 1)}, {0, (2, 4)}. Draw the straight line joiningthe last pair of points on the 2-dimensional Cartesian plane.

(iii): Consider the half-line {x = x(α) = (6−α,−7+α, 8−2α, 4+α) : α ≥ 0}. Mention which of the following points are on this half-lineand which are not: (a) (Yes) (−4, 3,−12, 14), (6,−7, 8, 4); (b) (No)(8,−9, 12, 2), (5,−6, 4, 6), (6,−7, 6, 5).

(iv): Mention which of the following points are contained on the rayof (−5,−4,−1,−2) and which are not: (a) (Yes) (−25,−20,−5,−10);

290 Ch. 3. Geometry of R n

(b) (No) (5, 4, 1, 2), (−10,−8, 2, 4).

(v): Mention which of the following points are contained on theline segment {x = x(α) = (3 − α, 4 + α, 6 + 2α,−7 − 4α) : 0 ≤α ≤ 1}, and which are not: (a) (Yes) (2, 5, 8,−11), (5/2, 9/2, 7,−9),(11/4, 17/4, 13/2,−8); (b) (No) (1, 6, 10,−15), (5, 2, 2, 1).

3.7 Straight Lines Through the Origin

Let a = (a1, . . . , an)T �= 0 be a point in R n. Then the straight line

{x = x(α) = αa : α takes all real values}

is the straight line joining 0 and a, because x(0) = 0 and x(1) = a.This straight line contains the ray of a.

Example 1: The ray of (1,−1)T ∈ R2 and the straight linejoining 0 and (1,−1)T are shown in Figure 3.11.

Example 2: Consider the following points in R4.

a =

1−1

2−3

, a1 =

−1515

−3045

, a2 =

30−30

60−90

, a3 =

15−15

40−60

We have

Straight line joining 0 and a = {(α,−α, 2α,−3α)T : α takes

all real values}Ray of a = {(α,−α, 2α,−3α)T : α ≥ 0}

3.7 St. Lines Through 0 291

x -axis1

2x -axis

.

..

0...

.

.

1 2

1

2

-1

-2

-1-2

.3

..

.3

.(1, -1)

T

(2, -2)T

(-1, 1)T

Figure 3.11: The straight line joining 0 and (1,−1)T in R 2. The dashed

portion of this straight line is the ray of (1,−1)T .

a1 is obtained from (α,−α, 2α,−3α)T when α = −15. Since thisvalue of α < 0, a1 is not on the ray of a, but is on the straight linejoining 0 and a.

a2 is obtained from (α,−α, 2α,−3α)T when α = 30. Since thisvalue of α > 0, a2 is on the ray of a.

There is no value of α that will make a3 = (α,−α, 2α,−3α)T . So,a3 is not on the straight line joining 0 and a.

Exercises

3.7.1: Let a = (−1,−2)T . Plot the ray of a on the two dimensionalCartesian plane, and check whether (−6,−12)T , (7, 14)T , (−8,−17)T

are on the ray of a, or on the straight line joining 0 and a.

292 Ch. 3. Geometry of Rn

3.8 Geometric Interpretation of Addition

of Two Vectors, Parallelogram Law

Let

a =

(12

), b =

(21

); a + b =

(12

)+

(21

)=

(33

)

i.e., a + b is the fourth vertex of the parallelogram whose other threevertices are 0, a, b. See Figure 3.12.

x -axis1

2x -axis

.

..

0...

.

.

.

.

..

.a

b

a+b

Figure 3.12: Parallelogram law of addition of two vectors.

This is also true in Rn for any n ≥ 2. Given two distinct nonzerovectors a, b ∈ Rn, the two line segments [0, a], [0, b] are the two sides ofa unique two dimensional parallelogram whose fourth vertex is a + b.This gives the geometric interpretation for the sum of two vectors, andis known as the parallelogram law of addition of two vectors.

3.9. Linear Hulls 293

Exercises

3.8.1: Plot the points (1, 2), (2, 1), (3, 3) = (1, 2) + (2, 1) on the2-dimensional Cartesian plane and verify that these three points and 0form the vertices of a parallelogram.

3.9 Linear Combinations, Linear Hulls or

Spans, Subspaces

A subset S ⊂ R n is said to be a subspace of R

n if it satisfies thefollowing property:

for any two vectors a, b ∈ S, all linear combinations of a, b(any vector of the form αa+βb, for any real values of α, β)also belongs to S.

Since the origin, 0, is obtained as the zero linear combination ofany pair of vectors, 0 is contained in every subspace of R

n. Stating itanother way, a subset of R

n not containing 0, cannot be a subspace ofR

n.Examples of subspaces of R

n are the various coordinate planes ofvarious dimensions.

Linear Combinations of Vectors

Let {a1, . . . , ak } be a set of k vectors in R n. By our convention,

either all these are column vectors, or all these are row vectors; we willtreat the case where they are all column vectors. As defined in Section1.5, a linear combination of {a1, . . . , ak } is a vector of the form

α1a1 + . . . + αka

k

where α1, . . . , αk, the coefficients of the vectors a1, . . . , ak in this linearcombination, can be any real numers.

The set of all such linear combinations of {a1, . . . , ak} is a subspaceof Rn called the linear hull or the linear span of, or the subspace

294 Ch. 3. Geometry of R n

spanned by the set of vectors {a1, . . . , ak }. A parametric representa-tion of this subspace is

{α1a1 + . . . + αka

k : α1, . . . , αk take all real values}It may be possible to get a parametric representation of the same

subspace using fewer parameters. This is discussed later on in thissection.

Example 1: Let n = 2, and consider the single point set {(−1, 1)T }.Any vector of the form (−α, α)T for α real is a linear combination ofthis singleton set. So, the linear hull of this singleton set is the straightline joining 0 and (−1, 1)T . See Figure 3.13.

x -axis1

2x -axis

.

..

0...

.

.

1 2

1

2

-1

-2

-1-2

.3

..

3

(-1, 1)T

Figure 3.13: The subspace spanned by (−1, 1)T .

Example 2: Let n = 3, and consider the set of vectors in R 3

100

,

010

3.9. Linear Hulls 295

A linear combination of this set is a vector of the form (α1, α2, 0)T ,i.e., a vector in the x1x2-coordinate plane in R3. See Figure 3.14.

x1

x2

x3

0

.

.(1, 0, 0)

(0, 1, 0)

Figure 3.14: The linear hull of {(1, 0, 0)T , (0, 1, 0)T} in R3 is the x1x2-coordinate plane.

296 Ch. 3. Geometry of R n

Algorithm to Check if A Column Vector b ∈ R n

Is In the Linear Hull of a Set of Column Vectors{a1, . . . , ak } ⊂ R

n

BEGIN

For this, you have to solve the following system of linear equationsin which the variables are the coefficients α1, . . . , αk.

α1 . . . αk

a1 . . . ak b

Either the GJ pivotal method, or the G elimination method dis-cussed earlier (Sections 1.16, 1.20) can be used to solve this system.

If the system has no solution, b is not in the linear hull of {a1, . . . , ak }.If the system has a solution, let a solution for it be (α1, . . . , αk).

Then b = α1a1 + . . . + αka

k, and hence it is in the linear hull of{a1, . . . , ak }.

END

Example 3: Let

a1 =

1000

, a2 =

0100

′

a3 =

1−1

1−1

b1 =

3−1

2−2

, b2 =

203

−4

.

To check whether b1 is in the linear hull of {a1, a2, a3} we solve thefollowing system of equations by the GJ method. Since the first two

3.9. Linear Hulls 297

column vectors in this tableau are already unit vectors, we can selectα1, α2 as basic variables in rows 1, 2 directly. Then we select the columnof α3 as the pivot column to perform a GJ pivot step in Row 3. Thecolumn headed “BV” stores the basic variables selected in the rows.“PC”, “PR” indicate the pivot column, pivot row respectively for thepivot step, and the pivot element common to them is boxed. “RC” in-dicates a “redundant constraint discovered, which is to be eliminated”.“IE” indicates an “inconsistent equation” detected in the algorithm.

BV α1 α2 α3

α1 1 0 1 3α2 0 1 −1 −1

0 0 1 2 PR0 0 −1 −2

PCα1 1 0 0 1α2 0 1 0 1α3 0 0 1 2

0 0 0 0 RC

So, the system has the unique solution (α1, α2, α3) = (1, 1, 2), i.e.,b1 = a1 + a2 + 2a3.

So b1 is in the linear hull of {a1, a2, a3}.To check whether b2 is in the linear hull of {a1, a2, a3} we need to

solve the following system linear equations.

BV α1 α2 α3

α1 1 0 1 2α2 0 1 −1 0

0 0 1 3 PR0 0 −1 −4

PC↑α1 1 0 0 −1α2 0 1 0 3α3 0 0 1 3

0 0 0 −1 IE

298 Ch. 3. Geometry of Rn

From the last row of the final tableau, we see that this system hasno solution. So, b2 is not in the linear hull of {a1, a2, a3}.

Parametric Representation of the Linear Hull ofa Set of Vectors Using the Smallest Number of Pa-rameters

Earlier in this section, we discussed the parametric representationof the linear hull of {a1, . . . , ak} ⊂ Rn as

L = {α1a1 + . . . + αka

k : α1, . . . , αk take all real values}involving k parameters, that comes directly from its definition. How-ever, it may not be necessary to include all the vectors a1, . . . , ak inthis parametric representation to get L. As an example, let

a1 =

100

, a2 =

010

a3 =

110

, a4 =

1−1

0

Here a3, a4 are themselves linear combinations of a1, a2. Hence lin-ear hull of {a1, a2, a3, a4} = linear hull of {a1, a2}.

In the same way, given {a1, . . . , ak} ⊂ Rn, if one of the vectors inthis set, say ak, is itself a linear combination of the remaining vectorsin the set, then

Linear hull of {a1, . . . , ak} = Linear hull of {a1, . . . , ak−1}.

So, to write the parametric representation of {a1, . . . , ak}, we canomit the vector ak from this set. The same process can be repeatedwith the remaining set {a1, . . . , ak−1}.

In the example given above, we have a3 = a1 + a2; or

−a1 − a2 + a3 = 0

An expression like this, which expresses the 0-vector as a linearcombination of vectors in a set, with at least one nonzero coefficient,is called a linear dependence relation for that set of vectors.

3.9. Linear Hulls 299

Whenever there is a linear dependence relation for a set of vectors,that relation can be used to express one of the vectors in the set as alinear combination of the others, and in this case the set of vectors issaid to be linearly dependent.

On the contrary, if there is no linear dependence relation for a setof vectors, that set is said to be linearly independent.

Given a set of vectors Γ = {a1, . . . , ak} ⊂ Rn, a subset ∆ of Γ is saidto be a maximally linearly independent subset of Γ if it satisfiesthe following two conditions:

(a): ∆ must be linearly independent, and

(b): including any other vector from Γ\∆ in ∆ makes itlinearly dependent.

All maximal linearly independent subsets of a set of vectors Γ ⊂ Rn

always contain the same number of vectors in them, this number iscalled the rank of the set of vectors Γ.

The concepts of linear independence, linear dependence of sets ofvectors will be discussed at greater length in Chapter 4. For now, allwe need is the fact that L = linear hull of Γ is the same as the linearhull of any maximally linearly independent subset of vectors of Γ, andthat expressing it this way leads to a parametric representation for Lusing the smallest number of parameters.

Representing the Linear Hull of a set of VectorsBy a System of Homogeneous Linear Equations

We have the following important result.

Result 3.9.1: Representing the linear hull of a given set ofvectors as the solution set of a homogeneous system of linearequations: Let Γ = {a1, . . . , ak} ⊂ Rn where, for t = 1 to k

at =

at1...

atn

300 Ch. 3. Geometry of Rn

and let L = linear hull of Γ. Then there exists a matrix B of orders × n such that

L = linear hull of Γ = {x : Bx = 0}where s ≤ n; and s is the smallest number of homogeneous linear equa-tions required to represent L this way, i.e., the s equations in Bx = 0are linearly independent. (Defined in Section 1.9, it means that none ofthese equations can be obtained as a linear combination of the others).In fact s = n − rank of the set Γ as explained above.

We will now discuss an algorithm for deriving these equations inBx = 0 given {a1, . . . , ak}.

Algorithm to Find Representations for L = Lin-ear Hull of {a1, . . . , ak} From Rn

BEGIN

Step 1: Let (b1, . . . , bn) �= 0 denote the coefficients in a generalhomogeneous equation

b1y1 + . . . + bnyn = 0 (1)

satisfied by each of the vectors at = (at1, . . . , a

tn)T . So, the unknown

coefficients b1, . . . , bn satisfy the following k equations expressed in de-tached coefficient tableau form.

Original Tableaub1 b2 . . . bn

a11 a1

2 . . . a1n 0

a21 a2

2 . . . a2n 0

......

......

ak1 ak

2 . . . akn 0

Notice that the tth row in this tableau is the vector at written as arow vector, i.e., (at)T , for t = 1 to k. The coefficients b1, b2, . . . , bn inthe homogeneous equation (1) are the variables in this system.—

3.9. Linear Hulls 301

Step 2: Now apply the GJ or the G pivotal method (Sections 1.16,1.23) to find the general solution of this homogeneous system. In thisprocess, the method may discover some constraints in the tableau asbeing redundant and eliminate them. In the final tableau obtainedunder this method, a basic variable is selected in each remaining row.Rearrange the variables and their column vectors in the final tableau sothat the basic variables appear first in their proper order, followed bynonbasic variables if any. Let r denote the number of rows in the finaltableau; and suppose the basic variables are bi1 , . . . , bir in that order;and the nonbasic variables are bj1 , . . . , bjn−r . Let the final tableau be

Final tableau after rearrangement of variablesBasic bi1 bi2 . . . bir bj1 . . . bjn−r

Vars.bi1 1 0 . . . 0 a1,j1 . . . a1,jn−r 0bi2 0 1 . . . 0 a2,j1 . . . a2,jn−r 0...

......

......

......

bir 0 0 . . . 1 ar,j1 . . . ar,jn−r 0

This will be the final tableau if the GJ method is used. If the Gmethod is used instead, the portion of the tableau corresponding to thebasic variables will be an upper triangular matrix instead of the unitmatrix.

Case 1: If r = n, i.e., there are no nonbasic variables in the finaltableau; then (b1, . . . , bn) = 0 is the unique solution for the homoge-neous system in the tableau. In this case, the only equation of the form(1) satisfied by all of the vectors a1, . . . , ak is the one in which all ofb1, . . . , bn are 0. In this case the linear hull of {a1, . . . , ak} is all of Rn.

Case 2: Suppose r < n, i.e., there is at least one nonbasic variablein the final tableau. Then the number of equations s required to rep-resent the linear hull of {a1, . . . , ak} is n − r. One of these equationscorresponds to each nonbasic variable in the final tableau.

For each nonbasic variable bt in the final tableau, t = j1, . . . , jn − r,obtain the unique solution of the homogeneous system given by—

302 Ch. 3. Geometry of Rn

bt = 1, and all other nonbasic variables = 0.

The values of the basic variables in this solution can be obtainedfrom the final tableau by substituting the above values for the nonbasicvariables. Write this solution as a row vector. When you put each ofthese row vectors one below the other, we get a matrix of order s × n(where s = n − r). Call this matrix B. Then

the linear hull of {a1, . . . , ak} = {x : Bx = 0}.

So, Bx = 0 gives a linear equation representation for the linear hullof {a1, . . . , ak}.

Therefore, typically, the linear hull of a set of vectors can be ex-pressed as the set of solutions of a homogeneous system of linear equa-tions.

The number r of equations in the final tableau, is called the rankof the set of given vectors {a1, . . . , ak}. The concept of rank will be dis-cussed at greater length in Chapter 4. This rank is also the dimensionof the linear hull of {a1, . . . , ak}.

In the original tableau, the tth row corresponds to the vector at fort = 1 to k. In this method, suppose pivot steps are carried out fromtop row to bottom row in that order. When we come to row t, if wefind that it is “0 = 0” in the current tableau at that time, clearly thereis a linear dependence relation among {a1, . . . , at} from which we canget an expression for at as a linear combination of {a1, . . . , at−1}. Thislinear dependence relation can actually be obtained by introducingthe memory matrix as discussed in Section 1.12 at the beginning ofcomputation.

Thus if we number the rows in the original tableau as 1 to k from topto bottom, and keep this numbering in all subsequent tableaus obtainedin the method, the rows that remain in the final tableau (i.e., those noteliminated as redundant constraints during the algorithm) correspondto vectors in a maximal linearly independent subset of {a1, . . . , ak}.Suppose the numbers of the rows in the final tableau are p1, . . . , pr.Then—

{ap 1, . . . , ap r }

3.9. Linear Hulls 303

is a maximal linearly independent subset of {a1, . . . , ak }.Then a parametric representation of the linear hull of {a1, . . . , ak }

involving the smallest number of parameters is

L = Linear hull of {a1, . . . , ak }= {x = α1a

p 1 + α2ap 2 + . . . αra

p r : α1, . . . , αr take all possible

real values}

END

Example 4: Let

a1 =

1−1

011

, a2 =

2−2

022

, a3 =

−101

−1−1

a4 =

0−1

100

, a5 =

1−2

111

To get the linear constraint representation for the linear hull of{a1, a2, a3, a4, a5}, we start with the following system of homogeneousequations, on which we apply the GJ pivotal method to get the gen-eral solution as discussed in Section 1.23. We number the equationsto get the parametric representation of this linear hull using the small-est number of parameters. PR (pivot row), PC (pivot column), RC(redundant constraint to be eliminated), BV (basic variable selectedin this row) have the same meanings as in Section 1.23, and in eachtableau the pivot element for the pivot step carried out in that tableauis boxed.

304 Ch. 3. Geometry of Rn

BV b1 b2 b3 b4 b5 RHS Eq. no.

1 −1 0 1 1 0 PR 12 −2 0 2 2 0 2

−1 0 1 −1 −1 0 30 −1 1 0 0 0 41 −2 1 1 1 0 5

PC↑b1 1 −1 0 1 1 0 1

0 0 0 0 0 0 RC 2

0 −1 1 0 0 0 PR 30 −1 1 0 0 0 40 −1 1 0 0 0 5

PC↑b1 1 −1 0 1 1 0 1b3 0 −1 1 0 0 0 3

0 0 0 0 0 0 RC 40 0 0 0 0 0 RC 5

Final tableau after rearrangement ofbasic variables in their order

BV b1 b3 b2 b4 b5 RHS Eq. no.b1 1 0 −1 1 1 0 1b3 0 1 −1 0 0 0 3

There are three nonbasic variables in the final tableau, b2, b4, b5. Byfixing one of these nonbasic variables equal to 1, and the others equalto 0, we are lead to the following solutions of this homogeneous system:

(b2, b4, b5) = (1, 0, 0) yields (b1, b2, b3, b4, b5) = (1, 1, 1, 0, 0)

(b2, b4, b5) = (0, 1, 0) yields (b1, b2, b3, b4, b5) = (−1, 0, 0, 1, 0)

(b2, b4, b5) = (0, 0, 1) yields (b1, b2, b3, b4, b5) = (−1, 0, 0, 0, 1)

Letting

B =

1 1 1 0 0−1 0 0 1 0−1 0 0 0 1

, x = (x1, x2, x3, x4, x5)

T

3.9. Linear Hulls 305

the linear hull of {a1, a2, a3, a4, a5 } is {x : Bx = 0}.It can be verified that each of a1 to a5 satisfies all these equations,

and so every linear combination of a1 to a5 also does.

Since there are only two equations left in the final tableau, the rankof {a1, a2, a3, a4, a5 } is 2, which is also the dimension of their linear hullin R

5.

The original row numbers corresponding to those in the final tableauare 1 and 3. Therefore, the corresponding set of original vectors {a1, a3 }forms a maximal linearly independent subset of {a1, . . . , a5 }, and hencethe

linear hull of {a1, . . . , a5 } = {x = α1a1 + α3a

3 : α1, α3 takeall real values }.

Example 5: Let

a1 =

111

, a2 =

1−1

1

, a3 =

11

−1

, a4 =

110

To get the linear constraint representation for the linear hull of{a1, a2, a3, a4, }, we start with the following system of homogeneousequations, on which we apply the GJ pivotal method to get the generalsolution. PR, PC, RC have the same meanings as in the above example.The pivot elements are boxed.

BV b1 b2 b3 RHS Eq. no.

1 1 1 0 PR 11 −1 1 0 21 1 −1 0 31 1 0 0 4

PC↑

306 Ch. 3. Geometry of R n

BV b1 b2 b3 RHS Eq. no.b1 1 1 1 0 1

0 −2 0 0 PR 20 0 −2 0 30 0 −1 0 4

PC↑b1 1 0 1 0 1b2 0 1 0 0 2

0 0 −2 0 PR 30 0 −1 0 4

PC↑b1 1 0 0 0 1b2 0 1 0 0 2b3 0 0 1 0 3

0 0 0 0 RC 4Final tableau

b1 1 0 0 0 1b2 0 1 0 0 2b3 0 0 1 0 3

The final tableau has no nonbasic variables, hence (b1, b2, b3) = 0 isthe unique solution of this homogeneous system. Therefore, the linearhull of {a1, a2, a3, a4 } here is R

3.Also, the original row numbers for the rows in the final tableau

are 1, 2, 3. Therefore the set of corresponding vectors, {a1, a2, a3 } isa maximal linearly independent subset of {a1, a2, a3, a4 }, and a para-metric representation of the linear hull of this set using the smallestnumber of parameters is

R 3 = linear hull of {a1, a2, a3, a4 } = {x = α1a

1 + α2a2 +

α3a3 : α1, α2, α3 take all real values}.

Exercises

3.9.1 (i): Let A.1 to A.5 be the column vectors of A given below.

3.10. Column, Row, and Null Spaces 307

(a) (Yes) Are b1, b2 in the linear hull of {A.1, . . . , A.5}? If so, findthe actual linear combination for each of them.

(b) (No) Are b3, b4 in the linear hull of {A.1, . . . , A.5}?

A =

1 −1 1 0 −1−1 2 2 2 1−1 4 8 6 1

0 2 3 2 −1

, b1 =

−235

−1

, b2 =

(3 −2 0 2

),

b3 =(−1 −3 −9 2

), b4 =

(2 4 13 3

)

(ii): Let L represent the linear hull of the set of row vectors ofthe following matrix. Express L as the set of solutions of a system ofhomogeneous linear equations. ((a) none, (b) one, (c) three, (d) three).

(a)

1 −1 −1 2−1 2 0 3

0 −1 1 −40 −1 0 0

−1 1 0 4

, (b)

−2 3 −1 01 −4 1 22 1 0 −31 0 0 −1

−1 3 −1 −1

,

(c)

−3 1 −4 5−7 2 2 −1

4 −1 −6 6−13 4 −6 9−17 5 0 3

, (d)

2 3 3 212 18 18 128 12 12 8

10 15 15 10

3.9.2: Find a linear constraint representation of the linear hull of :{(2, 4, 6,−8, 0)T}.

3.10 Column Space, Row Space, and Null

Space of a Matrix

Let A be a given matrix of order m × n. So, the set of row vectors ofA is {A1., . . . , Am.}, and its set of column vectors is {A.1, . . . , A.n}.

308 Ch. 3. Geometry of Rn

The row space of A, the space spanned by the set of row vectorsof A, is the linear hull of {A1., . . . , Am.}. A general point in the rowspace of A, treated as a row vector, is α1A1. + . . .+αmAm., for any realnumbers α1, . . . , αm; i.e., it is (α1, . . . , αm)A. So, the row space of Aviewed as a set of row vectors is {yA : y ∈ Rm}.

The column space of A, the space spanned by the set of columnvectors of A, is the linear hull of {A.1, . . . , A.n}. A general point in thecolumn space of A, treated as a column vector, is β1A.1 + . . . + βnA.n,for any real numbers β1, . . . , βn; i.e., it is Aβ where β = (β1, . . . , βn)T .In mathematics books, the column space of A is also called the rangespace of A, or the image space of A, or the range of A, and denotedby R(A). It is viewed as a set of column vectors, i.e., R(A) = {Ax :x ∈ Rn}.

When the row space of A is viewed as a set of column vectors bytransposing each vector in it, it is R(AT ) in the above notation.

The null space of A, denoted by N(A), is the set of all solutionsof the homogeneous system of linear equations Ax = 0. So, N(A) ={x ∈ Rn : Ax = 0}, and it is a set of column vectors.

3.11 Affine Combinations, Affine Hulls or

Spans, Affinespaces

Let {a1, . . . , ak} be a set of k vectors in Rn. By our convention, eitherall these are column vectors, or all these are row vectors. We will treatthe case where they are all column vectors. As defined in Section 1.5,an affine combination of {a1, . . . , ak} is a vector of the form

α1a1 + . . . + αka

k

where α1, . . . , αk, the coefficients of the vectors a1, . . . , ak in this lin-ear combination, are real numers satisfying the affine combinationcondition

α1 + . . . + αk = 1.

3.11. Affine Hulls 309

The set of all such linear combinations of {a1, . . . , ak } is called theaffine hull or the affine span or the flat or the affine space of theset of vectors {a1, . . . , ak }.

From the affine combination condition α1 + . . . + αk = 1, we havethe affine combination

α1a1 + . . . + αka

k = (1 − α2 − . . . − αk)a1 + α2a

2 + . . . + αkak

= a1 + [α2(a2 − a1) + . . . + αk(a

k − a1)]

Let α2(a2 − a1) + . . . + αk(a

k − a1) = y. y is a linear combinationof {(a2 −a1), . . . , (ak −a1)}, and as α2, . . . , αk assume all real values, yvaries over all of the linear hull of {(a2 −a1), . . . , (ak −a1)}. Therefore,the affine hull of {a1, . . . , ak } is

a1 + (linear hull of {(a2 − a1), . . . , (ak − a1)})a translate of the linear hull of {(a2 − a1), . . . , (ak − a1)} to a1.

A Straight Line Is the Affine Hull of Any twoPoints On It

The importance of affine hulls stems from the fact that the straightline joining any two points in R

n is the affine hull of those two points.Consider two distinct points a = (a1, . . . , an)T , c = (c1, . . . , cn)T in R

n.From Section 3.4, we know that the parametric representation of thestraight line joining a, c is {x = a + α(c − a) : α takes all real values}.So, the general point on this straight line is a+α(c−a) = (1−α)a+αc,an affine combination of {a, c}.

Thus a straight line is the affine hull of any two distinct points onit.

Example 1: Let

a1 =

2−1

1

, a2 =

12

−1

, a3 =

−211−7

, a4 =

2−1

0

.

310 Ch. 3. Geometry of Rn

x -axis1

2x -axis

.

..

0...

.

.

1 2

1

2

-1

-2

-1-2

.3

.3

(1, 0)(0, 1)

Figure 3.15: The Straight Line Joining (1, 0)T , (0, 1)T is their affinehull.

The parametric representation of the straight line joining a1, a2 is

{x = a1+α(a2−a1) = (2−α,−1+3α, 1−2α)T : α takes all real values}

which is the same as {x = (1 − α)a1 + αa2 : α takes all real values}.

How to Check if ak +1 Is In the Affine Hull of{a1, . . . , ak }?

ak+1 is in the affine hull of {a1, . . . , ak} iff the following system ofequations

α1a1 + . . . + αka

k = ak+1

α1 + . . . + αk = 1

in the parameters α1, . . . , αk, has a solution. This system can be solvedby the GJ pivotal method (Section 1.16) or the G elimination method

3.11. Affine Hulls 311

x1

x2

x3

0

..

.a1

a3

a2

Figure 3.16: The affine hull of {a1, a2, a3 } in R 3 is the shaded two

dimensional plane containing these three points.

(Section 1.20). If this system has no solution, ak +1 is not in the affinehull of {a1, . . . , ak }. If the system has the solution (α1, . . . , αk) thenak +1 is the affine combination α1a

1 + . . . + αkak.

Example 2: Let

a1 =

1000

, a2 =

0100

, a3 =

1−1

1−1

, a4 =

3−1

2−2

, a5 =

2−4

3−3

.

To check whether a4 is in the affine hull of {a1, a2, a3}, we solve thefollowing system in the top tableau by the GJ pivotal method. BV(basic variable selected in row), PR (pivot row), PC (pivot column),RC (redundant constraint to be deleted), IE (inconsistent equation

312 Ch. 3. Geometry of Rn

identified) have the same meanings as before, and the pivot element ineach step is boxed.

BV α1 α2 α3

1 0 1 3 PR0 1 −1 −10 0 1 20 0 −1 −21 1 1 1

PC↑α1 1 0 1 3

0 1 −1 −1 PR0 0 1 20 0 −1 −20 1 0 −2

PC↑α1 1 0 1 3α2 0 1 −1 −1

0 0 1 2 PR0 0 −1 −20 0 1 −1

PC↑α1 1 0 0 1α2 0 1 0 1α3 0 0 1 2

0 0 0 0 RC0 0 0 −3 IE

Since an inconsistent equation is discovered, the system has no so-lution, therefore a4 is not in the affine hull of {a1, a2, a3}.

To check whether a5 is in the affine hull of {a1, a2, a3}, we solve thefollowing system of equations in the top tableau given below by the GJpivotal method. The abbreviations have the same meanings as above,and in each step the pivot element is boxed.

3.11. Affine Hulls 313

BV α1 α2 α3

1 0 1 2 PR0 1 −1 −40 0 1 30 0 −1 −31 1 1 1

PC↑α1 1 0 1 2

0 1 −1 −4 PR0 0 1 30 0 −1 −30 1 0 −1

PC↑α1 1 0 1 2α2 0 1 −1 −4

0 0 1 3 PR0 0 −1 −30 0 1 3

PC↑α1 1 0 0 −1α2 0 1 0 −1α3 0 0 1 3

0 0 0 0 RC0 0 0 0 RC

This system has the solution (α1, α2, α3) = (−1,−1, 3). So, a5 is inthe affine hull of {a1, a2, a3}. In fact

a5 = −a1 − a2 + 3a3.

Two Ways of Representing the Affine Hull of aGiven Set of Vectors

Let {a1, . . . , ak} be a given set of column vectors in Rn, and let Fdenote their affine hull. From the above discussion we know that

314 Ch. 3. Geometry of R n

F = a1 + L

where L = linear hull of {(a2 − a1), . . . , (ak − a1)}.Find the representation of L as discussed in Section 3.9. If

rank{(a2 − a1), . . . , (ak − a1)} is r

and {ai1 −a1), . . . , (air −a1)} is a maximal linearly independent subsetof {(a2 −a1), . . . , (ak −a1)}, and a parametric representation of L usingthe smallest number of parameters is

L = {x = α1(ai1 − a1) + . . . + αr(a

ir − a1) : α1, . . . , αr real}.

Thus, a parametric representation of F using the smallest numberof parameters is

F = {x = a1 + α1(ai1 − a1) + . . . + αr(a

ir − a1) :

α1, . . . , αr take all possible real values}.

For another way of representing the affine hull F of {a1, . . . , ak },let the linear constraint representation of L = linear hull of {(a2 −a1), . . . , (ak −a1)} obtained as discussed in Section 3.9 be {x : Ax = 0}.Let

Aa1 = b

Then, in linear constraint representation,

F = {x : Ax = b}

So, typically, the affine hull of a set of vectors is the set of solutionsof a nonhomogeneous system of linear equations.

Example 3: Let

3.11. Affine Hulls 315

a1 =

11111

, a2 =

20122

, a3 =

3−1

133

,

a4 =

01200

, a5 =

10211

a6 =

2−1

222

.

Then we have

Γ =

a2 − a1 =

1−1

011

, a3 − a1 =

2−2

022

, a4 − a1 =

−101

−1−1

,

a5 − a1 =

0−1

100

, a6 − a1 =

1−2

111

The linear constraint representation of the linear hull of Γ was de-rived in Section 3.9 to be Ax = 0 (in Section 3.9 the coefficient matrixwas denoted by B, here we denote the same by A) where

x = (x1, x2, x3, x4, x5)T

A =

1 1 1 0 0−1 0 0 1 0−1 0 0 0 1

Verify that Aa1 = (3, 0, 0)T = b.

316 Ch. 3. Geometry of Rn

Then the linear constraint representation of the affine hull of {a1 toa6} is Ax = b with, A, b given above.

Also, in Section 3.9 we found that a maximal linearly independentsubset of Γ is {a2 − a1, a4 − a1}. So, a parametric representation ofthe affine hull of {a1 to a6} using the smallest number of parametersis {x = a1 + α1(a

2 − a1) + α2(a3 − a1) : α1, α2 take all real values}.

Direct Algorithm to Find a Linear Equation Rep-resentation of the Affine Hull of a Given Set of Vec-tors Directly Using GJ or G Pivot Steps

Let F be the affine hull of {a1, . . . , ak}, a given set of column vectorsin Rn. So,

F = {x = α1a1 + α2a

2 + . . . + αkak : α1 + . . . + αk = 1}

= {x = a1 + β1c1 + . . . + βk−1c

k−1 : β1, . . . , βk−1 real}where β1 = α2 − α1, . . . , βk−1 = αk − α1(and hence as α1, . . . , αk takeall real values subject to α1 + . . . + αk = 1, β1, . . . , βk−1 take all realvalues) and c1 = a2 − a1, . . . , ck−1 = ak−1 − a1.

This representation of F is like the parametric representation of thegeneral solution to a system of linear equations discussed in Section1.8, but somewhat more general. In the parametric representation inSection 1.8, the following property holds:

Special property of pramet-ric representation obtainedin GJ, or G methods (Sec-tions 1.16, 1.20)

Each of the parameters isthe value of one of the vari-ables xj .

For instance, here is a parametric representation from Section 1.8

x1

x2

x3

=

10 − x3

20 − x3

x3

, x3 real parameter

Here the only parameter x3 is the value of the variable x3 in thisrepresentation.

3.11. Affine Hulls 317

Under this special property, all you need to do to get a linear equa-tion system with this general solution is to replace each parameter inthe representation by the variable to which it is equal, and then writethe system of equations that comes from the representation.

For instance in the above representation, replacing the parameterx3 by x3 leads to

x1

x2

x3

=

10 − x3

20 − x3

x3

or the system: x1 + x3 = 10; x2 + x3 = 20 as the linear equationrepresentation.

For another example, we take the parametric representation of thegeneral solution in Example 1, Section 1.16; it is

x1

x2

x3

x4

x5

=

x1

−60 − 9x1 − 9x4

−40 + 5x1 − 5x4

x4

25 − 2x1 + 2x4

Here the parameters x4, x1 are the values of x4, x1 in the repre-sentation. Replacing x4 by x4, and x1 by x1 leads to the system ofequations

x2 = −60 + 9x1 − 9x4

x3 = −40 + 5x1 − 5x4

x5 = 25 − 2x1 + 2x4.

This system is exactly the system of equations from the final tableauobtained under the GJ method. It is equivalent to the original systemof equations in this example in the sense that both systems have thesame set of solutions.

However this special property may not hold for the general paramet-ric representation. For an example see Example 4 given below in which

318 Ch. 3. Geometry of Rn

each of the variables involves at least two parameters with nonzerocoefficients.

Algorithm to Find a Linear Equation Represen-tation of F = {x = a1+β1c

1+ . . .+βk−1ck−1 : β1, . . . , βk−1

real}BEGIN

Step 1. Write the n equations expressing each of the variables xj

in terms of the parameters, with the terms involving the parameterstransferred to the left, and put it in detached coefficient tableau form.It is the original tableau.

Original tableaux1 . . . xn β1 . . . βk−1

I.1 . . . I.n −c1 . . . −ck−1 a1

where I.1, . . . , I.n are the n column vectors of the unit matrix of ordern.

Step 2. Perform GJ or G pivot steps on this tableau in as manyrows as possible using only the columns of the parameters as pivotcolumns. Whenever a pivot step is completed with a row as the pivotrow, record the parameter associated with the pivot column as thebasic parameter in that row.

Terminate the pivot steps when each of the remaining rows in whichpivot steps have not been carried out so far has a 0-entry in the columnassociated with each of the parameters. This is the final tableau in theprocedure.

Step 3. If every row in the final tableau has a basic parameter, Fis the whole space Rn, terminate.

If there are rows without a basic parameter in the final tableau, eachsuch row must have a 0-entry in the column of every parameter (other-wise Step 2 in incomplete as a pivot step can be carried out in this row).—

3.11. Affine Hulls 319

So, the equation corresponding to this row in the final tableau involvesonly the xj variables and none of the parameters. Write the equationscorresponding to all such rows in a system, this system provides a linearequation representation for F , terminate.

END

Example 4: Find a linear equation representation for the affineset

F =

x1

x2

x3

=

9 − β1 − 2β2

−6 + β1 + 3β2 − β3

−7 − β2 + 2β3

: β1, β2, β3 real

We apply the algorithm using GJ pivot steps. The original tableauis the one at the top. Pivot elements are boxed. The PR, pivot row isthe row with the pivot element; and the PC, pivot column is the columnwith the pivot element. BP stands for “Basic Parameter selected in thisrow”.

BP x1 x2 x3 β1 β2 β3

1 0 0 1 2 0 90 1 0 −1 −3 1 −60 0 1 0 1 −2 −7

β1 1 0 0 1 2 0 9

1 1 0 0 −1 1 30 0 1 0 1 −2 −7

β1 3 2 0 1 0 2 15β2 −1 −1 0 0 1 −1 −3

1 1 1 0 0 −1 −4Final tableau

β1 5 4 2 1 0 0 7β2 −2 −2 −1 0 1 0 1β3 −1 −1 −1 0 0 1 4

Since every row in the final tableau has a basic parameter, F = R3,the whole space in this example.

320 Ch. 3. Geometry of R n

Example 5: Find a linear equation representation for the affinehull of {a1, a2, a3, a4, a5 } given below.

a1 =

1111

, a2 =

2122

, a3 =

3133

,

a4 =

1211

a5 =

2222

.

So, F = {x = a1 + β1c1 + β2c

2 + β3c3 + β4c

4 : β1 to β4 take all realvalues}, where c1 to c4 are given below.

c1 = a2 − a1 =

1011

, c2 = a3 − a1 =

2022

c3 = a4 − a1 =

0100

, c4 = a5 − a1 =

1111

We apply the algorithm using GJ pivot steps and the terminologyof the previous example.

3.11. Affine Hulls 321

BP x1 x2 x3 x4 β1 β2 β3 β4

1 0 0 0 −1 −2 0 −1 10 1 0 0 0 0 −1 −1 10 0 1 0 −1 −2 0 −1 10 0 0 1 −1 −2 0 −1 1

β1 −1 0 0 0 1 2 0 1 −1

0 1 0 0 0 0 −1 −1 1−1 0 1 0 0 0 0 0 0−1 0 0 1 0 0 0 0 0

β1 −1 0 0 0 1 2 0 1 −1β3 0 −1 0 0 0 0 1 1 −1

−1 0 1 0 0 0 0 0 0−1 0 0 1 0 0 0 0 0

In rows 3, 4 in which pivot steps have not been carried out yet,all the parameter columns have 0 entries, hence the pivot steps areterminated. The system of equations corresponding to rows 3, 4 in thefinal tableau which do not have a basic parameter is

−x1 + x4 = 0

−x1 + x5 = 0

which provides a linear equation representation for F .Also, from earlier discussion we see that the column vectors of the

nonbasic parameters β2, β4 are linear combinations of column vectorsof β1, β3. This implies that

{β1c1 + . . . + β4c

4 : β1, . . . , β4 real} = {β1c1 + β3c

3 : β1, β3 real}So, a parametric representation of F using the smallest number of

parameters is {a1 +β1c1 +β3c

3 : β1, β3 take all real values. This impliesthat F = affine hull of {a1, a1 + c1, a1 + c3 } = affine hull of {a1, a2, a4 }.

Example 6: Find a linear equation representation for the affinehull of {a1 = (−3, 4, 0,−6)T , a2 = (−1, 3,−3,−4)T} which is a straightline L in R4.

322 Ch. 3. Geometry of R n

So, L = {x = (x1, x2, x3, x4)T = (−3, 4, 0,−6)T + α(2,−1,−3, 2)T :

α takes all real values}.To get the linear equation representation of L we apply the algo-

rithm using GJ pivot steps and the terminology of the previous exam-ples.

BP x1 x2 x3 x4 α1 0 0 0 −2 −3

0 1 0 0 1 40 0 1 0 3 00 0 0 1 −2 61 2 0 0 0 5

α 0 1 0 0 1 40 −3 1 0 0 −120 2 0 1 0 14

Now the only parameter α in this representation, is eliminated fromrows 1, 3, and 4. The equations corresponding to these rows form thefollowing system of three equstions in four unknowns, which providesa linear equation representation for the straight line L.

x1 + 2x2 = 5

−3x2 + x3 = −12

2x2 + x4 = 14

Example 7: Find a linear constraint representation for thehalf-line L+ = {x = (−3, 4, 0,−6)T + α(2,−1,−3, 2)T : α ≥ 0}.

Let L be the straight line containing L+. In Example 6 above wefound the linear equation representation for L.

But in L+ we have the restriction α ≥ 0. From the 2nd row in thefinal tableau in Example 6 we get the equation x2+α = 4, or α = 4−x2

expressing the parameter α in terms of the variables xj on L. So therestriction α ≥ 0 is equivalent to the inequality constraint 4 − x2 ≥ 0or x2 ≤ 4. When we augment the linear equation representation for L

3.11. Affine Hulls 323

with this additional constraint, we get the following system which isthe linear constraint representation for L+.

x1 + 2x2 = 5

−3x2 + x3 = −12

2x2 + x4 = 14

x2 ≤ 4

Exercises

3.11.1 Let A.1 to A.5 be the column vectors of A given below.(a) (Yes) Are b1, b2 in the affine hull of {A.1, . . . , A.5}? If so, find

the actual affine combination for each of them.(b) (No) Are b3, b4 in the affine hull of {A.1, . . . , A.5}?

A =

1 −1 1 0 −1−1 2 2 2 1−1 4 8 6 1

0 2 3 2 −1

, b1 =

5−5−5

3

, b2 =

(4 −3 −1 2

),

b3 =(

3 2 12 −8), b4 =

(−4 −2 −14 10

)

3.11.2: Find the linear equation representation of the affine hull ofthe set of row vectors of the following matrices.

(a)

−3 1 −4 5−7 2 2 −1

4 −1 −6 6−13 4 −6 9−17 5 0 3

, (b)

2 3 3 212 18 18 128 12 12 8

10 15 15 10

3.11.3: Obtain the linear equation representations of the follow-ing straight lines given in parametric form: {x = (−3, 7, 9,−13)T +α(−2,−7, 1,−2)T : α real}, {x = (2, 3) + α(1,−2) : α real}.

324 Ch. 3. Geometry of Rn

3.12 Convex Combinations, Convex Hulls

Let a1, a2 be two distinct points in Rn. A convex combination ofa1, a2 is

α1a1 + α2a

2

where the coefficients α1, α2 have to satisfy the convex combinationconditions

α1 + α2 = 1, and α1, α2 ≥ 0

The set of all convex combinations of {a1, a2} is called the convexhull of {a1, a2}, and it is the line segment joining a1 and a2. Itsparametric representation is

{x = α1a1 + α2a

2 : α1 + α2 = 1, α1, α2 ≥ 0}or {x = αa1 + (1 − α)a2 : 0 ≤ α ≤ 1}.

x -axis1

2x -axis

.

..

0...

.

.

1 2

1

2

-1

-2

-1-2

.3

.3

(2, 0)

(0, 2)

Figure 3.17: The convex hull of any two points in any space is the linesegment joining them.

3.12. Convex Hulls 325

Example 1: Let

a1 =

2222

, a2 =

6666

, a3 =

3333

, a4 =

10101010

, a5 =

3425

.

To check if a3 is in the line segment joining a1, a2 we solve thefollowing equations for α.

2α + 6(1 − α)2α + 6(1 − α)2α + 6(1 − α)2α + 6(1 − α)

=

3333

which yields α = 0.25. Since 0 ≤ 0.25 ≤ 1, we conclude that a3 is inthe line segment joining a1, a2.

To check if a4 is in the line segment joining a1, a2 we solve

2α + 6(1 − α)2α + 6(1 − α)2α + 6(1 − α)2α + 6(1 − α)

=

10101010

which yields α = −1. Since −1 is not between 0 and 1, we concludethat a4 is on the straight line joining a1, a2 but not on the line segmentjoining them.

To check if a5 is in the line segment joining a1, a2 we solve

2α + 6(1 − α)2α + 6(1 − α)2α + 6(1 − α)2α + 6(1 − α)

=

3425

which has no solution. So, a5 is not even on the straight line joininga1, a2.

326 Ch. 3. Geometry of R n

Midpoint, and the Two Orientations of a LineSegment

Let a1, a2 be two distinct column vectors in R n, and also geomet-

rically the two points corresponding to them in n-dimensional space.The line segment joining a1, a2 is the portion of the straight line join-ing a1, a2 lying between the points a1, a2. It is also the convex hull of{a1, a2 }.

The midpoint of this line segment joining a1, a2 is the convex com-bination

12a1 +

12a2.

The line segment joining a1, a2 can be oriented in two different ways.

One orientation is obtained by treating a1 as the starting point, andthen move towards a2. When viewed this way, the line segment becomesa portion of the half-line starting at a1 and moving towards a2 andcontinuing in the same direction. In this orientation, the parametricrepresentation of the line segment is {a1 + α(a2 − a1) : 0 ≤ α ≤ 1}.In this representation α = 0 corresponds to the starting point a1; andα = 1 gives a2, the end point in the line segment in this orientation.a1 + α(a2 − a1) is the point on the line segment that is a fraction αaway from a1 towards a2 for 0 ≤ α ≤ 1. Verify that α = 1/2 gives themidpoint of the line segment.

The 2nd orientation is obtained by starting at a2 and moving to-wards a1, under this orientation the representation of the line segmentis {a2 +α(a1 −a2) : 0 ≤ α ≤ 1}; a2 +α(a1 −a2) is the point on the linesegment that is a fraction α away from a2 towards a1 for 0 ≤ α ≤ 1.

Example 2: Let a1 = (1, 0,−2, 3)T , a2 = (−2, 2, 1,−7)T . So,a2 − a1 = (−3, 2, 3,−10)T , and a1 − a2 = (3,−2,−3, 10)T .

The midpoint of the line segment joining a1, a2 is 12a1 + 1

2a2 =

(−1/2, 1,−1/2,−2)T .

The orientation of this line segment from a1 towards a2 is {(1 −3α, 2α,−2 + 3α, 3 − 10α)T : 0 ≤ α ≤ 1}. Taking α = 2/3, we see that(−1, 4/3, 0,−11/3)T is the point on this line segment that is two-thirdsof the way away from a1 towards a2.

The other orientation of this line segment is {(−2 + 3α, 2− 2α, 1−3α,−7 + 10α)T : 0 ≤ α ≤ 1}.

3.12. Convex Hulls 327

Convex Combinations of a General set of k Points

Let {a1, . . . , ak} be a set of k column vectors in Rn where k is finite.A convex combination of {a1, . . . , ak} is

α1a1 + . . . + αka

k

where the coefficients α1, . . . , αk have to satisfy the convex combina-tion conditions

α1 + . . . + αk = 1, and all of α1, . . . , αk ≥ 0

The set of all convex combinations of {a1, . . . , ak} is called the con-vex hull of {a1, . . . , ak}, it is {x = α1a

1 + . . . + αkak : α1 + . . . + αk =

1, α1, . . . , αk ≥ 0}, it is also known as a convex polytope or abounded convex polyhedron. So, the convex hull of a set of pointsis a subset of its affine hull, which is itself a subset of its linear hull.

How to Check if ak +1 Is In the Convex Hull of{a1, . . . , ak }?

ak+1 is in the convex hull of {a1, . . . , ak} iff the following system oflinear constraints

α1a1 + . . . + αka

k = ak+1

α1 + . . . + αk = 1

α1, . . . , αk ≥ 0

in the parameters α1, . . . , αk, has a feasible solution. Since this systeminvolves inequality constraints (nonnegativity constraints on the vari-ables), to solve it we need to use linear programming techniques, whichare outside the scope of this book. We can ignore the nonnegativityconstraints on the variables, and solve the remaining system which isa system of equations, by the GJ pivotal method, or the G eliminationmethod. This could terminate in 3 possible ways discussed below.

328 Ch. 3. Geometry of Rn

x -axis1

2x -axis

.

..

0...

.

.

.

..

..

.

.

.

Figure 3.18: A convex polytope, the convex hull of its 6 corner pointsin R2.

1. It may yield a solution which satisfies the nonnegativity con-straints. In this case we conclude that ak+1 is in the convex hullof {a1, . . . , ak}.

2. It may conclude that the system of equations itself is inconsistent,i.e., has no solution. In this case we conclude that ak+1 is not inthe convex hull of {a1, . . . , ak}.

3. The method may terminate with a solution to the system of equa-tions α = (α1, . . . , αk) where some αt < 0. In this case, if thissolution is the unique solution of the system of equations, we areable to conclude that the system of equations has no nonnega-tive solution, and therefore that ak+1 is not in the convex hull of{a1, . . . , ak}. On the other hand if the system of equations hasmultiple solutions, this method has been inconclusive. To resolvethe question, we need to check whether there is a nonnegative

3.12. Convex Hulls 329

solution to the system for which we need linear programmingtechniques.

Example 3: Let

a1 =

10001

, a2 =

01001

, a3 =

1−1

1−1

1

, a4 =

2−4

3−3

1

, a5 =

3/4−1/4

1/2−1/2

1

.

To check if a4 is in the convex hull of {a1, a2, a3} we solve the follow-ing system of equations (after ignoring the nonnegativity constraints)by the GJ pivotal method. PR (pivot row), PC (pivot column), RC(redundant constraint to be deleted), BV (basic variable selected inrow) have the same meanings as before, and the pivot element is boxedin each step.

BV α1 α2 α3

1 0 1 2 PR0 1 −1 −40 0 1 30 0 −1 −31 1 1 1

PC↑BV α1 α2 α3

α1 1 0 1 2

0 1 −1 −4 PR0 0 1 30 0 −1 −30 1 0 −1

PC↑

330 Ch. 3. Geometry of Rn

BV α1 α2 α3

α1 1 0 1 2α2 0 1 −1 −4

0 0 1 3 PR0 0 −1 −30 0 1 3

PC↑α1 1 0 0 −1α2 0 1 0 −1α3 0 0 1 3

0 0 0 0 RC0 0 0 0 RC

The system of equations in this case has a unique solution (α1, α2, α3) =(−1,−1, 3) which violates the nonnegativity constraints. So, we areable to conclude that the system has no nonnegative solution, andtherefore a4 is not in the convex hull of {a1, a2, a3}.

To check whether a5 is in the convex hull of {a1, a2, a3}, we applythe GJ method on the following system of equations.

BV α1 α2 α3

1 0 1 3/4 PR0 1 −1 −1/40 0 1 1/20 0 −1 −1/21 1 1 1

PC↑α1 1 0 1 3/4

0 1 −1 −1/4 PR0 0 1 1/20 0 −1 −1/20 1 0 1/4

PC↑

3.12. Convex Hulls 331

BV α1 α2 α3

α1 1 0 1 3/4α2 0 1 −1/4 −4

0 0 1 1/2 PR0 0 −1 −1/20 0 1 1/2

PC↑α1 1 0 0 1/4α2 0 1 0 1/4α3 0 0 1 1/2

0 0 0 0 RC0 0 0 0 RC

So, the method yields the solution (α1, α2, α3) = (1/4, 1/4, 1/2)which satisfies the nonnegativity constraints. Hence a5 is in the convexhull of {a1, a2, a3 }. In fact a5 = (1/4)a1 + (1/4)a2 + (1/2)a3.

How to Represent the Convex Hull of a Given Setof Points Through a System of Linear Constraints?

The convex hull of a given set of points {a1, . . . , ak } in R n can be

represented through a system of linear constraints, but this systemwould involve some linear inequalities. Typically the number of linearinequalities in this representation grows exponentially with k. Deriv-ing these inequalities requires linear programming techniques which areoutside the scope of this book. However, as the number of the inequali-ties in the representation is typically very very large, even though linearprogramming techniques are available to generate them, those tech-niques are rarely used directly in their entirety in practice. However,implementations of these techniques which generate these constraintsone at a time are of great importance in the area of CombinatorialOptimization.

Exercises

3.12.1: Check whether the following vectors are in the convex hull

332 Ch. 3. Geometry of R n

of the set of column vectors of the matrix A given below: (a) (Yes)b1, b2; (b) (No) b3, b4.

A =

3 0 96 −3 09 9 −9

12 −6 3

, b1 =

4139

, b2 =

(5 4 3 9

),

b3 =(

15 9 18 21), b4 =

(6 15 9 30

)

3.12.2: For each pair of points given below, find: (i) the midpointof the line segment joining them, (ii) parametric representation of theline segment oriented with each point in the pair as the starting point,and directed towards the other point in the pair, and (iii) the point onthe line segment that is two-thirds of the way from the starting pointtowards the other end point in each of the parametric representationsabove.

{(1,−1, 0, 2)T , (−3, 4,−7,−9)T }; {(0,−3, 4,−6, 7)T , (6, 0, 3,−4,−2)T };{(1,−3,−1)T , (−2,−1,−4)T }.

3.13 Nonnegative Combinations, Pos Cones

A nonnegative linear combination of a given set of vectors {a1, . . . , ak }in R

n is a vector of the form

α1a1 + . . . + αka

k where all the coefficients α1, . . . , αk ≥ 0

The set of all nonnegative linear combinations of {a1, . . . , ak } is{x = α1a

1 + . . .+αkak : α1, . . . , αk ≥ 0}. It is called the nonnegative

hull or the pos cone of {a1, . . . , ak } and is denoted by Pos{a1, . . . , ak }.It is a cone.

Example 1: Here we show how to draw a pos cone in R 2. Let

Γ =

{a1 =

(12

), a2 =

(2

−3

), a3 =

(10

), a4 =

(1

−1

)}

3.13. Pos Cones 333

To draw Pos(Γ), plot each of the points in Γ, and draw the ray ofeach of these points (the half-line joining the origin to the point andcontinuing in that direction indefinitely). The Pos cone of Γ is thesmallest angle at the origin containing all these rays. See Figure 3.19.

In higher dimensional spaces, even though we cannot physicallydraw the pos cone of a set of points, we can visualize it as the smallestsolid angle at the origin containing the rays of all the points in the set.

To check whether a point ak+1 is in Pos{a1, . . . , ak}, we need tosolve the following system of equations in nonnegative variables.

α1a1 + . . . + αka

k = ak+1

α1, . . . , αk ≥ 0

It is not always possible to get a feasible solution to this system, orconclude that it has no feasible solution, using the GJ pivotal methodor the G elimination method applied to the equality constraints only,ignoring the nonegativity requirements. To conclusively solve this sys-tem, linear programming techniques which are outside the scope of thisbook are needed.

Pos{a1, . . . , ak} can be represented through a system of linear con-straints. It leads to a homogeneous system of linear inequalitiesof the form Ax ≥ 0. To derive the constraints in this representationrequires linear programming techniques not discussed in this book. Un-fortunately, the number of constraints in such a representation typicallygrows exponentially with k, so this linear constraint representation israrely used in practice in its entirety, but implementations of thesetechniques that generate these constraints one at a time are of greatimportance in the area of Combinatorial Optimization.

Summary of Various Types of Combinations of a

Set of Vectors Γ = {a1, . . . , ak } ⊂ R n

Linear combination of Γ: Any point of the form α1a1 + . . . + αka

k

where α1, . . . , αk can take all possible real values. Set of all these

334 Ch. 3. Geometry of Rn

x -axis1

2x -axis

.

..

0...

.

.

1 2

1

2

-1

-2

-1-2

.3

.3

.

.

.

.-3

(1, 2)

(1, 0)

(1, -1)

(2, -3)

Figure 3.19: The cone Pos(Γ) in R2 is the smallest angle at the origincontaining the rays of all the points in Γ.

combinations is the linear hull of Γ. It is a subspace of Rn. If itis not Rn, it is the set of solutions of a homogeneous system oflinear equations of the form Ax = 0 where A is of order m × n,with 1 ≤ m ≤ n. Given Γ, this linear constraint representationof its linear hull can be found efficiently.

Affine combination of Γ: Any point of the form α1a1 + . . . + αka

k

where α1 + . . .+αk = 1. Set of all these combinations is the affinehull of Γ, it is the translate of a subspace of Rn. If it is not Rn,it is the set of solutions of a nonhomogeneous system of linearequations of the form Ax = b where A is of order m × n, with1 ≤ m ≤ n. Given Γ, this linear constraint representation of itsaffine hull can be found efficiently.

Convex combination of Γ: Any point of the form α1a1 + . . . +αka

k

where α1 + . . .+αk = 1, and α1, . . . , αk ≥ 0. Set of all these com-

3.14. Hyperplanes 335

binations is the convex hull of Γ. It is a convex polytope in Rn.It is the set of feasible solutions of a system of nonhomogeneouslinear inequalities of the form Ax ≥ b where A is of order m×n,with m typically growing exponentially in k. Given Γ, this linearconstraint representation of its convex hull can be derived usinglinear programming techniques.

Nonnegative linear combination of Γ: Any point of the form α1a1+

. . . + αkak where α1, . . . , αk ≥ 0. Set of all these combinations is

the pos cone of Γ. It is a cone in Rn. If it is not Rn, it is the setof solutions of a homogeneous system of linear inequalities of theform Ax ≥ 0.

3.14 Hyperplanes

For n ≥ 2, a hyperplane in Rn is defined to be the set of all vectorsx = (x1, . . . , xn)T which satisfy a single nontrivial linear equation,i.e., one of the form

a1x1 + . . . + anxn = b1 where (a1, . . . , an) �= 0 (b1 could be 0)

So, a nontrivial linear equation is one in which at least one of thevariables appears with a nonzero coefficient. In Rn, each hyperplanehas dimension n − 1. So, in R2 each straight line is a hyperplane andvice versa.

We have seen earlier the parametric representation for straight linesin Rn for any n ≥ 2, which is the most convenient way for representingstraight lines.

An alternate representation for a straight line in R2 only, is througha single linear equation. This works in R2 because every straight line inR2 is a hyperplane and vice versa. For n ≥ 3, in Rn a hyperplane hasdimension n− 1 ≥ 2, and is not a straight line. That’s why a singlelinear equation does not represent a straight line in spaces ofdimension ≥ 3.

We will now include figures of some hyperplanes in R2 and R3.Figure 3.21 to 3.23: In R3 the three coordinate hyperplanes are

given by “x3 = 0” (contains the x1 and x2 coordinate axes, Figure 3.21),

336 Ch. 3. Geometry of Rn

“x1 = 0” (contains the x2 and x3 coordinate axes, Figure 3.22), and“x2 = 0” (contains the x1 and x3 coordinate axes). You can visualizethese coordinate hyperplanes from these figures. We show a portion(shaded) of the hyperplane corresponding to “x1 + x2 + x3 = 1” inFigure 3.23.

x -axis1

2x -axis

.

..

0...

.

.

1 2

1

2

-1

-2

-1-2

.3

.3

(1, 0)(0, 1)

x = 02

Figure 3.20: In R2, “x2 = 0” (x1-axis), “x1 = 0” (x2-axis) are the twocoordinate hyperplanes. We also show the hyperplane (straight line)represented by “x1 + x2 = 1.

How to Draw the Straight Line Correspondingto an Equation in R 2?

Set up the x1, x2− coordinate axes and select a scale for measuringlengths on each axes.

If the coefficient of x2 in the equation is 0, then the equation isequivalent to “x1 = b” for some b. This represents the straight lineparallel to the x2−axis through the point (b, 0)T on the x1−axis.

If the coefficient of x1 in the equation is 0, then the equation isequivalent to “x2 = b” for some b. This represents the straight lineparallel to the x1−axis through the point (0, b)T on the x2−axis.

3.14. Hyperplanes 337

If the coefficients of both x1 and x2 in the equation are nonzero,find two points on the straight line represented by it by giving one ofthe variables two different values and finding the corresponding valuesof the other variable from the equation. For example, put x1 = 0, 1and let the value of x2 from the equation be respectively b0, b1. Then

x1

x2

x3

0

.

.(1, 0, 0)

(0, 1, 0)

x = 03

Figure 3.21: The x1, x2-coordinate hyperplane represented by the equa-tion x3 = 0 in R3.

(0, b0)T , (1, b1)

T are points on the straight line represented by it. Jointhe two points (0, b0)

T , (1, b1)T by a straight line, which is the desired

one.In Figure 3.24 given below, we show the straight lines in R2 cor-

responding to “x1 = 1.5” (dotted line), “x2 = 2” (dashed line), and“x1 + 2x2 = 3” (solid line). For the last one, we put “x1 = 1, 3, and

338 Ch. 3. Geometry of Rn

x1

x2

x3

0

.

.(1, 0, 0)

(0, 1, 0)

.(0, 0, 1) x = 01

Figure 3.22: The x2, x3-coordinate hyperplane represented by the equa-tion x1 = 0 in R

3.

find from the equation x2 = 1, 0 respectively; so (1, 1)T , (3, 0)T are twopoints on that line.

Exercises

3.14.1 Draw the straight lines in R2 corresponding to each of thefollowing equations.

(i) −3x1 = −9 (iii) 4x2 = 12 (v) 3x1 − x2 = 9(ii) x1 + 2x2 = 5

3.15 Halfspaces

For any n ≥ 2, a hyperplane in Rn divides the space into two halfspaces,each of which consists of all the points on one side of the hyperplane.

3.15 Halfspaces 339

x1

x2

x3

0

.

.(1, 0, 0)

(0, 1, 0)

.(0, 0, 1)

Figure 3.23: Hyperplane represented by x1 + x2 + x3 = 1 in R3.

Look at the hyperplane defined by the equation “x1 + x2 = 2” in R2

in Figure 3.25, it divides R2 into two halfspaces, one above it (thelined side), one below it. Points in the halfspace above this line satisfythe constraint “x1 + x2 ≥ 2”. Points in the halfspace below this line,containing the origin 0, satisfy the constraint “x1 + x2 ≤ 2”.

In any space a straight line always has dimension 1. One can verifythat a straight line in R3 does not divide R3 into two halfspaces becauseit does not have enough dimension. To divide R3 into two halfspacesyou need a plane of dimension 2.

In the same way, a hyperplane in Rn represented by the equation

a1x1 + . . . + anxn = a0

where (a1, . . . , an) �= 0 divides the space into two halfspaces, one ofwhich is

{x : a1x1 + . . . + anxn ≥ a0}and the other is

{x : a1x1 + . . . + anxn ≤ a0}.

340 Ch. 3. Geometry of Rn

x -axis1

2x -axis

.

..

0...

.

.

1 2

1

2

-1

-2

-1-2

.3

..

3

x = 22

(1, 1)

(3, 0)

Figure 3.24: The straight lines in R2 represented by the equationsx2 = 2, x1 = 1.5, and x1 + 2x2 = 3, and the halfspace represented byx1 + 2x2 ≥ 3.

The hyperplane represented by the equation “a1x1 + . . . + anxn =a0” is itself common to both these halfspaces. The above inequalitiesare said to be closed linear inequalities (because they both allowthe possibility of the two sides of the inequality being equal), and thehalfspaces represented by them are called closed halfspaces. Eachclosed halfspace includes the hyperplane defining it. The intersectionof the two halfspaces given above is the hyperplane defining them. Thesingle equation a1x1 + . . . + anxn = a0 is equivalent to the pair ofinequalities

a1x1 + . . . + anxn ≥ a0

a1x1 + . . . + anxn ≤ a0

This explains the fact that a hyperplane in Rn is the intesection ofthe two closed halfspaces defined by it.

3.15 Halfspaces 341

x -axis1

2x -axis

.

..

0...

.

.

1 2

1

2

-1

-2

-1-2

.3

.3

(2, 0)

(0, 2) x + x > 221 =x + x < 21 2 =

Figure 3.25: A hyperplane in R2 (Straight line) divides the space intotwo halfspaces. The thick straight line represents “x1 + x2 = 2”. Theinequality “x1 + x2 ≥ 2” represents the halfspace of all points in R2

lying above (i.e., on the marked side of) this line.

The strict inequalities

a1x1 + . . . + anxn > a0 or a1x1 + . . . + anxn < a0

are said to be open linear inequalities, they represent open half-spaces. An open halfspace does not contain the hyperplane definingit.

Thus each halfspace in Rn is the set of all solutions to a singlenontrivial linear inequality (i.e., one in which the coefficient of at leastone variable is nonzero).

342 Ch. 3. Geometry of R n

How to Draw the Halfspace Defined by a LinearInequality in Two Variables ?

Consider the inequality

a1x1 + a2x2 ≥ a0

where (a1, a2) �= 0. To draw the halfspace corresponding to it in R 2,

set up the Cartesian system of coordinates and plot the straight linecorresponding to a1x1 + a2x2+ = a0 as described earlier. Take anypoint x = (x1, x2)

T in one of the open halfspaces into which thisstraight line divides R

2. If a1x1+a2x2 > a0, the halfspace containingx is the one corresponding to a1x1 + a2x2 ≥ a0. On the other handif a1x1 + a2x2 < a0, the other halfspace not containing x is theone corresponding to a1x1 + a2x2 ≥ a0.

Example 1: In Figure 3.26 we draw the halfspaces correspondingto the four inequalities given below and mark each of them by an arrowon the straight line defining it.

x1 ≥ 0

x2 ≥ 0

x1 − x2 ≥ −2

−x1 + x2 ≤ −2

3.16 Convex Polyhedra

A convex polyhedron in Rn is the intersection of a finite number ofhalfspaces, i.e., it is the set of feasible solutions of a system of linearinequalities of the form

Ax ≥ b

3.16 Convex Polyhedra 343

x -axis1

2x -axis

.

..

0...

.

.

1 2

1

2

-1

-2

-1-2

.3

.3

x = 02

Figure 3.26: Four halfspaces in R 2, and the unbounded convex polyhe-

dron that is their intersection.

where A is a matrix of order m × n. Each inequality in this systemdefines a halfspace in R

n; and the set of feasible solutions of this systemis the intersection of all these halfspaces.

Example 1: In Figure 3.26 we plot the convex polyhedron inR

2 (shaded region) that is the set of feasible solutions of the followingsystem of constraints

x1 ≥ 0

x2 ≥ 0

x1 − x2 ≥ −2

−x1 + x2 ≤ −2

Example 2: In Figure 3.27 we plot the convex polyhedron in R 2

344 Ch. 3. Geometry of Rn

corresponding to the following system of constraints

x1 + x2 ≤ 2

−x1 + x2 ≥ −2

−x1 − x2 ≤ 2

x1 − x2 ≥ −2

A bounded convex polyhedron is called a convex polytope.

The set of feasible solutions of a homogeneous system of linear in-equalities, i.e., a system of the form

Ax ≥ 0

is said to be a convex polyhedral cone. As an example, considerthe system

x -axis1

2x -axis

.

..

0...

.

.

1 2

1

2

-1

-2

-1-2

.3

.3

Figure 3.27: A convex polytope (bounded convex polyhedron) in R2.

3.16 Convex Polyhedra 345

2x1 + x2 ≥ 0

x1 + 3x2 ≥ 0

Its set of feasible solutions is the polyhedral cone (marked by anangle sign) in Figure 3.28.

x -axis1

2x -axis

.

..

0...

.

.

1 2

1

2

-1

-2

-1-2

.3

.3

.

.

(-1, 2)

(3, -1)

Figure 3.28: A polyhedral cone in R 2.

Ex ercis es

3.16.1: Set up the 2-dimensional Cartesian coordinate system anddraw the convex polyhedra represented by the following systems of con-straints. Mention whether each of them is a polytope or an unboundedpolyhedron.

(i): (Unbounded) (a) −5 ≤ −x1+x2 ≤ 4, x1+x2 ≥ 3, x1+2x2 ≥ 4,x1, x2 ≥ 0.

346 Ch. 3. Geometry of Rn

(b) x1, x2 ≥ 0, x1 ≤ 50, x2 ≤ 80, −2x1 + x2 ≤ 60, 3x1 + x2 ≥ 30,x1 + 3x2 ≥ 30.

(c) −2x1 + x2 ≤ 0, x1 − 2x2 ≤ 0.

(ii): (Bounded) (d) −4 ≤ x1 ≤ 4, −4 ≤ x2 ≤ 4, −6 ≤ −x1+x2 ≤ 6,−6 ≤ x1 + x2 ≤ 6.

(e) x1, x2 ≥ 0, x1 + x2 ≥ 10, x1 + 4x2 ≤ −20.

3.17 Convex and Nonconvex Sets

A subset K ⊂ Rn is said to be a convex set if it satisfies the followingcondition

for every pair of points x1, x2 ∈ K;

αx1 + (1 − α)x2 ∈ K for all 0 ≤ α ≤ 1

Figure 3.29: Convex sets

3.18 Distance and Angles 347

i.e., for every pair of points x1, x2 in K, the entire line segment joiningx1, x2 must be in K.

Figure 3.29 has examples of convex sets in R 2; and Figure 3.30 has

examples of nonconvex sets.

Figure 3.30: Nonconvex sets

Some examples of convex sets in R n are

all hyperplanes, and halfspaces, convex polyhedra, convexpolyhedral cones, and balls.

3.18 Euclidean Distance, and Angles

The Euclidean Dis tance Between Two Point s i n Rn

Consider the point P = (x1, x2)T in R2. Let Q1 = (x1, 0)T . Then

from Figure 3.31 we see that 0, Q1, P form a right angled triangle. Thefamous Pythagoras theorem states that in any right angled trianglethe square of the diagonal is the sum of the squares of the two sides.

348 Ch. 3. Geometry of Rn

900

x -axis1

2x -axis

.

.

0

P = (x , x )1 2

Q = (x , 0)11

x2

x 1

Figure 3.31: Euclidean distance function in R2 explained

Applying this theorem we see that

distance between 0 and P = (x1, x2)T in R2 =

√x2

1 + x22

Now consider the point P = (x1, x2, x3)T in R3, see Figure 3.32.

Let the point Q2 = (x1, x2, 0)T in the x1, x2− coordinate plane for R3.From the above argument we know that 0Q2 = the length of the line

segment 0Q2 is√

x21 + x2

2. Also, 0, Q2, P form a right angled triangle

in the two dimensional plane of R3 containing the rays of P and Q2.Applying the Pythagoras theorem to this right angled triangle, we have

distance between 0 and P = (x1, x2, x3)T in R3 =

√(0Q2)2 + x2

3

=√

x21 + x2

2 + x23

Continuing this argument the same way, we get the following defi-nition of distance in Rn

3.18 Distance and Angles 349

900

x -axis1

2x -axis

3x -axis

(x , x , x )1 2 3

.

.

.

0

P = (x , x , x )1 2 3

Q = (x , x , 0)1 22

Figure 3.32: Eoclidean distance function in R3 explained.

distance between 0 and P = (x1, . . . , xn)T in Rn =

√x2

1 + . . . + x2n =

√xT x

The distance between 0 and x ∈ Rn given above is usually denotedby the symbol ||x||, and is known as the Euclidean norm of thevector x.

To find the distance between two points L = (y1, . . . , yn)T andP = (x1, . . . , xn)T in Rn, we translate the origin to L, this has noeffect on distances. After the translation L becomes 0 and P becomes(x1 − y1, . . . , xn − yn)T . From this and the above, we conclude that

distance between x = (x1, . . . , xn)T and y = (y1, . . . , yn)T in Rn =

√(x1 − y1)2 + . . . + (xn − yn)2 =

√(x − y)T (x − y)

350 Ch. 3. Geometry of Rn

The distance function given above is known as the Euclidean dis-tance. This is the most commonly used distance measure.

The Euclidean norm satisfies the following properties:

(i) ||x|| ≥ 0 for all x ∈ Rn; and ||x|| = 0 only if x = 0.

(ii) ||λx|| = |λ|||x|| for all x ∈ Rn, and all real numbers λ.

(iii) Triangle inequality: For all x, y ∈ Rn, ||x + y|| ≤ ||x|| + ||y||.(iv) Cauchy-Schwartz inequality: For all x, y ∈ Rn, | < x, y > |

≤ (||x||)(||y||) where < x, y > is the dot product of the twovectors x, y defined in Section 1.5 (if x, y are both column vectors,< x, y >= xT y). Also, | < x, y > | = (||x||)(||y||) iff one of thetwo vectors among x, y is a scalar multiple of the other.

Using the parallelogram law of addition of vectors, it can be seenthat the triangle inequality is the same as the geometric fact that forany n ≥ 2, the sum of the lengths of two sides of a two dimensionaltriangle in Rn is ≥ the length of the third side. To see this let x, ybe two nonzero vectors in Rn. Figure 3.33 shows the two dimensionalsubspace of Rn containing the rays of x and y. Look at the trianglewith vertices 0, x and x+y. The lengths of the sides of this triangle are||x|| (side joining 0 and x), ||y|| (side joining x, x+y), and ||x+y|| (sidejoining 0 and x+y); and so the triangle inequality ||x||+ ||y|| ≥ ||x+y||is an expression of this geometric fact.

Mathematicians have constructed many other functions which canbe used as distance functions (these distance functions do not satisfythe Pythagoras theorem), and they study properties of Rn under dif-ferent distance measures. In this book we will only use the Euclideandistance function defined above.

Relationship Between the Dot Pro duct o f T woVectors and t he Angle B etween their R ay s

Let x, y be two nonzero vectors in Rn. First assume that neither ofthese vectors is a scalar multiple of the other. Then the rays of x, y

3.18 Distance and Angles 351

x -axis1

2x -axis

.

.

.

0x

y

x + y

||x||

||y||||x + y||

Figure 3.33: Interpretation of triangle inequality using the parallelo-gram law of addition of vectors.

are contained in a two dimensional subspace of Rn which is the linearhull of {x, y}. In this subspace let θ denote the angle between the raysof x, y. See Figure 3.34. The three points 0, x, y form a triangle. Thelengths of the sides of this triangle are: ||x|| (side joining 0 and x), ||y||(side joining 0 and y), and ||x − y|| (side joining x and y).

The Law of cosines of Plane Trignometry, is:

Square of the length of one side of a triangle is = sum of thesquares of the lengths of the other two sides − two timestheir product multiplied by the cosine of the angle betweenthem.

Applying this to our triangle, we have

||x − y||2 = ||x||2 + ||y||2 − 2(||x||)(||y||) cos(θ)

352 Ch. 3. Geometry of Rn

x -axis1

2x -axis

.

.

0

x

y

||x||

||y||

||x - y||

Figure 3.34: Angle between the rays of two nonzero vectors in Rn.

But ||x − y||2 = (x − y)T (x − y) = xT x + yT y − 2 < x, y >.Substituting this in the above equation, we get

cos(θ) = < x, y > /((||x||)(||y||))If x is a positive multiple of y (see Figure 3.35), then the rays of

x and y are the same, so the angle between these rays is 0, and sincecos(0) = 1, the above equation holds.

If x is a negative multiple of y (see Figure 3.36), then the rays ofx, y are the two halfs of a straight line, i.e., the angle θ between theserays is 1800 (or π if measured in radians) and since cos(1800) = −1,the above equation holds in this case too. Hence we have

For any pair of nonzero vectors x, y ∈ Rn, the cos of theangle between their rays is = the dot product of x and ydivided by the product of the norms of x and y.

For any pair of nonzero vectors x, y ∈ Rn, the angle between theirrays is also known as the angle between the nonzero vectors x, y.

A pair of nonzero vectors x, y ∈ Rn are said to be orthogonal

3.19 Volume and Determinants 353

x -axis1

2x -axis

.

.

0

x

y

Figure 3.35: Angle between the rays of x, y is 0 when x is a positivemultiple of y.

if the angle between them is 90 0 (or π/2 when measured in radians).

Hence x, y ∈ R n are orthogonal if they are both nonzero and if their

dot product is 0. See Figure 3.37.

Ex ercis es

3.18.1: Find the angle between the rays of the following pairs ofpoints in the space of appropriate dimension: {(1,−1, 1, 2), (0, 1,−2,−2)},{(1, 2,−3), (−2, 0, 4)}, {(1, 0,−3), (0,−2, 0)}. Which of these pairs isorthogonal?

3.19 The Determinant Interpreted As A

Volume Function

Consider a matrix B of order 2 × 2

354 Ch. 3. Geometry of Rn

x -axis1

2x -axis

.

.

0

x

y

Figure 3.36: Angle between the rays of x, y is 1800 when x is a negativemultiple of y.

B =

(b11 b12

b21 b22

)

and let B1., B2. be its row vectors (we can carry this discussion in termsof column vectors too, we are doing it in terms of row vectors). Let

f(B1., B2.) = area of the parallelogram with 0, B1., B2. as three ofits vertices in R2, see Figure 3.38.

Then from geometry, we know that the area of a parallelogram isthe product of the length of its base and height. From this, we verifythat the function f(B1., B2.) satisfies the following properties:

(a) f((1, 0), (0, 1)) = 1, since by definition, the area of the unit squareis 1.

3.19 Volume and Determinants 355

x -axis1

2x -axis

.

.

0

x

y

Figure 3.37: A pair of orthogonal vectors x, y.

(b) f(λB1., B2.) = f(B1., λB2.) = λf(B1., B2.) for all λ ≥ 0. SeeFigure 3.38 where we illustrate this for λ = 2, 3.

(c) f(B1. + B2., B2.) = f(B1., B2.) = f(B1., B2. + B1.). See Figure 3.39where this property is illustrated.

It turns out that there is a unique real valued function of the rowvectors B1., B2. that satisfies the three properties (a), (b), (c) mentionedabove, it is det(B) = the determinant of the matrix B with B1., B2. asits row vectors. Also, the area function defined above f(B1., B2.) =|det(B)|.

In the same way let A = (aij) be a square matrix of order n and letA1., . . . , An. be its row vectors. Define

V (A1., . . . , An.) = n-dimensional volume (volume in Rn,also called content) of the parallelopiped P with 0 as avertex and A1., . . . , An. as its adjacent vertices in Rn (P ={x : x =

∑ni=1 λiAi., 0 ≤ λi ≤ 1 for all i})

356 Ch. 3. Geometry of Rn

..

.0

.

B1.

B2.

1.2B

1.3B

.

.

Figure 3.38: For nonzero x, y ∈ R2, f(x, y) is the area of the par-allelogram with 0, x, y as three of its vertices. You can verify thatf(2B1., B2.) = 2f(B1., B2.) and f(3B1., B2.) = 3f(B1., B2.).

This volume function V (A1., . . . , An.) ia always nonnegative, it sat-isfies V (A1., . . . , An.) = |F (A1., . . . , An.)| where the function F (A1., . . . , An.)satisfies the following three properties:

(i) F (I1., . . . , In.) = 1, where I1., . . . , In. are the row vectors of the unitmatrix of order n.

(ii) F (A1., . . . , Ai−1., λAi., Ai+1., . . . , An.) = λF (A1., . . . , An.) for all λreal, and for all i.

(iii) F (A1., . . . , Ai−1., Ai.+Aj., Ai+1., . . . , An.) = F (A1., . . . , An.) for alli and j �= i.

It can be proven that there is a unique real valued function of thevectors A1., . . . , An. which satisfies the above properties (i), (ii), (iii),

3.19 Volume and Determinants 357

..

.0

.

B1.

B2.

. B + B1. 2. B + 2B1. 2.

Figure 3.39: For nonzero x, y ∈ R 2, f(x, y) is the area of the par-

allelogram with 0, x, y as three of its vertices. You can verify thatf(B1. + B2., B2.) = f(B1., B2.).

and that function is the determinant of the matrix A with A1., . . . , An.

as its row vectors, i.e., det(A). See the book Curtis [3.1] for a mathe-matical proof of this statement. Thus the absolute value of the deter-minant of a square matrix of order n, is the n-dimensional volume ofthe parallelopiped with 0 as a vertex and the row vectors of the matrixas its adjacent vertices.

This partly explains the major role that determinants play in math-ematical research.

Ex ercis es

3.19.1: Euler’s tetrahedron problem: In R3 let 0, A = (x, y, z)T ,B = (x′, y′, z′)T , C = (x′′, y′′, z′′)T be the coordinates of a tetrahedron.Prove that its volume is given by ∆/6 where

358 Ch. 3. Geometry of Rn

∆ = det

x y zx′ y′ z′

x′′ y′′ z′′

.

3.20 Orthogonal Projections (Nearest Points)

The following problem arises in many applications.

We are given a subset S ⊂ Rn, usually as the set of feasiblesolutions of a given system of constraints, or sometimes inparametric representation; and a point x �= S. We arerequired to find the nearest point in S to x.

In general this leads to a quadratic programming problem, aspecial version of nonlinear programming. Both these topics areoutside the scope of this book. However, we will show the solutionto this problem when S is a simple set such as a straight line, or ahyperplane, or the set of solutions of a system of linear equations.

Nearest Point When S Is a Straight Line

Consider the case where S = L = {x : x = a + λc, λ takes all realvalues }, where a = (ai), c = (ci) are given column vectors in Rn andc �= 0. L is given in parametric representation. So, a + λc is a generalpoint on L, and the problem of finding the nearest point in L to x (notethat minimizing the distance is the same as minimizing its square) is:

Minimize f1(λ) = (a + λc − x)T (a + λc − x)

=n∑

i=1

(ai + λci − xi)2

over λ taking all real values. From calculus, we know that the minimumof f1(λ) is attained at a λ which makes the derivative f ′

1(λ) = 0, i.e.,satisfies

3.20 Orthogonal Projections 359

2n∑

i=1

(ai + λci − xi)ci = 0

Let λ∗ denote the solution of this. So,

λ∗ =

∑ni=1(xi − ai)ci∑n

i=1 c2i

=cT (x − a)

||c||2So, the nearest point in L to x is a+λ∗c = x∗ say. It can be verified

that for any point x = a + λc ∈ L,

(x − x∗)T (x − x∗) = (x − a − λ∗c)T (a + λc − a − λ∗c)

= (λ − λ∗)(x − a − λ∗c)T c

= (λ − λ∗)((x − a)T c − λ∗cT c)

= (λ − λ∗)((x − a)T c − (x − a)T c)

= 0

See Figure 3.40. So, the nearest point, x∗, on the straight lineL to x satisfies the property that (x − x∗) is orthogonal to (x − x∗)for all x ∈ L. That’s why the nearest point x∗ is also known as theorthogonal projection of x on L.

Example 1: Consider the straight line L and point x ∈ R 3 given

below.

L =

x : x =

3−4−11

+ λ

0−4

9

, λ real

, x =

12

−3

Then from the above, the nearest point in L to x is obtained bytaking

λ = λ∗ =(0,−4, 9)((1, 2,−3)− (3,−4,−11))T

||(0,−4, 9)T ||2 =48

25

360 Ch. 3. Geometry of Rn

900

x -axis1

2x -axis

3x -axis

.

.

0

L

x_

x*

Figure 3.40: The nearest point, x∗ in L to x, is the orthogonal projec-tion of x on L.

So, the nearest point to x in L in this example is

x∗ = (3,−4,−11)T + (48/25)(0,−4, 9)T = (3,−11.68, 6.28)T

x∗ is the orthogonal projection of x on the straight line L.

Nearest Point When S Is a Hyperplane

Consider the case where S = H = {x : ax = a0}, where a =(a1, . . . , an) �= 0; a hyperplane. In this case the problem of finding thenearest point in H to x is the quadratic program

Minimize (x − x)T (x − x)

subject to ax = a0

We can solve this problem using the Lagrange multiplier tech-nique of calculus. We construct the Lagrangian for this problem, whichis

3.20 Orthogonal Projections 361

L(x, δ) = (x − x)T (x − x) − δ(ax − a0)

where δ is the Lagrange multiplier associated with the constraintin the above problem. The optimum solution to the problem, x∗, δ∗ isthen obtained by solving the system

∂L(x, δ)

∂x= 2(x − x)T − δa = 0

ax − a0 = 0

From the first equation we have x∗ = x + (δ∗/2)aT . Substitutingthis in the 2nd equation we get (δ∗/2) = −(ax − a0)/||a||2 = ∆∗ say.So, the nearest point x∗ in H to x is given by

x∗ = x + aT ∆∗

where ∆∗ = −(ax − a0)/||a||2. It can be verified that for any x ∈ H

(x − x∗)T (x − x∗) = −∆∗a(x − x − aT ∆∗)

= −∆∗(ax − ax + (ax − a0))

= ∆∗(a0 − ax − (ax − a0)) = 0

since ax = a0 for all x ∈ H . Hence (x − x∗) is orthogonal to (x − x∗)for all x ∈ H . See Figure 3.41. Here again, the nearest point x∗ istherefore known also as the orthogonal projection of x in H .

Nearest Po int When S = {x : Ax = b}Here consider S = {x : Ax = b} where A is a matrix of order m×n.

Without any loss of generality, assume that there are no redundantequations in the system “Ax = b” and that it has at least one solution.

362 Ch. 3. Geometry of Rn

900

x -axis1

2x -axis

3x -axis

.

.

0

900

x*

x_

H

Figure 3.41: x∗ is the orthogonal projection (and the nearest point to)x in H .

The problem of finding the nearest point in S to x is the quadraticprogram

Minimize (x − x)T (x − x)

subject to Ax = b

Again we will use the Lagrange multiplier technique to solve thisproblem. Since there are m constraints, we need to use m Lagrangemultipliers here, one for each constraint. Let π = (π1, . . . , πm) be therow vector of Lagrange multipliers. Then the Lagrangian function forthis problem is

L(x, π) = (x − x)T (x − x) − π(Ax − b)

The optimum solution, x∗, π∗ is then obtained by solving the system

∂L(x, π)

∂x= 2(x − x)T − πA = 0

Ax − b = 0

3.20 Orthogonal Projections 363

From the first equation we have

x∗ = x + AT ((π∗)T

2).

Substituting this in the 2nd equation we get

Ax − b + AAT ((π∗)T

2) = 0

The fact that the consistent system of equations “Ax = b” has noredundant equations implies that the matrix AAT of order m × m isnonsingular (see Chapter 4), and hence has an inverse. Therefore fromthe above equation, we have

(π∗)T

2= −(AAT )− 1(Ax − b)

and hence

x∗ = x − AT (AAT )− 1(Ax − b)

Again it can be verified that x − x∗ is orthogonal to x − x∗ for allx ∈ S = {x : Ax = b}. Hence the nearest point x∗ is also known as theorthogonal projection of x in S. See Figure 3.42.

Example 2: Consider the set S ⊂ R 3 of solutions to the system

of constraints “Ax = b” where

A =

(0 1 −10 1 1

), b =

(29

)

Let x = (1, 1, 0)T . Then

Ax − b = (0,−8)T

AAT =

(3 00 2

), (AAT )−1 =

(1/3 00 1/2

)

AT (AAT )−1(Ax − b) =

1 01 1−1 1

(

1/3 00 1/2

)(0−8

)=

0−4−4

.

364 Ch. 3. Geometry of R n

900

.

.

x*

x_

S x .

Figure 3.42:

So, the orthogonal projection of x in S is

x∗ = x − AT (AAT )− 1(Ax − b) =

110

−

0−4−4

=

154

.

x∗ is the nearest point in S to x.

Ex ercis es

3.20.1: Find the nearest point to x on the straight line L = {x :x = a + λc, λ takes all real values} when

(i): x = (2,−3, 0, 2), a = (1,−3, 2, 0), c = (−1, 1,−1, 1).(ii): x = (−4, 10,−12,−3), a = (2, 1,−3,−6), c = (2,−3, 3,−1).(iii): x = (4, 9, 18, 6), a = (3,−2, 6, 5), c = (−2, 2, 1, 0).

3.20.2: Find the nearest point to x on the hyperplane H = {x :ax = a0} where

(i): x = (2, 1, 1, 1), a = (1,−1,−1, 1), a0 = 6.(ii): x = (1,−2,−1,−2), a = (−2, 11, 6,−3), a0 = −24.(iii): x = (−1, 0, 3), a = (3,−4, 0), a0 = 6.

3.21 Geometric Procedures 365

3.20.3: Find the nearest point to x on the set of solutions of thesystem Ax = b where

(i): x = (2, 3, 1), A =

(1 0 −1

−1 1 0

), b =

(3

−9

).

(ii): x = (−2,−1, 1), A =

(1 2 12 1 0

), b =

( −3−5

).

3.21 Procedures for Some Geometric Prob-

lems

Here we describe briefly procedures for common geometric problemsthat arise in many applications, based on algorithms discussed in earliersections.

1. Identical, Parallel, Intersecting, Orthogonal, and Non-intersecting Straight Line Pairs

Let L1 = {x = a + αb : α real}, L2 = {x = c + βd : β real} be twostraight lines in Rn given in parametric representation. Assume thatall of a, b, c, d are column vectors.

(i): L1, L2 are identical, i.e., they are the same straight line ifb, d, a − c are all scalar multiples of each other.

When L1, L2 are not the same straight lines, but n = 2:

(ii): L1, L2 are distinct but parallel straight lines in R2 ifb, d are scalar multiples of each other, but a − c is not ascalar multiple of b.

(iii): If b is not a scalar multiple of d, then L1, L2 are notparallel lines in R2 and have a unique point of intersection.

In this case, if the dot product bT d =< b, d >= 0, thenL1, L2 are orthogonal straight lines.

When L1, L2 are not the same straight lines, but n ≥ 3:

366 Ch. 3. Geometry of Rn

(iv): If b, d are scalar multiples of each other, and (a−c) isnot a scalar multiple of b, then L1, L2 are two parallel linesin Rn both contained in an affine space of dimension 2.

(v): If b, d are not scalar multiples of each other, andrank{b, d, a − c} is 2, then L1, L2 are not parallel, but arecontained in an affine space of dimension 2, and they inter-sect at a unique point.

In this case L1, L2 are an intersecting orthogonal pair ofstraight lines in Rn if bT d = 0. If bT d �= 0, they are notorthogonal.

(vi): If b, d are not scalar multiples of each other, andrank{b, d, a − c} is 3, then L1, L2 are not parallel, but arenot contained together in any affine space of dimension 2in Rn, and do not intersect.

In this case they are a nonintersecting, orthogonal pair ofstraight lines if bT d = 0. If bT d �= 0 they are not orthogonal.

The rank of a set of vectors needed in the above conditions is brieflymentioned in Section 3.9, but detailed treatment of rank along withalgorithms for computing it are given in Sections 4.2, 4.3.

So, to summarize, for any n ≥ 2, L1, L2 are nonidentical parallelstraight lines iff b, d are scalar multiples of each other and a − c is nota scalar multiple of b.

Also, whether they intersect or not, they are an orthogonal pair iffbT d = 0.

2. Angle Between Half-Lines

Given two half-lines in Rn in parametric form, L+1 = {x = aαb :

α ≥ 0}, L+2 = {x = c + βd : β ≥ 0}, whether they have a common

point or not, the angle between them is defined to be the angle betweentheir directions. If this angle is θ, we therefore have cos(θ) = < b, d >/((||b||)(||d||)).

3.21 Geometric Procedures 367

The half-lines L+1 , L+

2 are said to make an acute angle if 0 ≤ θ ≤900, and an obtuse angle if 900 < θ ≤ 1800. Therefore L+

1 , L+2 make

an acute angle if < b, d >≥ 0; and an obtuse angle if < b, d >< 0.

3. To Find the Equation for a Hyperplane Containing aGiven Set of Points in R

n

Let the given set of points be Γ = {a1, . . . , ak} where each at is a col-umn vector. Let c1x1+ . . .+cnxn = c0 be the equation for a hyperplanecontaining all the points in Γ. We need to find c = (c1, . . . , cn) �= 0 andc0, these are the variables in this problem. If 0 ∈ Γ, we know that c0

must be zero. If 0 �∈ Γ we do not know the value of c0. The system ofequations that c, c0 satisfies is:

c c0

a1 −1 0...

......

ak −1 0

As in the algorithm discussed in Section 3.9, apply the algorithmof Section 1.23 on this system. This algorithm terminates in threedifferent ways with a final tableau.

(i): No nonbasic variable in the final tableau: In this case(c, c0) = (0, 0) is the only solution to the above system, so there is nohyperplane in Rn that contains all the points in Γ. This happens onlyif dimension(Γ) = n.

(ii): Exactly one nonbasic variable in the final tableau:Let (c, c0) be the solution of the above system obtained by setting thenonbasic variable = 1, and finding the values of the basic variables fromthe final tableau. In this case dimension(Γ) = n − 1, and cx = c0 isthe equation for the unique hyperplane in Rn containing all the pointsin Γ.

(iii): Two or more nonbasic variables in the final tableau:

368 Ch. 3. Geometry of Rn

Let (cr, cr0) be the solution of the above system obtained by setting

the rth nonbasic variable = 1, other nonbasic variables = 0, and thenobtaining the values of the basic variables from the final tableau; for r= 1 to k.

In this case dimension(Γ) < (n − 1), and there are many (in factan infinite number) hyperplanes containing all the points in Γ. Eachequation in the system

c1x = c10

......

ckx = ck0

represents a different hyperplane containing all the points in Γ, and anynonzero linear combination of these equations also gives such a hyper-plane. The above system of equations together defines the affine hullof Γ. So, this is one way of getting the linear equation representationof the affine hull of a given set of points.

4. Checking Whether a Straight Line Lies on a Hyper-plane, or Is Parallel to it, or Orthogonal to it

Let a, b be column vectors, and c a row vector in Rn. Let c0 be areal number. Consider the straight line L = {x = a + αb : α real}, andthe hyperplane H = {x : cx = c0}.

Consider the equation c(a + αb) = c0 in the parameter α.

If cb �= 0, the point on L given by α = (ca− c0)/(−cb) is the uniquepoint of intersection of L and H . So in this case L and H intersect.

If c, b are scalar multiples of each other, L is orthogonal to H .

If cb = 0 and ca−c0 �= 0, there is no finite value of the parameter αthat will satisfy the above equation. In this case, L and H are parallel.

If cb = 0, and ca − c0 = 0, the above equation is satisfied by allvalues of the parameter. in this case, L lies completely in H .

A straight line lies completely on a hyperplane, if they have twocommon points.

3.21 Geometric Procedures 369

5. Normal Line to a Hyperplane Through a Point On It

Let cx = c0 represent a hyperplane H in R n, and let x be a point

on it. So, cx = c0.

{x = x + αd : α real} represents a straight line through x lying inH if d �= 0 and cd = 0.

A stright line through x is said to be the normal line to H throughx if it is orthogonal to every straight line through x lying in H . So,the normal line to H through x is given by the representation {x =x + βc : β real}.

6. Parallel and Orthogonal Hyperplanes

Two hyperplanes in R n are said to be parallel iff they do not

intersect. This happens iff the equation for one of them can be obtainedfrom the equation for the other by changing its RHS constant. That’swhy changing the RHS constant in the equation for a hyperplane iscalled moving the hyperplane parallel to itself.

So, two hyperplanes H1 = {x : cx = c0 }, H 2 = {x : dx = d 0 } areparallel to each other if c, d are scalar multiples of each other, i.e., ifthere exists a nonzero number δ such that c = δd, and c − 0 �= δd0.

Two hyperplanes are said to be orthogonal if their normal linesform an orthogonal pair of straight lines. So, H1, H2 are orthogonal iffthe dot product < c, d > = 0.

Ex ercis es

3.21.1: The angle between two nonzero points is defined to be theangle between the rays of those two points. If a = (1, a2,−2), b =(3, 1, 2) what condition should a2 satisfy so that the angle between a, bis: (i) 900, (ii) < 900, (iii) > 900.

3.21.2: Formulate the system of equations to find a point x ∈ R3

that is orthogonal to both the vectors (1, 1,−1) and (1,−1, 1) and hasEuclidean norm 1.

370 Ch. 3. Geometry of Rn

3.21.3: Does the straight line through (1, 2, 3) and (−1, 0, 1) inter-sect the hyperplane defined by the equation 2x1 + 3x2 − x3 = 5? If sofind the point of intersection. Do the same for the same straight lineand the hyperplane defined by 6x1 + 8x2 − 14x3 = 15.

How many points of intersection does the straight line through (2,3, 0, 7) and (1,−8, 5, 0) have with the hyperplane defined by x1 +x2 −x3 − x4 = −2?

3.21.4: Find a hyperplane in R4 containing all the points (−1, 1, 0, 0),(0, 1,−1, 0), (0, 0,−1, 1), (0, 1, 0, 1), (−1, 0,−1, 0), (1, 2, 1, 2).

Do all the points (2, 0, 1), (0, 2, 1), (1, 1, 1), (1, 1,−1) lie on ahyperplane in R3? If not is there a half-space in R3 that contains allof them? If so find one such half-space.

In R3 find a hyperplane if possible, or a half-space otherwise, thatcontains all the points in the set {(1, 2, 0), (1, 0, 2), (1, 1, 0), (1, 0,1), (1, 1, 1), (2, 1, 0), (2, 0, 1), (0, 1, 1)}. Can you do this if therequirement is to contain all but one of the points in the set?

3. 21.5: Find a hyperplane in R3 that is orthogonal to both thehyperplanes represented by the equations x1+x2−x3 = 6, x1−x2+x3 =−2, and passes through the point (1, 1, 1).

Find the equation of the hyperplane in R3 that contains the point(1,−1, 1) and contains the entire straight line joining (2, 1, 1) and (1,2, 1).

3.21.6: Check whether the two straight lines joining the followingpairs of points are parallel.

(Yes){{(1,−1, 0, 7), (2, 1, 3, 6)}, {(5, 5, 5, 5), (2,−1,−4, 8)}}, {{(10,20, 30, 40), (9, 19, 31, 41)}, {(40, 40, 40, 40), (30, 30, 50, 50)}}

(No) {{(−6,−7, 0, 3), (−5,−6, 1, 4)}, {(14, 13, 20, 23), (34, 33, 40, 43) }}, {{(1, 1, 2), (2, 0, 3)}, {(5, 6, 7), (6, 7, 8)}}.

3.21.7: Check whether the following pair of straight lines, each onespecified by a pair of points on it, are orthogonal.

(Yes) {{(1, 2, 1, 1), (2, 1, 0, 2)}, {30, 40, 50, 60), (28, 38, 53, 63)}},{{(10, 10, 10, 10), (11, 9, 10, 10)}, {(11, 9, 10, 10), (11, 9, 11, 12)}}.

3.21 Geometric Procedures 371

(No) {{(2, 1, 1, 1), (2, 3, 0, 1)}, {(5, 5, 5, 5), (3, 6, 4, 15)}}, {{(2, 1, 2),(3, 3, 5)}, {(7, 0,−1), (1, 1, 0)}}.

3.21.8: Check whether the straight line L lies on the hyperplaneH in the following.

(Yes) {L joins (1,−2, 3) and (2,−1, 2), H = {x : 2x1 + 3x2 + 5x3 =11}}, {L joins (2, 1, 4, 3) and (3, 0, 3, 4), H = {x : 2x2 + 4x3 + 6x4 =36}}.

(No) {L joins (2, 1, 5) and (1, 0, 6), H = {x : 3x1+2x2+4x3 = 28}},{L joins (10, 15, 20, 25) and (11, 16, 21, 26), H = {x : 3x1 + 2x2 −2x3 − 3x4 = −50}}.

3.21.9: Find the normal lines to the hyperplanes H at points xgiven on them.

x = (1,−1,−1, 2)T on H = {x : 2x1 − 3x2 + 4x4 = 13}.x = (0, 3,−7)T on H = {x : x1 − x3 = 7}.

3.21.10: Find the equation of a hyperplane containing the point(5, 0, 3,−3)T for which the normal at that point is the straight line{x = (1, 2,−1,−1)T + α(2,−1, 2,−1)T : α real}.

3.21.11: Check whether the following pairs of hyperplanes H1 andH2 are parallel.

(Yes) {H1 = {x : 2x1 − 6x2 + x3 = −10}, H2 = {x : −6x1 + 18x2 −3x3 = 30}}, {H1 = {x : 4x1 − 10x4 = 6}, H2 = {x : 160x1 − 400x4 =240}}, {H1 is the hyperplane through the points {(1, 1, 1), (2, 0, 1), (−1,2, 0)}, H2 is the hyperplane through the points {(1, 1, 0), (2, 1, 1), (1, 2,1)}}.

(No) {H1 = {x : 6x1 − 7x2 +18x3 − 19x4 = 6}, H2 = {x : 6x1 − 7x2

+18x3 − 18x4 = 6}}, {H1 = {x : 4x1 − 2x2 + 8x3 − 7x4 + 20x5 = 500},H2 = {x : 200x1 − 100x2 + 400x3 − 350x4 + 990x5 = 25, 000}}, {H1 isthe hyperplane through the points {(1, 1, 1), (1, 0, 2), (−1,−3, 1)}, H2

is the hyperplane through the points {(1, 1, 1), (2, 0, 2), (1, 2,−1)}}.

3.21.12: Find the angle between L1, L2 in the following pairs andcheck whether they make an acute or obtuse angle.

372 Ch. 3. Geometry of Rn

(Acute) {L1 joins {(3, 2,−1, 2) to (4, 4,−2, 2)}, L2 joins {(5, 5, 5, 5)to (4, 7, 4, 7)}}, {L1 joins {(1, 2, 2, 1) to (2, 1, 1, 2)}, L2 joins {(3, 1, 1, 3)to (2, 0, 2, 4)}}.

(Obtuse) {L1 joins {(7,−4, 3, 2) to (8,−3, 4, 3)}, L2 joins {(9, 10,−1,−2) to (8, 9,−2,−3)}}, {L1 joins {(0, 0) to (1, 0)}, L2 joins {(0, 0) to(−1, 0)}}.

3.21.13: Determine whether the straight line L and the hyperplaneH in the following pairs are parallel.

(Yes) {L joins {(1, 1, 1), (2, 1, 1)}, H is the hyperplane through{(1, 1, 0), (1, 0, 0), (0, 1, 0)}}, {L joins {(5, 2,−2, 2), (6, 1,−2, 2)}, H isthe hyperplane through {(1, 1, 2, 1), (−1, 1, 1, 0), (−1,−1, 0,−1), (1, 0,−1,−2)}}.

(No) {L joins {(1, 1, 1, 1), (2, 2, 3, 2)}, H is the hyperplane through{(1, 1, 0, 0), (0, 1, 1, 0), (1, 2, 1, 0), (0, 0, 1, 1}}, {L joins {(5, 2,−2, 2), (6,1,−2, 2)}, H is the hyperplane through {(1, 0, 1, 1), (1, 1, 0, 1), (0, 0, 1, 2),(1,−1, 1,−1)}}.

3.21.14: Check whether the following pairs of hyperplanes H1, H2

are orthogonal.

(YES) {H1 is the hyperplane through {(1, 1, 0), (2, 0, 1), (0, 2, 1)},H2 is the hyperplane through {(1, 0, 1), (2, 1, 1), (1, 1, 2)}}, {H1 is thehyperplane through {(1, 1, 1, 0), (0, 1, 1, 1), (1, 0, 0, 1), (0, 0, 1, 1)}, H2 isthe hyperplane through {(0,−1,−1, 0), (2, 0, 0, 1), (1,−1, 0, 0), (1, 0,−1,1)}}.

(No) {H1 is the hyperplane through {(1, 1, 0, 0), (2, 0, 1, 0), (2, 0, 0,1), (1, 1, 1, 1)}, H2 is the hyperplane through {(0, 1, 1, 0), (1, 2, 0, 0), (1, 0,2, 1), (1, 1, 1, 1)}}, {H1 is the hyperplane defined by the equation x1

+x2 −x3 = 2, H2 is the hyperplane through {(−1,−1, 0, 0), (0,−1,−1, 0),−(0, 0,−1,−1), (0,−1,−1, 0)}}.

3.21.15 Find a hyperplane through the following sets of vectors Γ.

(None) Γ = {(1, 1, 0), (1, 0, 1), (0, 1, 1), (1, 1, 1)}.(Unique) Γ = {(1, 1, 1, 0), (0, 1, 1, 1), (1, 0, 0, 1), (0, 0, 1, 1)}.(Nonunique) Γ = {(2, 0, 1), (1, 1, 2), (3,−1, 0)}.

3.22 Additional Exercises 373

3.21.16 Find the equation of the hyperplane through the point(7,−8,−6), parallel to the hyperplane containing the points (1,1,0),(0,1,1), (1,0,1).

3.22 Additional Exercises

1. The general equation for a circle in the x1, x2-coordinate planeis a0(x

21 + x2

2) + a1x1 + a2x2 + a3 = 0, where a0 �= 0. Given threepoints which do not all lie on a straight line, there is a unique circlecontaining all of them, develop a method for computing the equation ofthat circle using the algorithm in Section 1.23. Use that method to findthe equations of the circles containing all the points in the followingsets: {(1, 0), (0, 1), (1, 2)}, {(1, 1), (2, 3), (3, 5)}, {(2, 2), (2,−2), (4, 0)}.

2. xr, r = 1, 2, 3 are the coordinates of the vertices of a triangleABC in R2 in that order. P, Q are the midpoints of the sides AB, ACof this triangle. Prove that the line PQ is parallel to the line BC, andthat the length of the line segment PQ is half the length of the sideBC of the triangle.

Let D, E be the points on the sides AB, AC of this triangle thatare a fixed fraction α of the length of that side away from A, for some0 < α < 1. Prove that the line segment DE is parallel to the side BCof the triangle, and that its length is α times the length of the side BC.

3. xr, r = 1, 2, 3, 4 are the coordinates of the vertices of a quadri-lateral ABCD in R2 in that order. P, Q, R, S are the midpoints of thesides of this quadrilateral. Prove that P, Q, R, S are the vertices of aparallelogram. Also show that the diagonals of this parallelogram in-tersect at the centroid of the quadrilateral which is (x1+x2+x3+x4)/4.

4. A, B, C, D, E are the points (4, 4, 0), (8, 2, 0), (8, 8, 0), (4, 8, 0),(6, 6, 4) in R3 respectively. They are the vertices of a pyramid in R3.The 4 vertices A, B, C, D all lie on a hyperplane which is the base ofthis pyramid. E lies above this base and is known as the apex of thepyramid.

Let P, Q, R, S be the midpoints of the edges AE, BE, CE, DE of

374 Ch. 3. Geometry of Rn

this pyramid. Show that all these four points lie on a hyperplane whichis parallel to the base of the pyramid.

5. Consider a parallelogram in R2. set up the axes of coordinateswith one of the vertices of the parallelogram as the origin. Let a =(a1, a2), b = (b1, b2) be the coordinate vectors for the two vertices ofthe parallelogram adjacent to 0. By parallelogram law, the 4th vertexmust be a + b. Using this framework, show that the diagonals of theparallelogram intersect at the point which is the midpoint of both thediagonals.

6. Let a, b be two nonzero vectors in Rn. Show that the parallelo-gram {x = αa + βb : 0 ≤ α ≤ 1, 0 ≤ β ≤ 1} is a convex set.

7. Finding the path of a flying bird: Axes of coordinates havebeen set up to trace the path of a bird flying high up in the sky. Thex1, x2-axes are on the ground, and the x3-axis is along the verticalaltitude. The unit of measurement along each axis is 1 foot. At time0-minutes, she is at the point (100,−200, 400). She is flying in thedirection (−2, 1, 0) at a constant speed of 200 feet per minute. At heraltitude, the wind is blowing in the direction of (2,−3, 0) at a constantspeed of 1500 feet per minute. Derive a parametric representation ofher path, and find her location at time 10 minutes.

3.23 References

[3.1] C. W. Curtis, Linear Algebra: An Introductory Approach, Springer,1991.

Index

For each index entry we providethe section number where it isdefined or discussed first. Click-ing on the light blue section num-ber takes you to the page wherethis entry is defined or discussed.

Affine combinations 3.11, 3.13Affine hull (span) 3.11, 3.13Affine space 3.11Analytical geometry 3.1Angles & dot products 3.18, 3.21

Cartesian coordinate system3.1

Cauchy-Schwartz inequality 3.18Closed linear inequalities 3.15Column space 3.10Convex combinations 3.12, 3.13Convex hull 3.12, 3.13Convex polyhedron 3.12, 3.16Convex polyhedral cone 3.16Convex polytope 3.8, 3.16Convex set 3.17Coordinate axes 3.1Coordinate planes 3.1Coordinate vector 3.1

Determinant 3.19

Direction vector 3.5

Euclidean distance (norm) 3.18Equation of hyperplane 3.21

Flat 3.11

Half-lines 3.5Half-spaces 3.15Homogeneous linear inequalities

3.13Homogeneous system represen-

tation 3.9Hyperplanes 3.14

Law of cosines 3.18Line segments 3.6Linear combinations 3.9, 3.13Linear dependence relation 3.9Linearly dependent 3.9Linear hull (span) 3.9, 3.13Linearly independent 3.9

Maximal linear independence3.9

Midpoint of a line segment 3.6

Nearest points 3.20Nonconvex sets 3.17

375

376 Ch. 3. Geometry of Rn

Nonnegative combinations 3.13Nontrivial linear equation 3.14Normal to hyperplane 3.21Null space 3.10

Origin 3.1Orthogonal hyperplanes 3.21Orthogonal lines 3.21Orthogonal projections 3.20Orthogonal Pair of vectors 3.18

Parallel hyperplanes 3.21Parallel linesParallelogram law 3.8Parametric representation 3.3Points & vectors 3.1Pos cones 3.13Pythogoras theorem 3.18

Rank 3.9Rays 3.3Real line 3.1Row space 3.10

Straight lines 3.4 , 3.11, 3.14Identical 3.4, 3.21Parallel 3.21Intersecting 3.4, 3.21Orthogonal 3.21Nonintersecting 3.21

Step length 3.5Subspaces 3.9

Translation 3.2Triangle inequality 3.18

Volume 3.19

Index for Ch. 3 377

Summary of different types of combinations of a given setof vectors: linear, affine, convex, and nonnegative

Procedures for geometric problems.

1. To check whether two straight lines are: same, parallel, inter-secting, nonintersecting, or orthogonal

2. Angle between two half-lines3. To find the equation for a hyperplane containing a given set of

points4. Checking whether a straight line: lies on, is parallel to, or or-

thogonal to, a hyperplane5. Find normal line to a hyperplane through a point on it6. Parallel and orthogonal hyperplanes.

List of algorithms

1. To check if b is in the linear hull of {a1, . . . , ak }2. To find representations for the linear hull of {a1, . . . , ak }3. To find the linear equation representation of F = {x = a1 +

β1c1 + . . . + βk − 1c

k − 1 : β1, . . . , βk − 1 real}.

List of results

Result 3.9.1: Representing the linear hull of a given set of vectorsas the solution set of a homogeneous system of linear equations

List of figures

Figure 3.1 Real lineFigure 3.2 2-dimensional Cartesian planeFigure 3.3 3-dimensional Cartesian spaceFigure 3.4 3-dimensional Cartesian spaceFigure 3.5 Translating the originFigure 3.6 x + S for a given set SFigure 3.7 A Straight line in R

2

378 Ch. 3. Geometry of Rn

Figure 3.8 Half-lines and rays in R 2

Figure 3.9 A Half-line and the ray parallel to itFigure 3.10 A line segment in R

2

Figure 3.11 A straight line and a rayFigure 3.12 Parallelogram lawFigure 3.13 A subspace in R

2

Figure 3.14 A subspace in R 3

Figure 3.15 A Straight line as an affine hullFifure 3.16 An affine hull in R

3

Figure 3.17 Convex hull of two points in R 2

Figure 3.18 A convex hull in R 2

Figure 3.19 A cone in R 2

Figure 3.20 Straight lines in R 2

Figure 3.21 A coordinate plane in R 3

Figure 3.22 A coordinate plane in R 3

Figure 3.23 A hyperplane in R 3

Figure 3.24 Straight lines and a half-space in R 2

Figure 3.25 A hyperplane and one of its haslf-spaces in R 2

Figure 3.26 A convex polyhedron in R 2

Figure 3.27 A convex polytope in R 2

Figure 3.28 A convex polyhedral cone in R 2

Figure 3.29 Convex setsFigure 3.30 Nonconvex setsFigure 3.31 Explanation for the Euclidean distance function in R

2

Figure 3.32 Explanation for the Euclidean distance function in R 3

Figure 3.33 Triangle inequalityFigure 3.34 Angle between the rays of two nonzero vectorsFigure 3.35 0-angle explainedFigure 3.36 1800-angle explainedFigure 3.37 Orthogonal vectorsFigure 3.38 Properties of the area of a parallelogramFigure 3.39 Properties of the area of a parallelogramFigure 3.40 Orthogonal projection on a straight lineFigure 3.41 Orthogonal projection on a hyperplaneFigure 3.42 Orthogonal projection on a flat

Contents

4 Numerical Linear Algebra 3794.1 Linear Dependence, Independence of Sets of Vectors . . 3794.2 Rank and Bases For a Set of Vectors . . . . . . . . . . 3824.3 Two Algorithms to Check Linear Dependence, Find Rank

and a Basis for a Given Set of Vectors . . . . . . . . . 3864.4 How to Get an Alternate Basis by Introducing a Non-

basic Vector Into a Basis ? . . . . . . . . . . . . . . . . 3944.5 The Rank of a Matrix . . . . . . . . . . . . . . . . . . 3974.6 Computing the Inverse of a Nonsingular Square Matrix

Using GJ Pivot Steps . . . . . . . . . . . . . . . . . . . 4024.7 Pivot Matrices, Elementary Matrices, Matrix Factoriza-

tions, and the Product Form of the Inverse . . . . . . . 4094.8 How to Update the Inverse When a Column of the Ma-

trix Changes . . . . . . . . . . . . . . . . . . . . . . . . 4234.9 Results on the General System of Linear Equations Ax =

b . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4274.10 Optimization, Another Important Distinction Between

LA, LP, and IP; and the Duality Theorem for LinearOptimization Subject to Linear Equations . . . . . . . 432

4.11 Memory Matrix Versions of GJ, G Elimination Methods 4444.12 Orthogonal and Orthonormal Sets of Vectors, Gram-

Schmidt Orthogonalization, QR-Factorization, Orthog-onal Matrices, Orthonormal Bases . . . . . . . . . . . . 461

4.13 Additional Exercises . . . . . . . . . . . . . . . . . . . 4704.14 References . . . . . . . . . . . . . . . . . . . . . . . . . 472

i

ii

Chapter 4

Numerical Linear Algebra

This is Chapter 4 of “Sophomore Level Self-Teaching Web-book for Computational and Algorithmic Linear Algebra andn-Dimensional Geometry” by Katta G. Murty.

4.1 Linear Dependence, Independence of

Sets of Vectors

Given a set of vectors (all of them column vectors, or all of themrow vectors; here we will assume that they are all column vectors)∆ = {a1, . . . , ak} from Rn, it is sometimes important to be able tocheck whether any of the vectors in this set can be expressed as alinear combination of the others.

In Section 1.9 we encountered this problem in order to check whethera system of linear equations contains a redundant constraint that canbe eliminated without effecting the set of solutions. For this, the coef-ficient vector of one of the equations in the system should be a linearcombination of the others. In Section 3.9 we encountered this prob-lem again in the process of developing parametric representations forthe linear hull of a given set of vectors using the smallest number ofparameters.

Broadly speaking, we will say that the set ∆ = {a1, . . . , ak} of vec-tors is linearly dependent if one of the vectors in it can be expressed

379

380 Ch. 4. Numerical Linear Algebra

as a linear combination of the others, linearly independent other-wise. For example, if a1 can be expressed this way as in

a1 = α2a2 + . . . + αka

k.

This leads to

−a1 + α2a2 + . . . + αka

k = 0.

Now defining α1 = −1, this is

α1a1 + α2a

2 + . . . + αkak = 0

where (α1, . . . , αk) �= 0 (because α1 = −1, the vector (α1, . . . , αk) �= 0).In keeping with the name that we gave for a similar relation amongconstraints, we will call a relation like this among vectors in the set ∆a linear dependence relation as long as the vector of coefficients init (α1, . . . , αk) �= 0.

When k = 1, i.e., ∆ contains only one vector a1 in it, there isno possibility of expressing a vector in ∆ as a linear combination ofthe others because the only vector in ∆ is a1. In this special case, fortechnical reasons of consistency, we will define the singleton set ∆ to belinearly dependent if the only vector a1 in it is 0, linearly independentif a1 �= 0. With this convention, we again see that in this special casealso, the set ∆ is linearly dependent if there is a linear dependencerelation for it (because if a1 = 0, then α1a

1 = 0 for all α1 �= 0).The only other special set to consider is the empty set ∅, i.e., the set

containing no vectors at all. Again for technical reasons of consistency,we define the empty set to be linearly independent. So, here is themathematical definition.

Definition: The set of vectors ∆ = {a1, . . . , ak} in Rn is linearlydependent iff it is possible to express the 0-vector as a linear com-bination of vectors in it with at least one nonzero coefficient; i.e., iffthere exists α = (α1, . . . , αk) �= 0 such that

α1a1 + α2a

2 + . . . + αkak = 0. (4.1)

4.1 L inear D ep endence, Indep endence 381

Since each vector here is from R n, this is a homogeneous system

of n equations in k variables. In this case (4.1) is known as a lin-ear dependence relation for ∆. The set ∆ is said to be linearlyindependent iff there is no linear dependence relation for it. ��

If the set ∆ has two or more vectors in it and it is linearly dependent,then from a linear dependence relation for it, we can find an expressionfor one of the vectors in ∆ as a linear combination of the others. Forexample, in (4.1), since α = (α1, . . . , αk) �= 0, at least one of the αt

must be nonzero. If α1 �= 0, then from (4.1) we have

a1 = − 1

α1

(α2a2 + . . . + αka

k)

Example 1: Let

∆ =

a1 =

100

, a2 =

010

, a3 =

001

, a4 =

5−6

0

Since −5a1+6a2+a4 = 0, ∆ is linearly dependent and this equationis a linear dependence relation for it. Also, from this relation we have

a1 = −(6/5)a2 − (1/5)a4

a2 = (5/6)a1 − (1/6)a4

a4 = 5a1 − 6a2

Example 2: Let

∆ =

a1 =

100

, a2 =

010

, a3 =

001

,

The system of equations (4.1) for this set ∆ here is given below indetached coefficient form.

382 Ch. 4 . Numerical Linear Algebra

α1 α2 α3

1 0 0 00 1 0 00 0 1 0

This homogeneous system has the unique solution α = (α1, α2, α3) =0. So, there is no linear dependence relation for this set ∆, hence thisset ∆ is linearly independent.

Using the same argument, it can be verified that for any n ≥ 2, theset of n unit vectors in R

n is linearly independent.We will discuss two algorithms for checking linear dependence or

independence of a given set of vectors in Section 4.3. When the setis linearly dependent, these algorithms will always output one or morelinear dependence relations for it.

4.2 Rank and Bases For a Set of Vectors

From the definition it is clear that any subset of a linearly indepen-dent set of vectors is linearly independent. So, the property of linearindependence of a set of vectors is always preserved when vectors aredeleted from it, but may be lost if new vectors are included in it. Forexample, the linearly independent set {(1, 0, 0)T , (0, 1, 0)T , (0, 0, 1)T}remains linearly independent when vectors are deleted from it, butbecomes linearly dependent when any vector from R

3 is included in it.

Maximal Linearly Independent Subsets

Let Γ ⊂ R n be a set of vectors. As defined in Section 3.9, a subset

E ⊂ Γ is said to be a maximal linearly independent subset of γif E is linearly independent, and either E = Γ, or E ∪ {a} is linearlydependent for all a ∈ Γ\E.

If Γ = ∅ or Γ = {0}, the only maximal linearly independent subsetof Γ is the emptyset.

If Γ has at least one nonzero vector, one procedure for finding amaximal linearly independent subset of Γ is the following simple aug-mentation scheme:

4.2 Rank, Bases 383

Start the scheme with the subset consisting of one nonzerovector from Γ. Always maintain the subset linearly inde-pendent. Augment it by one new element of Γ at a time,making sure that linear independence holds; until no newvectors from Γ can be included, at which stage the subsetbecomes a maximal linearly independent subset of Γ.

But this scheme is not computationally efficient. We will discussan efficient method for finding a maximal linearly independent subsetbased on pivot operations, in the next section.

Minimal Spanning Subsets

Let Γ ⊂ Rn be a set of vectors. Let F ⊂ Γ satisfy the property thatevery vector in Γ can be expressed as a linear combination of vectorsin F . Then F is said to be a spanning subset of Γ (or, to have theproperty of spanning Γ). As an example, let

Γ1 =

a1 =

100

, a2 =

−200

, a3 =

110

,

a4 =

−3−4

0

, a5 =

11

−1

Then the subset F1 = {a1, a3, a5} is a spanning subset of Γ1. ButF2 = {a1, a3, a4} is not a spanning subset of Γ1, because a5 in Γ1 cannotbe expressed as a linear combination of vectors in F2.

Given Γ ⊂ Rn, it is clear from the definition that a spanning subsetof Γ retains its spanning property when more vectors are included init, but may lose the spanning property if vectors are deleted from it.

A subset F ⊂ Γ is said to be a minimal spanning subset of Γ ifit is a spanning subset of Γ, but none of the proper subsets of F havethe spanning property for Γ.

The definition itself suggests a simple deletion scheme for findinga minimal spanning subset of Γ. The scheme maintains a subset of Γ.Initially that subset is Γ itself. In each step, delete one vector from the

384 Ch. 4. Numerical Linear Algebra

subset making sure that the remaining subset always has the spanningproperty for Γ (for this it is enough if the vector to be deleted is inthe linear hull of the remaining elements in the subset at that stage).When you reach a stage where no more vectors can be deleted from thesubset without losing the spanning property for Γ (this will happen ifthe subset becomes linearly independent), you have a minimal spanningsubset of Γ.

Again this scheme is computationally expensive. We will discuss anefficient method for finding a minimal spanning subset based on rowoperations in the next section.

The Rank of a Set of Vectors, Bases

We have the following results.

Result 4.2.1: Maximal linear independence and minimalspanning property imply each other: Every maximal linearly in-dependent subset of a set Γ ⊂ R

n, is a minimal spanning subset for Γand vice versa.

Result 4.2.2: Rank of a set of vectors: All minimal spanningsubsets of Γ ⊂ Rn have the same cardinality (i.e., same number of el-ements in it); it is the same as the cardinality of any maximal linearlyindependent subset of Γ; as mentioned in Section 3.9 earlier, this num-ber is called the rank of the set Γ. It is also the dimension of the linearhull of Γ.

Definition: Every maximal linearly independent subset of Γ or aminimal spanning subset of Γ is called a basis for Γ. ��

Thus a basis for a set of vectors Γ in Rn is a linearly independentsubset satisfying the property that all vectors in the set Γ not in thebasis can be represented as linear combinations of vectors in the basis.A set of vectors Γ may have many bases, but every basis for Γ consistsof the same number of vectors, this number is the rank of Γ.

When referring to a basis B for a set Γ ⊂ Rn, vectors in B are called

4.2 Rank, Bases 385

basic vectors, and vectors in Γ not in B are called nonbasic vectors.In most applications, the basic vectors themselves are arranged in someorder and called the first basic vector, second basic vector, etc.Then the word basis refers to this ordered set with the basic vectorsarranged in this order.

Let I be the unit matrix of order n. Its set of column vectors{I.1, I.2, . . . , I.n} is a basis for Rn called the unit basis. For example,for n = 3

100

,

010

,

001

is the unit basis for R3. Hence we have the following result.

Result 4.2.3: Ranks of subsets of R n: The rank of R

n is n.Every basis for Rn always consists of n vectors. And therefore any setof n + 1 or more vectors from Rn is always linearly dependent. Therank of any subset of Rn is ≤ n.

Result 4.2.4: Basic vectors span each nonbasic vector unique-ly: Every nonbasic vector can be expressed as a linear combination ofbasic vectors in a unique manner.

By definition, every nonbasic vector can be expressed as a linearcombination of basic vectors. We will now show that this expression isalways unique.

Let the basis B = {A1, . . . , Ar} and let Ar+1 be a nonbasic vector.Suppose Ar+1 can be expressed as a linear combination of vectors in Bin two different ways, say as

Ar+1 =r∑

t=1

βtAt and as Ar+1 =r∑

t=1

γtAt

where (β1, . . . , βr) �= (γ1, . . . , γr). So, if α = (α1, . . . , αr) = (β1 −γ1, . . . , βr − γr) then α �= 0 and we have

r∑t=1

αtAt =r∑

t=1

βtAt −r∑

t=1

γtAt = Ar+1 − Ar+1 = 0

386 Ch. 4. Numerical Linear Algebra

This is a linear dependence relation for B = {A1, . . . , Ar}, a con-tradiction since B is a basis. So, the expression for the nonbasic vectorAr+1 as a linear combination of basic vectors must be unique.

Definition: The vector of coefficients in the expression of a nonba-sic vector as a linear combination of basic vectors, arranged in the sameorder in which the basic vectors are arranged in the basis, is called therepresentation of this nonbasic vector in terms of this basis.

Thus if the basis B = {A1, . . . , Ar} with the vectors arranged inthis order, and the nonbasic vector Ar+1 =

∑rt=1 βtAt, then (β1, . . . , βr)

is the representation of Ar+1 in terms of the basis B. As an example,let

Γ1 =

a1 =

100

, a2 =

−200

, a3 =

110

,

a4 =

−3−4

0

, a5 =

11

−1

Let F1 = {a1, a3, a5}. F1 is a basis for the set Γ1. The nonbasicvector a4 = a1 −4a3, and hence the representation of a4 in terms of thebasis F1 for Γ1 is (1,−4, 0)T .

4.3 Two Algorithms to Check Linear De-

pendence, Find Rank and a Basis for

a Given Set of Vectors

Let ∆ = {a1, . . . , ak} be a given set of vectors in Rn. Here we discusstwo algorithms using pivot steps, for checking whether ∆ is linearly de-pendent, find its rank, a basis, and the representation of each nonbasicvector in terms of that basis. Algorithm 1 is based on using the memorymatrix discussed in Section 1.12. Algorithm 2 operates directly on thelinear dependence relation for ∆ treated as a system of homogeneous

4.3 Check Linear Dependence 387

linear equations in the coefficients, and tries to find a nonzero solutionfor it using the method discussed in Section 1.23.

Algorithm 1: Checking Linear Dependence usingthe Memory Matrix

In this algorithm, row operartions involved in G or GJ pivot stepsare carried out. Because it is based on row operations, it is necessaryto enter each vector in ∆ as a row vector in the original tableau, eventhough the vectors in ∆ may be column vectors.

BEGIN

Step 1: Enter the vectors a1, . . . , ak in ∆ in that order, as rowvectors in the original tableau (if the vectors in ∆ are column vectors,you will enter their transposes as rows, but still call them by the samenames).

Set up a memory matrix (initialy unit matrix of order k) with itscolumns labeled a1, . . . , ak in that order, on the left side of the tableau.

Step 2: Try to perform GJ pivot steps (see Sections 1.15, 1.16),or G pivot steps with row interchanges as necessary (see Sections 1.19,1.20, these lead to a more efficient algorithm) in each row of the tableauin any order.

A pivot step is not carried out with a row r as pivot row only ifthat row has become the 0-row (i.e., all the entries on the right handportion of it in the current tableau are zero) when it has come up forconsideration as pivot row.

Step 3: If you are only interested in checking whether ∆ is linearlydependent and obtain a single linear dependence relation for it whenit is, then this algorithm can be terminated when a 0-row shows up inthe algorithm for the first time. Then the entries in the memory part(the left hand part) of the 0-row in the final tableau are the coefficientsfor a linear dependence relation for ∆.

If pivot steps are carried out in every row of the tableau (i.e., no—

388 Ch. 4 . Numerical Linear Algebra

0-rows in the final tableau, then ∆ is linearly independent, its rank isk, and ∆ is the basis for itself.

If only s < k pivot steps are performed, then rank of ∆ is s, thesubset {ai : i such that the i-th row in the final tableau is not a 0-row}(these are the vectors corresponding to rows in which pivot steps arecarried out) is a basis for ∆. Also, from the linear dependence relationof each 0-row, you can get the representation of the nonbasic variablecorresponding to that row in terms of the basis.

END

Example 1: Consider the set Γ = {A. 1, to A. 5 } of columnvectors in R6 of the matrix given below. We will find the rank and abasis for this set.

A.1 A.2 A.3 A.4 A.5

1 2 0 2 12 4 0 4 −1

−1 −2 3 1 −11 2 1 3 10 0 2 2 −2

−2 −4 2 −2 1

We enter each of the vectors A.1 to A.5 as a row in a original tableau(the top one among those given below), set up the memory matrix, andapply the algorithm. The various tableaus obtained are given below.We carry out G pivot steps, no row interchanges were necessary. PR(pivot row), PC (pivot column) are indicated in each tableau, and thepivot element is boxed. “0-row” indicates 0-rows discovered. In thebottommost tableau, PR, PC are indicated, but since PR there is thelast row, and the pivot step is the G pivot step, there is no work to bedone. Hence the bottommost tableau is the final tableau.

4.3 Check Linear D ep endence 389

Memory matrix TableauA. 1 A. 2 A. 3 A. 4 A. 5

1 0 0 0 0 1 2 −1 1 0 −2 PR0 1 0 0 0 2 4 −2 2 0 −40 0 1 0 0 0 0 3 1 2 20 0 0 1 0 2 4 1 3 2 −20 0 0 0 1 1 −1 −1 1 −2 1

PC↑1 0 0 0 0 1 2 −1 1 0 −2

−2 1 0 0 0 0 0 0 0 0 0 0-row

0 0 1 0 0 0 0 3 1 2 2 PR−2 0 0 1 0 0 0 3 1 2 2−1 0 0 0 1 0 −3 0 0 −2 3

PC↑1 0 0 0 0 1 2 −1 1 0 −2

−2 1 0 0 0 0 0 0 0 0 0 0-row0 0 1 0 0 0 0 3 1 2 2

−2 0 −1 1 0 0 0 0 0 0 0 0-row

−1 0 0 0 1 0 −3 0 0 −2 3 PRPC↑

The nonzero rows in the right-hand portion of the final (bottom-most) tableau are rows 1, 3, 5. Hence {A. 1, A. 3, A. 5 } is a basis for theoriginal set Γ in this example, and the rank of Γ is 3. From rows 2 and4 in the final tableau we have the linear dependence relations

−2A. 1 + A. 2 = 0 and −2A. 1 − A. 3 + A. 4 = 0.

So, A. 2 = 2A. 1 and A. 4 = 2A. 1 + A. 3.

So, the representations of the nonbasic vectors A. 2, A. 4 in terms ofthe basis {A. 1, A. 3, A. 5 } are (2, 0, 0)T and (2, 1, 0)T respectively.

Example 2: Consider the set Γ = {a1 = (1, 0,−1, 1), a2 =(−1,−1, 0, 1), a3 = (0, 1, 1,−1)} of row vectors in R4. We will find therank and a basis for this set. We enter each of these vectors as rows inthe original tableau (the top one among those given below), set up the

390 Ch. 4. Numerical Linear Algebra

memory matrix, and apply the algorithm using GJ pivot steps. Thevarious tableaus obtained are given below. PR (pivot row), PC (pivotcolumn) are indicated in each tableau, and the pivot element is boxed.The bottommost tableau is the final tableau.

a1 a2 a3

1 0 0 1 0 −1 1 PR0 1 0 -1 −1 0 10 0 1 0 1 1 −1

PC↑1 0 0 1 0 −1 1

1 1 0 0 −1 −1 2 PR0 0 1 0 1 1 −1

PC↑1 0 0 1 0 −1 1

−1 −1 0 0 1 1 −2

1 1 1 0 0 0 1 PRPC↑

0 −1 −1 1 0 −1 01 1 2 0 1 1 01 1 1 0 0 0 1

Pivot steps are carried out in every row of the tableau (i.e., no0-rows in the final tableau), therefore the set Γ here is linearly inde-pendent, and its rank is 3.

Algorithm 2: Checking Linear Dependence byRow Reduction of the System of Equations In theLinear Dependence Relation

BEGIN

Step 1: Let ∆ = {a1, . . . , ak} ⊂ Rn be the set of vectors whoselinear dependence is to be checked. This algorithm treats the lineardependence relation

α1a1 + . . . + αka

k = 0

4.3 Check Linear Dependence 391

as a system of n homogeneous equations in the unknown coefficientsα1, . . . , αk; and tries to find nonzero solutions to it by the algorithmdiscussed in Section 1.23 using GJ pivot steps. The system in detachedcoefficient tableau form is

Original tableauα1 . . . αj . . . αk

a1 . . . aj . . . ak 0

In this tableau, each of the vectors in ∆ is entered as a columnvector (if it is given originally as a row vector, enter its transpose). Forj = 1 to k, the vector aj is the column vector of the variable αj in theoriginal tableau.

Step 2: In the final tableau obtained at the end of the methodif every variable αj is a basic variable (i.e., if there are no nonbasicvariables at this stage), then α = (α1, . . . , αk) = 0 is the unique solutionfor the system. In this case ∆ is linearly independent, its rank is k,and a basis for ∆ is ∆ itself.

On the other hand, if there is at least one nonbasic variable inthe final tableau, the system has nonzero solutions, and ∆ is linearlydependent. Suppose there are p rows in the final tableau with αit asthe tth basic variable for t = 1 to p. Rearrange the variables and theircolumn vectors in the final tableau so that the basic variables appearin their proper order first, followed by nonbasic variables.

Final tableau after rearrangement of variablesBasic αi1 αi2 . . . αip αj1 . . . αjk−p

Vars.αi1 1 0 . . . 0 a1,j1 . . . a1,jk−p

0αi2 0 1 . . . 0 a2,j1 . . . a2,jk−p

0...

......

......

......

αip 0 0 . . . 1 ap,j1 . . . ap,jk−p0

Then the rank of ∆ is p = the number of pivot steps we were ableto carry out on the system.—

392 Ch. 4 . Numerical Linear Algebra

The basis (maximal linearly independent subset of vectors) of ∆identified by the method is {ai1 , . . . , aip } with ait as the tth basic vectorfor t = 1 to p. These correspond to columns of the tableau in whichpivot steps were carried out during the method.

For each u = 1 to k − p, by fixing the nonbasic variable αj u = 1,and all other nonbasic variables = 0; leads to the nonzero solution

αit = −at, j u , for t = 1 to p,

αj u = 1, and other nonbasic variables αj v = 0 for v �= u

for the linear dependence relation for ∆. From this we conclude thatthe nonbasic vector

aj u =p∑

t=1

at, j uait

Hence, for u = 1 to k− p, the representation of the nonbasic vectoraj u in terms of the basis (ai1 , . . . , a

ip) is (a1,ju, . . . , ap,ju).Hence, the column vector of the nonbasic variable αj u in the fi-

nal tableau, called its updated column, is the representation of itscolumn vector in the original tableau in terms of the basis obtained.

END

Example 3: Let ∆ be the set of column vectors of the matrixgiven below, with names as indicated there.

A.1 A.2 A.3 A.4 A.5

1 2 0 2 12 4 0 4 −1

−1 −2 3 1 −11 2 1 3 10 0 2 2 −2

−2 −4 2 −2 1

We will find a basis for ∆ using the above algorithm. The detachedcoefficient original tableau for the linear dependence relation is the

4.3 Check Linear Dependence 393

first tableau given below. We apply the GJ pivotal method on it,showing the BV (basic variables in each row), PR (pivot row), PC(pivot column), and enclosing the pivot element in a box in each step.RC indicates redundant constraint to be eliminated.

BV α1 α2 α3 α4 α5

1 2 0 2 1 0 PR2 4 0 4 −1 0

−1 −2 3 1 −1 01 2 1 3 1 00 0 2 2 −2 0

−2 −4 2 −2 1 0PC↑

α1 1 2 0 2 1 0

0 0 0 0 −3 0 PR0 0 3 3 0 00 0 1 1 0 00 0 2 2 −2 00 0 2 2 3 0

PC↑α1 1 2 0 2 0 0α5 0 0 0 0 1 0

0 0 3 3 0 0 PR0 0 1 1 0 00 0 2 2 0 00 0 2 2 0 0

PC↑α1 1 2 0 2 0 0α5 0 0 0 0 1 0α3 0 0 1 1 0 0

0 0 0 0 0 0 RC0 0 0 0 0 0 RC0 0 0 0 0 0 RC

α1 1 2 0 2 0 0α5 0 0 0 0 1 0α3 0 0 1 1 0 0

394 Ch. 4. Numerical Linear Algebra

Final tableau with variables rearrangedBV α1 α5 α3 α2 α4

α1 1 0 0 2 2 0α5 0 1 0 0 0 0α3 0 0 1 0 1 0

The basic variables are (α1, α5, α3) in that order; and α2, α4 arethe nonbasic variables. So, ∆ is linearly dependent, its rank is 3, and{A.1, A.5, A.3} is a basis (maximal linearly independent subset) for it.

Fixing the nonbasics α2 = 1, α4 = 0 leads to the solution

(α1, α5, α3, α2, α4) = (−2, 0, 0, 1, 0)

to the linear dependence relation. Therefore

−2A.1 + A.2 = 0 or A.2 = 2A.1

Similarly fixing the nonbasics α4 = 1, α2 = 0 leads to the solution

(α1, α5, α3, α2, α4) = (−2, 0,−1, 0, 1)

to the linear dependence relation. Therefore

−2A.1 − A.3 + A.4 = 0 or A.4 = 2A.1 + A.3

Hence the representations of the nonbasic vectors A.2, A.4 in terms ofthe basis (A.1, A.5, A.3) are (2, 0, 0)T and (2, 0.1)T respectively; and weverify that these are the column vectors of the corresponding variablesα2, α4 in the final tableau.

4.4 How to Get an Alternate Basis by In-

troducing a Nonbasic Vector Into a

Basis ?

Let ∆ = {a1, . . . , ak} be a set of vectors in Rn whose rank is r < k,and let B = {a1, . . . , ar} be a basis for it.

4.4 Enter a Nonbasic Vector Into a Basis 395

In some applications, it is important to find out whether there areother subsets of ∆ which are also bases for it; and if so, how one canobtain them starting with the known basis B. A procedure calledintroducing a nonbasic vector into a basis can be used to do this.We will now discuss this procedure.

Any nonbasic vector which is nonzero can be introduced into a basis.For the operation of introducing the nonbasic vector ar +1 �= 0 into thebasis B, ar +1 is called the entering vector.

By Result 4.2.2 of Section 4.2, every basis for ∆ always containsthe same number of vectors. So, when ar +1 is introduced into B, oneof the present basic vectors in B must be deleted from it, this basicvector is called the dropping basic vector in this operation.

The answer to the question: which basic vector in B can the enteringvector ar +1 replace? is contained in the representation of ar +1 in termsof B. Suppose this representation is (α1, . . . , αr), i.e.,

ar +1 = α1a1 + . . . + αra

r.

Then ar +1 can replace any of the basic vectors aj for which αj �= 0,to lead to a new basis for ∆.

Example 1: As an example, consider the set Γ = {A. 1, to A. 5 }of column vectors in R6 of the matrix given below, from the examplesdiscussed in Section 4.3.

A.1 A.2 A.3 A.4 A.5

1 2 0 2 12 4 0 4 −1

−1 −2 3 1 −11 2 1 3 10 0 2 2 −2

−2 −4 2 −2 1

In Section 4.3, we found the basis B1 = {A.1, A.3, A.5} for this set∆, and that the representations of the nonbasic vectors in terms of thisbasis are:

396 Ch. 4 . Numerical Linear Algebra

A. 2 representation is (2, 0, 0); i.e., A. 2 = 2A. 1

A. 4 representation is (2, 1, 0); i.e., A. 4 = 2A. 1 + A. 3

Therefore, the nonbasic vector A. 2 can only replace the basic vectorA. 1 from the basis B1 (because A. 1 is the only basic vector with anonzero coefficient in the representation of A. 2), leading to anotherbasis B2 = {A. 2, A. 3, A. 5 } for ∆.

This implies that if we replace any of the basic vectors A. 3, A. 5 fromthe basis B1 by A. 2, we will get subsets of vectors which are linearlydependent, and hence are not bases. In fact, we can easily verify thatthe subsets {A. 1, A. 2, A. 5 }, {A. 1, A. 3, A. 2 } are both linearly dependentbecause −2A. 1 + A. 2 = 0 is a linear dependence relation for them.

In the same way, we conclude that the nonbasic vector A. 4 canreplace either of the basic vectors A. 1, A. 3 from the basis B1, but notA. 5. Hence, B3 = {A. 4, A. 3, A. 5 }, B4 = {A. 1, A. 4, A. 5 } are also basicvectors for ∆.

Exercises

4.4.1: Find the rank of the following set of vectors Γ, a basis for thisset, representation of each nonbasic vector in terms of this basis; anddetermine for each nonbasic vector which basic vectors it can replaceto lead to another basis: (i): Γ = {(1, 0,−1, 2, 1), (−1, 1, 2,−1, 0),(3,−1,−4, 5, 2), (1, 1, 1, 0, 2), (1, 2, 2, 1, 3)} (ii): Γ = {(1, 1,−1, 2, 1),(2,−1, 2, 1, 2), (4, 1, 0, 5, 4), (0, 0, 1, 0, 0), (7, 1, 2, 8, 7)}.

4.4.2: Show that the following sets of vectors are linearly indepen-dent: i: {(0, 1,−1, 0, 1), (1,−1, 1, 1, 1), (1, 1, 1, 2,−1), (0, 0, 1, 3, 2),(0, 0,−1,−1,−1)} (ii): {(2,−2, 2, 1), (−2, 3,−3,−1), (1,−2, 1, 0),(0, 2, 2, 2)} (iii): {(3, 1, 4), (7, 2, 6), (−2, 0, 5)}.

4.4.3: If the set {A.1, A.2, A.3} is linearly independent, show thatthe set {A.2+A.3, A.3+A.1, A.1+A.2} must also be linearly independent.Is the converse also correct?

4.5 R ank o f a Ma trix 397

Let y 1 = A. 1 + A. 2 + A. 3, y

2 = A. 1 + αA. 2, y 3 = A. 2 + βA. 3. If

{A. 1, A. 2, A. 3 } is linearly independent, what condition should α, β sat-isfy so that {y

1, y2, y3 } is also linearly independent?

4.4.4: Given a linearly independent set of vectors {A. 1, A. 2, . . . , A.m }in R

n, and another vector A.m+1 in Rn show that {A. 1, A. 2, . . . , A.m, A.m+1 }is linearly independent iff A.m+1 is not in the linear hull of {A.1, A. 2, . . . , A.m }.

4.5 The Rank of a Matrix

Let A = (aij) be a matrix of order m×n. In our notation, {A1., . . . , Am. }is the set of row vectors of A. The rank of this set is called the rowrank of matrix A.

{A. 1, . . . , A.n } is the set of column vectors of A. The rank of thisset of vectors is called the column rank of matrix A.

Result 4.5.1: Row rank and column rank are the same: Forany matrix A, its row rank is always equal to its column rank. Theircommon value is called the rank of matrix A. If A is of order m × nand rank r, then r ≤ m and r ≤ n.

Historical note on the rank of a matrix: The term rank of amatrix A was introduced in the 1870s by the German mathematicianF. G. Frobinius, as the order of the largest order square submatrixof A with a nonzero determinant. His definition of rank turns outto be equivalent to the number of vectors in any maximal linearlyindependent subset of column (or row) vectors of A, or a minimalspanning subset of these vectors.

The rank of a matrix A can be computed by applying either of thealgorithms discussed in Section 4.3 to find the rank of the set of row(or column) vectors of A.

Example 1: The Row Reduction Method for finding the

398 Ch. 4. Numerical Linear Algebra

rank of a matrix, and bases for its column space using GJpivot steps.

This is exactly Algorithm 2 discussed in Section 4.3. Let A be thematrix whose rank we want to find. Put A in a tableau form, andlabel its column vectors by their names A.1, A.2, etc. in a row at thetop. Try to perform a GJ pivot step in each row of this tableau, oneafter the other, say from top to bottom (any other order would be OKtoo). When a pivot step is performed in a row i, if the pivot columnfor that pivot step is A.j, then record A.j as the basic column vectorselected in this step in row i. If a zero row is encountered, move to thenext row, and continue the same way until each row is either the zerorow or had a pivot step performed in it. Then rank of A is the totalnumber of pivot steps performed in this algorithm, and the final tableaucontains a basis for the column space of A, and the representation ofeach nonbasic column in terms of this basis.

We will apply this method on the following matrix A of order 6×5.For each pivot step PR, PC indicate the pivot row, column respectivelyand the pivot element is enclosed in a box.

A =

1 2 0 2 12 4 0 4 −1

−1 −2 3 1 −11 2 1 3 10 0 2 2 −2

−2 −4 2 −2 1

Basic Vector A.1 A.2 A.3 A.4 A.5

1 2 0 2 1 PR2 4 0 4 −1

−1 −2 3 1 −11 2 1 3 10 0 2 2 −2

−2 −4 2 −2 1PC↑

4.5 Rank of a Matrix 399

Basic Vector A.1 A.2 A.3 A.4 A.5

A.5 1 2 0 2 1

3 6 0 6 0 PR0 0 3 3 00 0 3 3 00 0 1 1 02 4 2 6 0

−3 −6 2 −4 0PC↑

A.5 0 0 0 0 1A.1 1 2 0 2 0

0 0 3 3 0 PR0 0 1 1 00 0 2 2 00 0 2 2 0

PC↑Basic vector Final Tableau

A.1 A.2 A.3 A.4 A.5

A.5 0 0 0 0 1A.1 1 2 0 2 0A.3 0 0 1 1 0

0 0 0 0 00 0 0 0 00 0 0 0 0

We have labeled the updated columns in the final tableau by thesymbols A.j. Since a total of three pivot steps were carried out, therank of matrix A is three. The basis obtained for the column space ofA is (A.5, A.1, A.3). In the final tableau each basic vector is transformedinto the appropriate unit vector. For each nonbasic column vector A.j ,when the zero entries in zero rows are deleted from its updated columnA.j, we get the representation of A.j in terms of the basis. Thus, therepresentation of

A.2 is (0, 2, 0), i.e., A.2 = 2A.1

400 Ch. 4. Numerical Linear Algebra

A.4 is (0, 2, 1), i.e., A.4 = 2a.1 + A.3.

In linear programming literature, the final tableau with the basiccolumn vectors transformed into the various unit vectors is called thecanonical tableau WRT the selected basis.

To get an alternate basis for the column space of this matrix, we canselect any nonbasic column as an entering column in the final tableau,and select any nonzero entry in its updated column as the pivot elementand the row of the pivot element as the pivot row. The entering columnreplaces the basic column recorded in the pivot row leading to a newbasis for the column space of the matrix, Performing the pivot stepleads to the canonical tableau WRT the new basis.

Relationship Of Nonsingularity to Rank

Let A be a square matrix of order n. According to the definitiongiven in Section 2.8, A is nonsingular iff its determinant is �= 0; singularotherwise. The following result relates this property to the rank of A.

Result 4.5.2: Rank and nonsingularity: The square matrix Aof order n is nonsingular iff its rank is n, singular iff its rank is ≤ n−1.

If rank of A is ≤ n − 1, then one of the row vectors of A can beexpressed as a linear combinations of the other row vectors of A. Byresults 2, 3 of Section 2.8 this implies that the determinant of A is 0,i.e., A is singular.

If rank of A is n, then in the algorithm discussed above for comput-ing the rank, pivot steps can be carried out in every row. This impliesthat in the algorithm for computing the determinant of A discussedin Section 2.8, the number of G pivot steps that can be carried out isexactly n. Therefore the determinant of A is = ± (the product of theboxed pivot elements), which is nonzero because every pivot elementis nonzero, i.e., A is nonsingular.

Note: In some linear algebra books a square matrix of order n isalternately defined as being nonsingular if its rank is n, singular if its

4.5 R ank o f a Ma trix 401

rank is ≤ n − 1.

Other Meanings for the Word “Basis”

Let A be a matrix of order m×n and rank m. Then A is said to bea matrix of full row rank. Speaking in terms of the column space ofsuch a matrix A, we defined a basis to be a linearly independent subsetof m column vectors of A.

In linear programming literature, the word basis is often used todenote a submatrix of A of order m × m consisting of a linearly inde-pendent subset of m column vectors of A. Thus, if the set of columnvectors, {A. 1, . . . , A.m } of A form a linearly independent set, then inour notation this set of column vectors constitutes a basis for the col-umn space of A. In linear programming literature they would use theword basis to denote the submatrix of A

B = (A. 1... . . .

...A.m)

consisting of these column vectors.

Exercises

4.5.1: If A, B are square matrices of order n satisfying AB = BA

and rank(AB) = n, show that rank

(AB

)= n also.

4.5.2: Find the ranks of the following matrices, a basis for itscolumn space, and an expression for each nonbasic column as a linearcombination of basic vectors.

1 1 −1 2 0−1 −2 3 −3 2−1 −4 7 −5 6

0 1 −1 2 20 1 0 3 6

,

2 −1 3 2−1 1 −4 −1

4 −1 1 41 0 2 7

,

402 Ch. 4. Numerical Linear Algebra

2 −4 6 8 10−4 9 −13 0 −21

6 −13 19 8 310 −1 2 2 1

.

4.6 Computing the Inverse of a Nonsin-

gular Square Matrix Using GJ Pivot

Steps

As defined in Section 2.8, among matrices, the concept of an inverseapplies only to square matrices, and it only exists for nonsingular squarematrices. The inverse is not defined for rectangular matrices which arenot square. When the inverse of a matrix exists, it is always unique. Ifsquare matrix A has an inverse, it is denoted by the symbol A−1 andnot as 1/A.

The definition of the inverse of a square matrix when it exists, isanalogous to the definition of the inverse of a nonzero real number γas the number δ satisfying γδ = 1. Among square matrices of ordern, the unit or identity matrix I of order n, plays the role of the realnumber 1. And so, for a square matrix A, its inverse is defined to bethe square matrix D of order n satisfying

DA = AD = I

when it exists, which happens only if A is nonsingular, it is denoted bythe symbol A−1.

Inverses of Some Special Matrices

Consider the unit matrix I of order n. Its inverse is itself, i.e.,I−1 = I.

Consider a permutation matrix P , which is a matrix obtained fromthe unit matrix by permuting its row vectors. For example

4.6 Pivotal Method for Computing the Matrix Inverse 403

P =

0 0 1 01 0 0 00 0 0 10 1 0 0

is a permutation matrix of order 4. It can be verified that P T P =PP T = I. The same thing holds for all permutation matrices. Hencefor every permutation matrix P , we have P−1 = P T .

Consider a diagonal matrix D with diagonal entries d1, . . . , dn. Bydefinition dj �= 0 for all j. It can be verified that D−1 is given by thefollowing.

D =

d1 0 . . . 00 d2 . . . 0...

......

0 0 . . . dn

, D−1 =

1/d1 0 . . . 00 1/d2 . . . 0...

......

0 0 . . . 1/dn

.

Hence the inverse of diag(d1, . . . , dn) is diag(1/d1, . . . , 1/dn).

How To Compute the Inverse of a General Non-singular Matrix Efficiently?

In Section 2.8 we derived a formula for the inverse of a general non-singular square matrix in terms of its cofactors. That formula pointsout the theoretical relation between the inverse of a matrix, and the de-terminants of its submatrices. It points out the many different placesin matrix algebra where determinants play an important theoreticalrole. However, that formula is not practically efficient for computingthe inverse. We will now discuss a much more efficient method forcomputing the inverse using GJ pivot steps.

Let A = (aij) be a square nonsingular matrix of order n. A1., . . . , An.

denote the various row vectors of A. Let I be the unit matrix of ordern, and I1., . . . , In. are its row vectors (these are the various unit rowvectors in Rn). Suppose

404 Ch. 4. Numerical Linear Algebra

A−1 =

β11 . . . β1n...

...βn1 . . . βnn

Then from the definition of A−1 we have

A−1A =

β11 . . . β1n...

...βn1 . . . βnn

A = I

From the definition of matrix products, this implies that for each i= 1 to n

(βi1, . . . , βin)A = βi1A1. + . . . + βin = Ii.

i.e., the ith row vector of A−1 is the vector of coefficients in an ex-pression of Ii. as a linear combination of the row vectors of A. Thisobservation is the key behind the algorithm for computing A−1 usingrow operations.

The algorithm begins with a tableau with n rows which are the rowsof A; A1., . . . , An.. As discussed in Section 1.12 we set up a memorymatrix on the left of this tableau to maintain the expression of each rowin the current tableau as a linear combination of rows in the originaltableau. When we carry out row operations to transform the rowsof the tableau into the unit rows I.1, . . . , I.n, the memory matrix getstransformed into A−1.

We now provide a statement of various steps in the algorithm.When applied on a general square matrix A of order n, the algorithmterminates by either producing A−1, or a linear dependence relation forthe set of row vectors of A showing that A is singular, and hence A−1

does not exist.

4.6 Pivotal Method for Computing the Matrix Inverse 405

Algorithm to Compute the Inverse of a SquareMatrix A Using GJ Pivot Steps

BEGIN

Step 1: Start with the following initial tableau, where I is the unitmatrix of the same order as A.

Memory Matrix Original Tableau

I A

We will perform GJ pivot steps in each row of the tableau in someorder (say top to bottom). Select the first row for pivoting, as the pivotrow in the initial tableau and go to Step 2.

Step 2: General Step: Suppose row r is the pivot row in thepresent tableau.

If all the entries in row r of the present tableau are zero, A is singularand A−1 does not exist, terminate. If the entries in the memory matrixin this row are (βr1, . . . , βrn), then

βr1A1. + . . . + βrnAn. = 0

is a linear dependence relation for the set of row vectors of A.If there are nonzero elements in row r of the present tableau, select

one of them as the pivot element, and its column as the pivot column,and perform the GJ pivot step. With the new tableau obtained afterthis pivot step, move to Step 3 if GJ pivot steps have been performedin all the rows; or repeat Step 2 to perform a GJ pivot step in the nextrow for pivoting in it.

Step 3: Final Step: The present tableau at this stage contains apermutation matrix P . Rearrange the rows of the present tableau so—

406 Ch. 4 . Numerical Linear Algebra

that P becomes the unit matrix (this is equivalent to multiplying thepresent tableau on the left by P

T ) leading to the final tableau. Thesquare matrix contained in the memory matrix portion of the finaltableau is A− 1, terminate.

END.

Example 1: As an example, we will compute the inverse of thefollowing matrix A.

A =

0 −2 −11 0 −11 1 0

.

We apply the algorithm discussed above. The pivot element in eachstep is enclosed in a box, and PR, PC indicate pivot row, pivot columnrespectively..

Memory Matrix Tableau

1 0 0 0 −2 −1 PR0 1 0 1 0 −10 0 1 1 1 0

PC↑−1/2 0 0 0 1 1/2

0 1 0 1 0 −1 PR1/2 0 1 1 0 −1/2

PC↑−1/2 0 0 0 1 1/2

0 1 0 1 0 −1

1/2 −1 1 0 0 1/2 PR

PC↑−1 1 −1 0 1 0

1 −1 2 1 0 01 −2 2 0 0 1

4.6 Pivotal Metho d for Computing the Ma trix Inverse 407

Final TableauMemory Matrix Tableau

1 −1 2 1 0 0−1 1 −1 0 1 0

1 −2 2 0 0 1

Hence

A− 1 =

1 −1 2−1 1 −1

1 −2 2

It can be verified that AA− 1 = A− 1A = I, the unit matrix of order3.

Example 2: As another example, we will find the inverse of

the following matrix A. The algorithm is applied below, boxing thepivot element in each step (PR, PC indicate pivot row, pivot columnrespectively).

A =

1 0 −2−1 1 1

0 1 −1

Memory Matrix Tableau

1 0 0 1 0 −2 PR0 1 0 −1 1 10 0 1 0 1 −1

PC↑1 0 0 1 0 −2

1 1 0 0 1 −1 PR0 0 1 0 1 −1

PC↑1 0 0 1 0 −21 1 0 0 1 −1

−1 −1 1 0 0 0

408 Ch. 4 . Numerical Linear Algebra

Since the last row has all zero entries in the tableau part, we con-clude that A is singular and hence has no inverse. From the memorymatrix part of the last row we obtain the linear dependence relationfor the set of row vectors of A as

−A1. − A2. + A3. = 0.

Computational Effort Needed To Find the In-verse

A matrix is said to be a:

sparse matrix if a large proportion of the entries in it arezero

dense matrix otherwise.

If the various tableaus obtained in the inverse finding algorithm aredense, the method described above for finding the inverse needs about2n3 arithmetic operations.

Exercises

4.6.1: Find the inverses of the following nonsingular matrices.

(2 84 −18

),

(13 −1

−90 7

),

0 2 −41 0 2

−2 −1 0

,

2 −1 −3−5 2 8−3 1 6

,

−2 −4 1−5 −8 2

3 5 −1

,

0 2 2 22 0 2 22 2 0 22 2 2 0

,

1 0 1 −1−1 1 −2 2

0 −1 2 −20 0 −1 2

,

1 −1 −1 1−2 3 3 −1

1 1 1 0−1 0 0 1

.

4.7 Product Form of the Inverse 409

4.6.2: Apply the inverse finding algorithm on the following matricesand show that they are singular. For each matrix, obtain a lineardependence relation for its row vectors.

(2 48 16

),

( −12 −3−60 −15

),

0 2 −42 0 24 2 0

,

3 6 −9−3 −8 7−3 −12 3

,

0 5 −15 202 0 4 −60 −5 0 −154 20 −7 53

,

0 2 6 83 0 9 66 2 24 203 −2 3 −2

,

4.7 Pivot Matrices, Elementary Matrices,

Matrix Factorizations, and the Prod-

uct Form of the Inverse

Consider a pivot step on a matrix A = (aij) of order m × n, with rowr as the PR (pivot row), and column s as the PC (pivot column). So,the pivot element ars, boxed in the following, is �= 0.

Matrix A A = Matrix after pivot stepa11 . . . a1s . . . a1n a11 . . . 0 . . . a1n...

......

......

...ar−1,1 . . . ar−1,s . . . ar−1,n ar−1,1 . . . 0 . . . ar−1,n

ar1 . . . ars . . . arn PR ar1 . . . 1 . . . arn

ar+1,1 . . . ar+1,s . . . ar+1,n ar1 . . . 0 . . . ar+1,n...

......

......

...am1 . . . ams . . . amn am1 . . . 0 . . . amn

PC

where

arj = arj/ars for j = 1 to n

aij = aij − (arj/ars)ais for i �= r, j = 1 to n.

410 Ch. 4 . Numerical Linear Algebra

This pivot step has transformed the pivot column into the rth unitcolumn (r is the number of the pivot row). Now consider the squarematrix P of order m × m which is the same as the unit matrix exceptfor its rth column, known as its eta column; obtained by carrying outon the unit matrix I of order m, exactly the same row operations inthe pivot step carried out on the matrix A.

P =

1 0 . . . 0 −a1s/ars 0 . . . 00 1 . . . 0 −a2s/ars 0 . . . 0...

......

......

...0 0 . . . 1 −ar − 1,s/ars 0 . . . 00 0 . . . 0 1/ars 0 . . . 00 0 . . . 0 −ar +1,s/ars 1 . . . 0...

......

......

...0 0 . . . 0 −ams/ars 0 . . . 1

rth col.eta col.

It can be verified that A = PA. Thus performing the pivot stepon the matrix A of order m × n with row r as the pivot row, columns as the pivot column; is equivalent to multiplying A on its left (i.e.,premultiplying A) by the square matrix P of order m × m that differsfrom the unit matrix in just its rth column (called its eta column).That’s why this matrix P is called the pivot matrix corresponding tothis pivot step. The eta column in P is derived from the data in thepivot column, and its position in P is the same as the position of thepivot row.

Example 1: Consider the following matrix A of order 5× 6 andthe pivot step on it with the PR (pivot row) and PC (pivot column)as the row and column of the boxed pivot element.

A =

9 1 0 −2 8 2−7 −1 −2 0 −6 0−2 0 0 4 2 3

0 2 4 6 2 −2−1 0 2 −2 −4 1

4.7 Product Form of the Inverse 411

We put I, the unit matrix of order 5 on the left hand side of A andcarry out all the row operations in this pivot step on it too. After thispivot step, the unit matrix becomes the pivot matrix P and the matrixA becomes A.

Initial tableau1 0 0 0 0 9 1 0 −2 8 20 1 0 0 0 −7 −1 −2 0 −6 00 0 1 0 0 −2 0 0 4 2 3

0 0 0 1 0 0 2 4 6 2 −20 0 0 0 1 −1 0 2 −2 −4 1

After the pivot step

P A1 0 0 −4 0 9 −7 −16 −26 0 100 1 0 3 0 −7 5 10 18 0 −60 0 1 −1 0 −2 −2 −4 −2 0 50 0 0 1/2 0 0 1 2 3 1 −10 0 0 2 1 −1 4 10 10 0 −3

The pivot matrix corresponding to this pivot step is the 5×5 matrixin the above tableau, and it can be verified that A = PA. Notice thatthe eta column in P is its 4th column, its number is the same as thatof the pivot row for this pivot step.

Thus carrying out a pivot step on a matrix of order m×n is equiva-lent to premultiplying it by the corresponding pivot matrix (which canbe derived from the formula given above, using the pivot column) oforder m.

Since the pivot element for every pivot step will be nonzero, a pivotmatrix is always square and nonsingular. Also, since a pivot matrixalways differs from the unit matrix in just one column (its eta column),to store the pivot matrix, it is sufficient to store its eta column, andthe position of this eta column in the pivot matrix. Thus even thougha pivot matrix is a square matrix, it can be stored very compactly andrequires almost the same storage space as a column vector.

Pivot matrices belong to a special class of square matrices known

412 Ch. 4 . Numerical Linear Algebra

as elementary matrices. The main elementary matrices are definedas square matrices obtained by performing a single elementary rowoperation (or a similarly defined column operation) on a unit matrix.From Sections 1.11, 1.20, we recall that there are three elementary rowoperations (and correponding column operations) which are:

1. Scalar multiplication: Multiply each entry in a row (or column)by the same nonzero scalar.

2. Add a scalar multiple of a row to another row (or of a colmn toanother column).

3. Row (or column) interchange: Interchange two rows (or columns)in the matrix.

and there is an elementary matrix associated with each of these opera-tions. We show these matrices for elementary row operations first, andfor column operations next. We use the following notation. A denotesthe original matrix of order m × n on which the elementary operationis carried out, Ai the matrix obtained after the ith operation on A,and Ei the matrix obtained by carrying out the same ith operation onthe unit matrix of order m for row operations (or order n for columnoperations). Then Ei is the elementary matrix associated with the ithoperation, and

Ai = EiA for row operations

Ai = AEi for column operations.

Thus carrying out a row (column) operation on a matrix is equiv-alent to premultiplying (postmultiplying) it by the associated elemen-tary matrix. For illustrative examples, we, will use

A =

1 −3 −5 6−1 4 −2 3

2 5 7 8

Example 2: Scalar multiplication of a row (row 2) of A byscalar α (α = 10), leads to

4.7 Pro duct Form o f t he Inverse 413

A1 =

1 −3 −5 6−10 40 −20 30

2 5 7 8

, E1 =

1 0 00 10 00 0 1

and it can be verified that A1 = E1A. E1 is the elementary matrixassociated with this operation.

Example 3: Adding a scalar multiple of a row of A to anotherrow: We will consider adding (−2) times row 3 of A to row 1. Thisleads to

A2 =

−3 −13 −19 −10−1 4 −2 3

2 5 7 8

, E2 =

1 0 −20 1 00 0 1

and it can be verified that A2 = E2A. E2 is the elementary matrixassociated with this operation.

Example 4: Interchanging two rows of A : We will considerinterchanging rows 1 and 3. This leads to

A3 =

2 5 7 8−1 4 −2 3

1 −3 −5 6

, E3 =

0 0 10 1 01 0 0

and it can be verified that A3 = E3A. E3 is the elementary matrixassociated with this operation.

Example 5: Scalar multiplication of a column (column 4) ofA by scalar α (α = −5), leads to

A4 =

1 −3 −5 −30−1 4 −2 −15

2 5 7 −40

, E4 =

1 0 0 00 1 0 00 0 1 00 0 0 −5

and it can be verified that A4 = AE4. E4 is the elementary matrixassociated with this operation.

414 Ch. 4 . Numerical Linear Algebra

Example 6: Adding a scalar multiple of a column of A toanother column: We will consider adding 3 times column 4 of A tocolumn 2. This leads to

A5 =

1 15 −5 6−1 13 −2 3

2 29 7 8

, E5 =

1 0 0 00 1 0 00 0 1 00 3 0 1

and it can be verified that A5 = AE5. E5 is the elementary matrixassociated with this operation.

Example 7: Interchanging two columns of A : We will considerinterchanging columns 1 and 4. This leads to

A6 =

6 −3 −5 13 4 −2 −18 5 7 2

, E6 =

0 0 0 10 1 0 00 0 1 01 0 0 0

and it can be verified that A6 = AE6. E6 is the elementary matrixassociated with this operation. ��

These are the elementary matrices normally discussed in linear al-gebra textbooks, but numerical analysts include a much larger class ofsquare matrices under the name elementary matrices. In this generaldefinition, an elementary matrix of order n is any square matrix ofthe form

E = I − αuvT

where I is the unit matrix of order n, α is a scalar, and u = (u1, . . . , un)T ,v = (v1, . . . , vn)T are column vectors in Rn.

Recall from Sections 1.5, 2.2, 2.5; when u, v are column vectors inRn, uTv is a scalar known as the dot (or inner) product of u, v; whileuvT is the square matrix of order n given below, known as the outer(also cross) product of u, v.

4.7 Product Form of the Inverse 415

uvT =

u1v1 u1v2 . . . u1vn

u2v1 u2v2 . . . u2vn...

......

unv1 unv2 . . . unvn

In uvT each column vector is a scalar multiple of u, and each rowvector is a scalar multiple of vT ; and hence when u, v are both nonzero,uvT is a matrix of rank one (or a rank-one matrix). I − αuvT istherefore a rank-one modification of the unit(identity) matrix. In thisgeneral definition, an elementry matrix is a rank one modification ofthe unit matrix, and it includes all pivot matrices.

The special property of the elementary matrix E = I−αuvT is thateven though it is a square matrix of order n, it can be stored compactlyby storing u, v, and the scalar α; and that computing the product Exfor any column vector x in Rn can be carried out using u, v, α withoutexplicitly computing all the entries in E because

Ex = (I − αuvT )x = x − αu(vTx) = x − γu

where γ = α(vT x), which is α times the dot product of v and x. In thesame way, for any row vector π ∈ Rn, the product πE can be computedusing u, v, α without explicitly computing all the entries in E.

Product Form of the Inverse

The method for computing the inverse described in Section 4.6 isfine for computing the inverses of matrices of small orders like 3, 4 byhand, or less than about 100 on the computer; but implementing it tocompute the inverses of matrices of large orders (greater than severalhundreds) on a digital computer is beset with many serious difficulties.Some of these are:

The Problem of Fill-in: Matrices encountered in real world ap-plications are very very sparse (i.e., most of the entries in them are 0)so they can be stored very compactly by storing the nonzero entries

416 Ch. 4. Numerical Linear Algebra

only with their positions. It has been observed that the explicit inverseof such a matrix is almost always fully dense. This is called the fill-inproblem. Storing the dense inverse matrix of a large order needs a lotof computer memory space.

Increased Computer Time: Performing any operations with adense explicit inverse matrix takes a lot of computer time.

Round-off Error Accumulation: Digital computation introducesrounding errors which accumulate during the algorithm, and may resultin serious errors in the computed explicit inverse.

To alleviate these problems, numerical analysts have developed sub-stitute forms for the explicit inverse using what are called matrix fac-torizations. Several of these used in computational procedures (suchas LU factorization, LDLT factorization, etc.) are beyond thescope of this book. But, we will discuss the simplest among theseknown as the product form of the inverse to give the reader a fla-vor of matrix factorizations. The interested reader should consult theexcellent book Gill, Murray and Wright [4.1].

The product form of the inverse is based on the algorithm for com-puting the inverse using GJ pivot steps discussed in Section 4.6, but itdoes not obtain the explicit inverse. Instead, it saves the pivot matricescorresponding to the various pivot steps in that algorithm, as a stringin the order in which they are generated.

To find the inverse of a square matrix A of order n, n GJ pivotsteps are needed in that algorithm. Let

Pr = the pivot matrix corresponding to the rth pivotstep in that algorithm, for r = 1 to n,

and P = the permutation matrix into which A is reducedby these pivot steps.

Then from the discussion above, we know that

A−1 = P TPnPn−1 . . . P1.

4.7 Product Form of the Inverse 417

The product on the right has to be carried out exactly in the orderspecified, as matrix multiplication is not commutative. A−1 is notexplicitly computed, but P1, . . . , Pn, P T are stored, and the formula onthe right of the above equation can be used in place of A−1 whereverA−1 is needed. For instance consider solving the square nonsingularsystem of equations

Ax = b

for some RHS vector b. If the explicit inverse A−1 is known, thenmultiplying both sides of the above equation by A−1 on the left we get

A−1Ax = A−1b, i.e., Ix = x = A−1b

yielding the solution of the system. However, if we are given the inversein product form as above, then the solution of the system is

P T PnPn−1 . . . P1b.

This product can be computed very efficiently by a scheme known asright to left string multiplication. In this scheme, we first computeq1 = P1b, which is a column vector. Then multiply it on the left by P2

to get q2 = P2q1, and in general for r = 2 to n we compute

qr = Prqr−1

in this specific order r = 2 to n. When qn is obtained, one morepremultiplication P T qn = x yields the solution to the system.

Because the eta vectors in the various pivot matrices tend to pre-serve sparcity, using the product form of the inverse alleviates the prob-lem of fill-in, at least for matrices of orders upto several hundreds; andsome of the round-off error accumulation problem. Because of this,computing P TPnPn−1 . . . P1b through right to left string multiplicationusually takes much less computer time than computing A−1b with theexplicit inverse A−1.

Algorithm to Compute the Inverse in ProductForm

418 Ch. 4. Numerical Linear Algebra

Let A be the matrix of order n×n whose inverse we need to compute.In this algorithm we do not use any memory matrix, in fact we do noteven use the updated tableaus.

The algorithm goes through n pivot steps (one in each column, onein each row). At any stage we denote by C, R the sets of columns,rows in which pivot steps have been carried out already. In each stepthe next pivot column is computed and a pivot element is selected inthis column by the partial pivoting strategy. Then the pivot matrixcorresponding to this pivot step is generated and added to the string,and the method moves to the next step.

BEGIN

Initial Step: Select any column, say A.v1 = (a1,v1 , . . . , an,v1)T of

the original matrix A as the pivot column for this step.If A.v1 = 0, A−1 does not exist, terminate.If A.v1 �= 0, select one of the elements of maximum absolute value

in A.v1 as the pivot element, suppose it is au1,v1 . Then row u1 is thepivot row for this step. Generate the pivot matrix corresponding tothe pivot step with A.v1 as the pivot column, row u1 as the pivot row,and store this matrix as P1. C = {v1}, R = {u1}. Also store the pivotposition (u1, v1). Go to the next step.

General Step: Suppose r pivot steps have been carried out so far.Let C = {v1, . . . , vr}, R = {u1, . . . , ur} be the sets of columns, rows,in which pivot steps have been carried out so far, and let {(u1, v1), . . . ,(ur, vr)} be the set of positions of pivot elements in these pivot steps.Let P1, . . . , Pr be the string of pivot matrices generated so far, in theorder they are generated.

Select a column vector of A in which a pivot step has not beencarried out so far (i.e., one from {1, . . . , n}\C) for the next pivot step,suppose it is A.vr+1. A.vr+1 is the column in the original matrix A, thepivot column, A.vr+1 will be the updated column corresponding to it(i.e., the vr+1th column in the present tableau obtained by performingall the previous pivot steps on A). From previous discussion we seethat this is given by—

4.7 Product Form of the Inverse 419

A.vr+1 = PrPr−1 . . . P1A.vr+1

This can be computed quite efficiently by computing the producton the right using right to left string multiplication. Having obtainedA.vr+1 = (a1,vr+1, . . . , an,vr+1)

T , if

aivr+1 = 0 for all i ∈ {1, . . . , n}\R, then

A.vr+1 − (r∑

t=1

aut,vr+1A.vt) = 0

is a linear dependence relation satisfied by the columns ofA, so A−1 does not exist, terminate.

If ai,vr+1 �= 0 for at least one i ∈ {1, . . . , n}\R, then se-lect one of the elements of maximum absolute value among{ai,vr+1 : i ∈ {1, . . . , n}\R} as the pivot element in A.vr+1,suppose it is aur+1,vr+1, then the pivot row for this pivot stepis row ur+1. Generate the pivot matrix Pr+1 correspondingto this pivot step and store it in the string. Include ur+1

in R, vr+1 in C, and the position (ur+1, vr+1) in the list ofpivot element positions.

If r + 1 = n, let P be the permutation matrixof order n with entries of “1” in the positions(ut, vt),t= 1 to n; and “0” entries everywhereelse. Then the product form of the inverse of Ais P T Pn . . . P1; where P1, . . . , Pn are the storedpivot matrices in the order in which they are gen-erated. Terminate.

If r + 1 < n, go to the next step.

END

420 Ch. 4 . Numerical Linear Algebra

Example 8: Let

A =

1 0 11 1 0

−1 1 2

We will now compute the product form of the inverse of A. SelectingA.2 as the first pivot column, and row 3 as the pivot row for the firstpivot step (hence (3, 2) is the pivot element position for this step), leadsto the pivot matrix

P1 =

1 0 00 1 −10 0 1

Selecting column 1 for the next pivot step, the pivot column is

A.1 = P1A.1 =

1 0 00 1 −10 0 1

11

−1

=

12

−1

Row 2 is the pivot row for this pivot step by the partial pivotingstrategy (since “2” in row 2 is the maximum absolute value entry inthe pivot column among rows 1 and 2 in which pivot steps have notbeen carried out so far). So, the 2nd pivot element position is (2, 1);and the pivot matrix corresponding to this pivot step is

P2 =

1 −1/2 00 1/2 00 1/2 1

Column 3 is the column for the final pivot step, the pivot columnis

A.3 = P2P1A.3 =

1 −1/2 00 1/2 00 1/2 1

1 0 00 1 −10 0 1

102

=

1 −1/2 00 1/2 00 1/2 1

1−2

2

=

2−1

1

4.7 Pro duct Form o f t he Inverse 421

Row 1 is the pivot row, and hence (1, 3) is the pivot position forthis pivot step. The corresponding pivot matrix is

P3 =

1/2 1/2 −1/20 1 00 0 1

Finally the permutation matrix defined by the pivot positions is

P =

0 0 11 0 00 1 0

So, the product form of A− 1 is P T P3P 2P 1 with these matrices

given above.

Example 9: Let

A =

1 1 −50 1 −21 2 −7

We will now compute the product form of the inverse of A. SelectingA.2 as the first pivot column, and row 3 as the pivot row for the firstpivot step (hence (3, 2) is the pivot element position for this step), leadsto the pivot matrix

P1 =

1 0 −1/20 1 −1/20 0 1/2

Selecting column 1 for the next pivot step, the pivot column is

A.1 = P1A.1 =

1 0 −1/20 1 −1/20 0 1/2

101

=

1/2−1/2

1/2

Selecting Row 2 as the pivot row for this pivot step by the partialpivoting strategy (so, the 2nd pivot element position is (2, 1)) leads tothe pivot matrix corresponding to this pivot step

422 Ch. 4 . Numerical Linear Algebra

P2 =

1 1 00 −2 00 1 1

Column 3 is the column for the final pivot step. The pivot columnis

A. 3 = P 2P 1A. 3 =

1 1 00 −2 00 1 1

1 0 −1/20 1 −1/20 0 1/2

−5−2−7

=

1 1 00 −2 00 1 1

−3/23/2

−7/2

=

0−3−2

Row 1, the only remaining row in which a pivot step has not beencarried out so far, is the pivot row, but the entry in A. 3 in row 1 is0. This implies that there is a linear dependence relation among thecolumns of A, and therefore A− 1 does not exist. From A. 3 we see thatthe linear dependence relation is

A. 3 − (−3A. 1 − 2A. 2) = 0.

Exercises

4.7.1: Find the inverses of the following nonsingular matrices inproduct form.

0 2 −41 0 2

−2 −1 0

,

0 2 2 22 0 2 22 2 0 22 2 2 0

.

4.7.2: The following matrix is singular. Apply the algorithm tofind the inverse in product form on this matrix and show how thesingularity of this matrix is detected in that algorithm.

4.8 Updating Inverse When a Column Changes 423

0 2 −42 0 24 2 0

.

4.8 How to Update the Inverse When a

Column of the Matrix Changes

In some algorithms (for example, the Simplex algorithm for linear pro-gramming) we need to compute the inverses of a sequence of squarematrices. Consecutive elements of this sequence differ by just one col-umn. To make these algorithms efficient, we need an efficient scheme toupdate the inverse of a matrix when exactly one of its column vectorschanges. We discuss such a scheme here.

Let B be a nonsingular square matrix of order n with its columnvectors B.1, . . . , B.n. Let d = (d1, . . . , dn)T �= 0 be a given columnvector, and suppose B is the matrix obtained by replacing the rthcolumn vector B.r of B by d. So, the

original matrix B = (B.1... . . .

...B.r−1B.rB.r+1... . . .

...B.n)

modifed matrix B = (B.1... . . .

...B.r−1dB.r+1... . . .

...B.n).

Here the symbol r is the number of the column of the original matrixB that is being replaced by the new column vector d. Given B−1, wewant an efficient scheme to obtain (B)−1 from B−1. We show that(B)−1 can be obtained from B−1 with just one GJ pivot step.

Let us review how B−1 is computed in the first place. We start

with the initial tableau (I...B), and convert it by GJ pivot steps and

a row permutation at the end to the final tableau which is (B−1...I).Now suppose we also put the column vector d = (d1, . . . , dn)T in the

initial tableau, i.e., take the initial tableau to be (I...B

...d), and performexactly the same pivot operations and row permutation needed to get

424 Ch. 4. Numerical Linear Algebra

B−1 starting with this as the initial tableau. What will the columnvector d get transformed into in the final tableau in this process? Let

B−1 = (βij) =

β11 . . . β1n...

...βn1 . . . βnn

and suppose the final tableau is (B−1...I...d), where d = (d1, . . . , dn)T .

Since d �= 0, by Result 1.11.1(b) of Section 1.11 we know that d �= 0.Also, from the memory matrix interpretation of the matrix B−1 undercolumns 1 to n in the final tableau, we know that for i = 1 to n

di = βi1d1 + . . . + βindn

i.e., d = B−1d or d = Bd; i.e.,

d = d1B.1 + . . . + dnB.n.

So, d is the representation of d in terms of the columns of the matrixB.

If dr = 0, then by rearranging the terms in the above equation, wehave

d1B.1 + . . . + dr−1B.r−1 − d + dr+1B.r+1 + . . . + dnB.n = 0

a linear dependence relation for the set of column vectors of the matrixB. So, (B)−1 does not exist if dr = 0.

So, we assume that dr �= 0. In the final tableau, the rth column ofB, B.r got transformed into I.r. Suppose we delete I.r from the finaltableau. Then what we have left on the right hand part of the finaltableau are the updated columns of the matrix B, and all we need todo to make it the unit matrix again is to convert the column d into I.r,i.e., perform a GJ pivot step on the final tableau with d as the pivotcolumn and row r as the pivot row, this should convert B−1 into B−1.Since B is obtained by replacing B.r by d, we say that in this operation,d is the entering column replacing the dropping column B.r in B.We now summarize the procedure.

4.8 Updating Inverse When a Column Changes 425

Algorithm to Compute Inverse of B Obtained ByReplacing B.r in B By d, From B−1

BEGIN

Step 1: Compute d = B−1d. d is called the updated enteringcolumn, and the operation of computing it is called updating theentering column.

If d = (d1, . . . , dn)T where dr = 0, then B is singular and its inversedoes not exist, terminate.

Suppose dr �= 0. If B−1 is given explicitly, to get (B)−1 explicitly,go to Step 2. If B−1 is given in product form (discussed in Section 4.7)as PPnPn−1 . . . P1, where P is a permutation matrix, and P1, . . . , Pn

are pivot matrices; to get (B)−1 in product form go to Step 3.

Step 2: Here dr �= 0, B−1 is given explicitly, and we want tocompute (B)−1 explicitly. Put the updated entering column vector dby the side of B−1 as in the following tableau (PC = pivot column, PR= pivot row)

PCd1...

B−1 dr PR...

dn

and perform the GJ pivot step on this tableau. This pivot step convertsB−1 into (B)−1.

Step 3: Here B−1 is given in product form as PPn . . . P1 and wewant to compute (B)−1 in product form.

Let Pn+1 denote the pivot matrix corresponding to the pivot stepin Step 2. It is the unit matrix of order n with its rth column replacedby—

426 Ch. 4 . Numerical Linear Algebra

(d1

dr

, . . . ,− dr − 1

dr

,1

dr

,− dr +1

dr

, . . . ,− dn

dr

)T .

Then the product form for the inverse of B is P n +1PPnP n − 1 . . . P1.

END

Example 1: Let

B =

0 −2 −11 0 −11 1 0

, B−1 =

1 −1 2−1 1 −1

1 −2 2

, d =

121

Let B be the matrix obtained by replacing the third column of Bby d; and B the matrix obtained by replacing the 2nd column of B byd. So,

B =

0 1 −11 2 −11 1 0

, B =

0 −2 11 0 21 1 1

.

From B−1 given above we want to find the inverses of B, B by theupdating procedure given above. The updated entering column is

d = B−1d =

1 −1 2−1 1 −1

1 −2 2

121

=

10

−1

.

Since the 2nd entry in the updated column d is 0, it shows that Bis singular and does not have an inverse.

To find the inverse of B, we have to carry out the pivot step in thefollowing tableau (PC = pivot column, PR = pivot row, and the pivotelement is boxed).

4 .9 R esults o n Systems o f L inea r Equa t io ns 427

PC1 −1 2 1

−1 1 −1 0

1 −2 2 −1 PR2 −3 4 0

−1 1 −1 0−1 2 −2 1

So, we have

(B)− 1 =

2 −3 4−1 1 −1−1 2 −2

Exercises

4.8.1: For r = 1, 2, 3, and 4 in (ii), let Br refer to the matrixobtained by replacing the rth column vector of the following matrixB by d. Find the inverses of B1, B2, B3 and B4 in (ii) respectively byupdating B−1 given below.

(i)B =

0 −2 −11 0 −11 1 0

, B−1 =

1 −1 2−1 1 −1

1 −2 2

, d =

41

−1

(ii)B =

1 −1 0 10 1 0 10 2 1 10 0 0 1

, B−1 =

1 1 0 −20 1 0 −10 −2 1 10 0 0 1

, d =

0−1−1−1

4.9 Results on the General System of Lin-

ear Equations Ax = b

Here we summarize various results on systems of linear equations, witha brief explanation of each result. We consider the general system oflinear equations

428 Ch. 4. Numerical Linear Algebra

Ax = b

where the coefficient matrix A = (aij) is of order m × n, and the RHSconstants vector b = (bi) is a column vector in Rm. A1., . . . , Am. arethe row vectors of A; and A.1, . . . , A.n are its column vectors. The“system” in the following refers to this system of equations. The m ×(n + 1) matrix (A

...b) obtained by including the RHS constants vectoras an additional column in the coefficient matrix A, is known as theaugmented matrix of the system. The augmented matrix containsall the data in the system.

1. Constraint Representation: Each row vector of the coefficientmatrix A corresponds to a constraint in the system. These con-straints are:

Ai.x = bi for i = 1 to m

2. Interpretation of a Solution of the System: Another way of look-ing at the system through the columns of the coefficient matrixA yields the following:

x1A.1 + x2A.2 + . . . + xnA.n = b

i.e., each solution x = (xj) of the system is the coefficient vectorin an expression of b as a linear combination of the columns of A.

Hence the system has a solution iff b can be expressed as a linearcombination of the columns of A.

3. Conditions for the Existence of a Solution: System has a so-lution iff rank(A) = rank of the m × (n + 1) augmented matrix

(A...b).

Rank (A) = rank(A...b) iff a basis for the set of column vectors

of A is also a basis for the set of column vectors of (A...b), i.e.,

4.9 Results on Systems of Linear Equations 429

iff b can be expressed as a linear combination of the columns ofA, by Result 2 above, this is the condition for the existence of asolution to the system.

If rank(A) �= rank(A...b), then b cannot be expressed as a linear

combination of the columns of A, and the system has no solution.In this case, including b as an additional column vector in A

increases its rank, i.e., rank(A...b) = 1 + rank(A).

4. Condition for No Solution: System has no solution iff rank(A...b)

= 1 + rank(A).

Follows from Result 3 above.

5. Condition for Many Solutions: If the system has at least onesolution, and the set of column vectors of the coefficient matrix Ais linearly dependent, then the system has an an infinite numberof solutions. Conversely, if the system has more than one solution,the set of column vectors of A is linearly dependent, and in thiscase the system has an infinite number of solutions.

To see this, suppose x = (xj) is a solution for the system. Then,from Result 2 above we have

b = x1A.1 + . . . + xjA.j + . . . + xnA.n

If {A.1, . . . , A.n} is linearly dependent, let the linear dependencerelation for this set be

0 = α1A.1 + . . . + αjA.j + . . . + αnA.n

where (α1, . . . , αn) �= 0. Adding λ times the 2nd equation to thefirst (here λ is a scalar, an arbitrary real number) yields

b = (x1 + λα1)A.1 + . . . + (xjA.j + λαj)A.j + . . . + (xn + λαn)A.n

So, if we define x(λ) = (x1(λ), . . . , xn(λ))T where for j = 1 to n

430 Ch. 4. Numerical Linear Algebra

xj(λ) = xj + λαj

then x(λ) is a solution of the system for all λ real. Since (α1, . . . ,αn) �= 0, for each λ, x(λ) is a distinct vector; hence the systemhas an infinite number of solutions. Also, as λ takes all realvalues, x(λ) traces a straight line in Rn. This entire straight lineis contained in the set of solutions of the system.

Conversely, suppose the system has two distinct solutions x, x =(xj). Then we have

b = x1A.1 + . . . + xjA.j + . . . + xnA.n

b = x1A.1 + . . . + xjA.j + . . . + xnA.n

Subtracting the bottom one from the top one, we have

0 = (x1 − x1)A.1 + . . . + (xj − xj)A.j + . . . + (xn − xn)A.n

Since x �= x, at least one of the coefficients in the above equationis nonzero, hence this equation is a linear dependence relationfor {A.1, . . . , A.n}, showing that the set of column vectors of A islinearly dependent.

6. Condition For Unique Solution: If the system has a solution x,then x is the unique solution of the system iff the set of columnvectors of A is linearly independent.

This follows from Result 5 above.

7. At Least One Solution If A Is of Full Row Rank: The matrixA is said to be of full row rank if its set of row vectors is linearlyindependent, i.e., if its rank = the number of rows in it, m. Inthis case the system has at least one solution.

In this case when GJ method is applied to solve the system, abasic variable will be selected in every row. So, this method willterminate with a solution of the system after performing a pivotstep in each row.

4 . 9 R esults o n Systems o f L inea r Equa t io ns 431

8. In a Consistent System, Number of Constraints Can Be Reducedto ≤ Number Of Variables: If the system has at least one so-lution, either m ≤ n, or after eliminating redundant constraintsfrom it, an equivalent system in which the number of constraintsis ≤ n will be obtained.

If rank(A) = r, then r ≤ min{m, n}. The result follows becausewhen the system has at least one solution, the GJ method, aftereliminating redundant constraints, will reduce it to an equivalentsystem in which the number of constraints is r ≤ n. Also, if thesystem has a unique solution, it becomes a square nonsingularsystem of equations after the elimination of redundant constraintsfrom it.

Therefore, without any loss of generality, we can assume that ina consistent system of linear equations, the number of constraintsis ≤ the number of variables.

Exercises

4.9.1: Newton’s problem of the fields and cows (from hisArithmetica universalis, 1707): There are several fields of growinggrass, and cows. Assume that initially all the fields provide the sameamount of grass (say x1 per field), and that the daily growth of grassin all the fields remains constant (say x2 in each field), and that all ofthe cows eat exactly the same amount of grass each day (say exactlyx3 per cow per day). The following information is given:

a cows graze b fields bare in c days

a′ cows graze b′ fields bare in c′ days

a′′ cows graze b′′ fields bare in c′′ days.

Then what relationship exists between the 9 data elements a, a′, a′′;b, b′, b′′; c, c′, c′′?

432 Ch. 4 . Numerical Linear Algebra

4.10 Optimization, Another Important Dis-

tinction Between LA, LP, and IP;

and the Duality Theorem for Lin-

ear Optimization Subject to Linear

Equations

We already discussed in Chapter 1 that Linear Algebra (LA) is thesubject dealing with the analysis of systems of linear equations, LinearProgramming (LP) is the subject dealing with the analysis of systemsof linear constraints including inequalities, and that Integer Program-ming (IP) is the subject dealing with the analysis of systems of linearconstraints with some variables restricted to take integer values only. Inthis section we will present some differences in the mathematical prop-erties of these various systems when dealing with optimization. We willalso present the important duality theorem for linear optimization sub-ject to linear equality constraints, and its relationship to the alternatesystem for a system of linear equations discussed in Section 1.17.

Linear Inequalities Viewed as Nonlinear Equa-tions

Each linear inequality actually can be viewed as a nonlinear equa-tion. For example, consider the linear inequality

x1 + x2 + x3 ≤ 100.

This inequality is the same as 100 − x1 − x2 − x3 ≥ 0. Everynonnegative number has a nonnegative square root. So, let s1 denotethe nonnegative square root

√100 − x1 − x2 − x3. Then 100 − x1 −

x2 − x3 = s21, or

x1 + x2 + x3 + s21 = 100.

This equation is a nonlinear equation because of the square terms21 term in it, and it is clearly equivalent to the above inequality. In

4.10 Duality Theorem 433

the same way the inequality constraint x1 + 3x2 − 2x3 ≥ −10 isequivalent to the nonlinear equation x1 + 3x2 − 2x3 − s2

2 = −10.These additional variables s2

1, s22 that were used to convert the in-

equalities into equations are called slack variables associated withthose inequalities in LP literature. So linear inequalities can be con-verted into equations by introducing slack variables, but the resultingequations are nonlinear. As another example, consider the system oflinear constraints

2x1 + 3x2 − 7x3 = 60

x1 + x2 + x3 ≤ 100

x1 + 3x2 − 2x3 ≥ −10

This system is the same as the system of equations

2x1 + 3x2 − 7x3 = 60

x1 + x2 + x3 + s21 = 100

x1 + 3x2 − 2x3 − s22 = −10

which is nonlinear

Linear Optimization Subject to Linear EqualityConstraints

In most real world applications one is always interested in findingnot just any solution, but the best solution among all solutions to asystem of constraints, best in the sense of optimizing the value of anobjective function that is to be optimized (i.e., either minimized ormaximized as desired). Since LA considers systems of linear constraintsonly, it is reasonable to assume that the user may want to optimizean objective function that is itself a linear function of the decisionvariables. This would lead to a problem of the following form whichwe state using matrix notation.

Minimize z = z(x) = cx

434 Ch. 4. Numerical Linear Algebra

subject to Ax = b (1)

instead of finding an arbitrary solution to the system Ax = b. Such aproblem is called a linear optimization problem subject to linearequations. Here let A be an m×n coefficient matrix, b an m×1 RHSconstants vector, and c is a 1 × n row vector of objective functioncoefficients.

Instead of minimizing, if it is desired to maximize an objectivefunction c′x, it is the same as minimizing its negative −c′x subject tothe same constraints. Hence it is sufficient to consider minimizationproblems only.

If system (1) is either infeasible, or has a unique solution, thereis nothing to choose, so we will assume that (1) has many solutions.Even in this case, it turns out that there are only two possibilities forproblem (1). They are:

Possibility 1: Under this possibility all solutions of system (1)give the same value for the objective function cx, i.e., cx is a constanton the set of solutions of (1). If this possibility occurs, it is immaterialwhich solution of (1) is selected, all have the same objective value.

This possibility occurs iff the row vector c can be expressed as alinear combination of the row vectors of the coefficient matrix A, i.e.,iff c is in the row space of the matrix A as defined in Section 3.10.

Possibility 2: The minimum value of cx on the set of solutions of(1) is −∞, i.e., (1) has no finite optimum solution.

This possibility occurs iff the row vector c cannot be expressed asa linear combination of the row vectors of the coefficient matrix A.

Algorithm to Determine Which Possibility Oc-curs for (1)

The GJ or G elimination methods for solving systems of linearequations discussed in Sections 1.16, 1.20 can directly determine whichof Possibilities 1, 2 occurs in (1). Here is the simple modification of theGJ or G elimination methods to solve (1).—

4.10 Duality Theorem 435

BEGIN

Step 1: Detached coefficient tableau: Initiate the method withthe following detached coefficient tableau

x −z RHSA 0 bc 1 0

where the bottom row corresponds to the additional equation cx −z = 0 that just defines the objective function. When the algorithmdetermines that Possibility 1 occurs, if it is required to obtain thecoefficients in an expression of c as a linear combination of rows of A,we need to introduce a memory matrix (a unit matrix of order m + 1with its column vectors labeled as A1., . . . , Am., c) on the left side ofthe tableau as explained in Section 1.12, and carry out all the rowoperations on the columns of this memory matrix also.

Step 2: Carry out the GJ or G elimination methods of Sections 1.16or 1.20 on this tableau and perform pivot steps in all the rows of thetableau except the last row. At the end if the infeasibility conclusionis not reached, go to Step 3.

Step 3: In the final tableau suppose the entries in the bottom roware c1, . . . , cn; bm+1.

If all of c1, . . . , cn are = 0, then c is a linear combination of the rowsof A, and Possibility 1 occurs for (1). In this case the objective functionin (1), cx = −bm+1 at all the solutions of the system Ax = b. Deletethe last row from the final tableau, and obtain the general solution ofthe system from the remaining tableau, every one of these solutions isan optimal solution of (1).

If the memory matrix is introduced, its bottom row in the finaltableau gives the coefficients in an expression of c as a linear combina-tion of rows of A.

If at least one of c1, . . . , cn is �= 0, then the minimum value of cx onthe set of solutions of (1) is −∞, i.e., Possibility 2 occurs for (1). In this—

436 Ch. 4 . Numerical Linear Algebra

case we can construct a half-line every point on which is feasible to (1),and along which the objective function cx → −∞. The constructionof this half-line from the final tableau is illustrated in Example 2 givenbelow.

END

Example 1: Consider the following problem.

Minimize z = 6x1 + 22x2 + 34x3 + 6x4

subject to x1 + x2 + x3 + x4 = 15

x1 − x2 − 2x3 − x4 = −5

−x1 + 2x4 = 7

The original tableau for solving this problem by the GJ eliminationmethod with memory matrix (to find the coefficients in an expressionof c as a linear combination of the rows of the coefficient matrix A,if it turns out to be so) is the one given at the top. The columns ofthe memory matrix are labeled A1., A2., A3., and c since these are thevarious rows in the original tableau. The abbreviations used are: BV= basic variable, and all the pivot elements are boxed. PR, the pivotrow; and PC, the pivot column, are always the row, column containingthe boxed pivot element.

Memory matrix BV RHSA1. A2. A3. c x1 x2 x3 x4 −z

1 0 0 0 1 1 1 1 0 150 1 0 0 1 −1 −2 −1 0 −50 0 1 0 −1 0 0 2 0 70 0 0 1 6 22 34 6 1 01 0 0 0 x2 1 1 1 1 0 15

1 1 0 0 2 0 −1 0 0 100 0 1 0 −1 0 0 2 0 7

−22 0 0 1 −16 0 12 −16 1 −330

4.10 Duality Theorem 437

Memory matrix BV RHSA1. A2. A3. c x1 x2 x3 x4 −z

2 1 0 0 x2 3 1 0 1 0 25−1 −1 0 0 x3 −2 0 1 0 0 −10

0 0 1 0 −1 0 0 2 0 7−10 12 0 1 8 0 0 −16 1 −220

Final Tableau2 1 3 0 x2 0 1 0 7 0 46

−1 −1 −2 0 x3 0 0 1 −4 0 −260 0 −1 0 x1 1 0 0 −2 0 −7

−10 12 8 1 0 0 0 0 1 −164

GJ pivot steps have been carried out in all the constraint rows.Since the coefficients of all the original variables xj are 0 in the last rowof the final tableau, this is an indication that c is a linear combinationof rows of the coefficient matrix A in this example. From the final rowof the memory matrix in the final tableau we find

−10A1. + 12A2. + 8A3. + c = 0

or c = 10A1. − 12A2. − 8A3..

Also the equation corresponding to the last row in the final tableauis −z = −164 or z = 164. Thus in this example, the objectivefunction z has the same value 164 at all solutions of the system ofconstraints. From the final tableau we see that the general solution ofthe system is

x1

x2

x3

x4

x5

=

−7 − 2x3 + 2x5

46 + 4x3 − 7x5

x3

−26 − 3x3 + 4x5

x5

where the nonbasic (free) variables x3, x5 can be given any real values atthe user’s choice to get a solution of the system; and all these solutionsyield the same value 164 to the objective function z.

438 Ch. 4 . Numerical Linear Algebra

Example 2: Consider the following problem.

Minimize z = x1 + x2 − 2x4 − 4x5

subject to x1 + x2 − x3 + x5 = −6

x1 + 2x2 + x4 − 2x5 = 20

−x1 + x3 − x4 + 3x5 = 12

We solve this problem by the G elimination method, but use thesame notation as in the previous example. In this example we do notintend to find the expession of c as a linear combination of the rowvectors of A even if it it turns out to be in the linear hull of the rowsof A; so we do not introduce the memory matrix.

BV x1 x2 x3 x4 x5 −z RHS

1 1 −1 0 1 0 −6 PR1 2 0 1 −2 0 20

−1 0 1 −1 3 0 121 1 0 −2 −4 1 0

PC↑x1 1 1 −1 0 1 0 −6

0 1 1 1 −3 0 26 PR0 1 0 −1 4 0 60 0 1 −2 −5 1 6

PC↑x1 1 1 −1 0 1 0 −6x2 0 1 1 1 −3 0 26

0 0 −1 −2 7 0 −20 PR0 0 1 −2 −5 1 6

PC↑Final Tableau

x1 1 1 −1 0 1 0 −6x2 0 1 1 1 −3 0 26x3 0 0 1 2 −7 0 20

0 0 0 −4 2 1 −14

4.10 Duality Theorem 439

The nonbasic variables x4, x5 have nonzero coefficients in the objec-tive row in the final tableau. This indicates that the objective functionz diverges to −∞ on the set of solutions of the system of constraintsin this example, so there is no finite optimum solution in this problem.

To construct a half-line in the set of feasible solutions along whichthe objective function z diverges to −∞, select any nonbasic variablewith a nonzero coefficient in the objective row in the final tableau, callit the entering variable, xs say with cs �= 0 as its coefficient in theobjective row in the final tableau. Fix all the nonbasic variables otherthan the entering variable at 0 in the solution, and make the value of theentering variable = λ, a real valued parameter. Multiply the column ofthe entering variable in the final tableau by λ and move it to the right.Now compute the corresponding values of the basic variables in thesolution by the back substitution method. The solution is a functionof the single parameter λ. As λ → +∞ if cs < 0, or −∞ if cs > 0,this solution traces a half-line in the solution set along which the theobjective function z → −∞.

In this example we select x4 with c4 = −4 as the entering variable.We fix the other nonbasic variable x5 = 0, and entering variable x4 = λ.The triangular system to find the corresponding values of the basicvariables in the solution is

x1 x2 x3

0 1 −1 −60 1 1 26 − λ0 0 1 20 − 2λ

−z −14 + 4λ

The expression for −z is obtained from the equation correspondingto the last row in the final tableau after fixing the entering variable = λ,and all the other nonbasic variables = 0. Solving by back substitution,we find that the solution is

440 Ch. 4. Numerical Linear Algebra

x(λ) =

x1(λ)x2(λ)x3(λ)x4(λ)x5(λ)

=

8 − 3λ6 + λ

20 − 2λλ0

z = 14 − 4λ

So, as λ → +∞, x(λ) traces a half-line in the set of feasible solutionsof the system along which the objective function z → −∞.

Thus, in linear optimization problems subject to linear equalityconstraints only (of the form (1) above), when feasible solutions exist,either there is no finite optimum solution, or all the feasible solutionsare optimal. That’s why these problems are considered to be trivialmathematically, and none of the other books on LA discuss optimizinga linear function subject to linear equality constraints.

The Duality Theorem for Linear optimization Prob-lems Subject to Linear Equality Constraints

Consider the general linear optimization problem subject to linearequality constraints (1) above. There is another linear optimizationproblem subject to linear equality constraints associated with it calledits dual, sharing the same data A, b, c. In this context the originalproblem (1) is called the primal. It turns out that the dual of thedual problem is the primal; and the special relationship between thispair of problems is called the duality relationship, it is a symmetricrelationship.

In this context, the variables in the primal (1), x = (x1, . . . , xn)T ,are called the primal variables. Since the constraints in the primalproblem are a requirement on Ax =

∑nj=1 A.jxj , the primal variables

x are coefficients in a linear combination of the columns of the datamatrix A. Symmetrically, the variables in the dual problem, called thedual variables, written as a row vector y = (y1, . . . , ym) usually, arecoefficients in a linear combination of the rows of the data matrix A;

4.10 Duality Theorem 441

and the dual constraints will be a requirement on yA. That’s why thisduality relationship is an instance of row-column duality in matrices.

The dual problem of (1) is

Minimize v = v(y) = yb

subject to yA = c (2)

Notice that the primal variables do not appear in the statment ofthe dual problem and vice versa. The primal and the dual problemsare distinct problems, but both of them share the same data.

If Possibility 1 occurs for (1), then the algorithm discussed abovefinds a dual feasible solution y satisfying c = yA. In this case everyprimal feasible solution x has its primal objective value = z(x) = cx =yAx = yb. Also, in this case every dual feasible solution y has its dualobjective value = v(y) = yb = yAx = cx where x is any primal feasiblesolution.

Thus if Possibilty 1 occurs for (1), every primal feasible solutionhas the same primal objective value, every dual feasible solution hasthe same dual objective value, and the optimum objective values in thetwo problems are equal.

If Possibility 2 occurs for (1), it is not possible to express c as a linearcombination of the rows of A, i.e., the dual has no feasible solution;and we have seen that in this case z(x) is unbounded below in (1).

Now suppose that the dual has a feasible solution y say, but theprimal has no feasible solution. In this case by the results discussed inSection 1.17, we know that the alternate system for the constraints in(1)

πA = 0

πb = 1

has a feasible solution, π say. Since yA = c and πA = 0, we have(y +λπ)A = c for any λ real, i.e., y +λπ is feasible to the dual problem(2) for all λ. Also v((y + λπ) = (y + λπ)b = yb + λπb = yb + λ, andthis tends to +∞ as λ → +∞. Thus the dual objective function is

442 Ch. 4. Numerical Linear Algebra

unbounded above in the dual problem in this case. The feasibility ofthe alternate system for the constraints in (1), is a necessary conditionfor the dual objective function to be unbounded above on the set ofdual feasible solutions in (2).

These results can be summarized in the following statement knownas the duality theorem for linear optimization subject to linearequality constraints. It is a specialization of the duality theorem ofLP proved by OR people (John von Neumann, David Gale, HaroldKuhn, and Albert Tucker) soon after George Dantzig anounced hissimplex method for LP in 1947, to (1).

Duality Theorem: A primal, dual pair of linear optimizationproblems subject to linear equality constraints always satisfy the fol-lowing properties:

1. If both primal and dual are feasible, the optimum objective val-ues in the two problems are equal; and every feasible solution isoptimal in both problems.

2. If primal [dual] is feasible but the dual [primal] is infeasible, thenthe primal objective value [dual objective value] is unbounded be-low [above] in the primal [dual] problem.

3. It is possible for both the primal and dual problems to be infeasible.

Optimizing A Linear Objective Function Subjectto Linear Inequality Constraints

This is a problem of the following form stated in matrix notation.

Minimize z = z(x) = cx

subject to Ax = b (3)

Dx ≥ d

The presence of linear inequality constraints in (3) makes a signifi-cant difference in the possible outcomes for (3) as opposed to those of

4.10 Duality Theorem 443

(1) where all the constraints are linear equations only. (3) can have afinite optimum solution, without all the feasible solutions of the prob-lem being optimal. Thus (3) is a nontrivial optimization problem. Thisis an important distinction between linear equality systems and linearsystems including inequalities.

Traditionally LP is defined to be a problem of the form (3) involvingoptimization, not just that of finding any feasible solution.

But there is a beautiful result in LP theory which states that cor-responding to (3), there is an equivalent system of linear constraintsinvolving inequalities such that every feasible solution of it is optimalto (3) and vice versa. This result guarantees that LPs involving op-timization are mathematically equivalent to finding a feasible solutionto a system of linear constraints containing linear inequalities. In otherwords any algorithm for finding a feasible solution to a system of linearinequalities can be used directly to solve optimization problems like (3)without doing any optimization, and vice versa.

Optimizing A Linear Objective Function Subjectto Linear Constraints and Integer Restrictions OnVariables

In matrix notation, this IP is a problem of the following form. Herex = (x1, . . . , xn)T is the vector of decision variables. J ⊂ {1, . . . , n} isthe set of subscripts of decision variables whose values in the solutionare required to be integer.

Minimize z = z(x) = cx

subject to Ax = b (4)

Dx ≥ d

xj integer for all j ∈ J .

In LP all the variables are continuous variables. But in IP some orall the variables are restricted to a discrete set, and hence IPs are alsocalled discrete optimization problems.

Traditionally an IP is stated in this problem involving optimization.As in the case of LP, in IP also, an efficient algorithm that can find

444 Ch. 4. Numerical Linear Algebra

a feasible solution to a system of constraints like that in (4) can bemodified into an efficient algorithm for solving (4) involving optimiza-tion too. Also like in LPs, (4) can have a finite optimum without allthe feasible solutions being optimal to it. So, IPs also are nontrivialoptimization problems.

4.11 Memory Matrix Versions of GJ, G

Elimination Methods

The versions of the GJ, G elimination methods discussed in Sections1.16, 1.20 are the direct simple versions of those methods to solve asystem of linear equations dating back to ancient times. Around mid-20th century Operations Researchers have found that the results inSections 1.12, 4.7 can be used to organize the computation in thesemethods more efficiently by introducing a memory matrix on the orig-inal tableau and restricting the pivot operations to the columns of thememory matrix only, leaving the original system unchanged through-out the computation. The pioneering work for developing this efficientversion is due to George Dantzig in his 1953 paper on the revised sim-plex method for linear programming (G. B. Dantzig, “ComputationalAlgorithm of the Revised Simplex Method”, RM-1266, The RANDCorporation, Santa Monica, CA, 1953).

Suppose the original system consists of m equations in n variables.In this version pivot operations are carried out only on (m + 1) col-umn vectors (m column vectors of the memory matrix, and the RHSconstants vector) instead of the (n + 1) column vectors of the originalsystem. Since typically n ≥ m, the new version is computationallymore efficient than the versions discussed in Sections 1.16, 1.20.

We will now provide a complete statement of the new version alongwith detailed descriptions of all the formulas used in it, and then workout several numerical examples.

4.11 Memory Matrix Versions 445

Memory Matrix Version of GJ, G EliminationMethods

Let the original system being solved be Ax = b where A = (aij)is an m × n matrix.

BEGIN

Step 1: Setting up the initial tableau: It is convenient tointroduce the columns of the memory matrix (initially, a unit matrixof order m) on the left hand side of the detached coefficient tableauof the original system. Let us denote the constraints in the originalsystem by R1, . . . , Rm and enter these labels in a column on the righthand side of the original system. We label the columns of the memorymatrix with R1 to Rm, because as mentioned in Section 1.12, any rowvector in the memory matrix is the row vector of coefficients in anexpression of the corresponding row in the updated system as a linearcombination of rows in the original system. We also set up anothercolumn on the right to record the basic variables. We will use theabbreviations: BV = basic variable selected in the row, PC = pivotcolumn, PR = pivot row, OCL = original constraint label.

Initial tableau: Tableau 1BV OCL Original system Memory matrix

x1 . . . xj . . . xn RHS R1 . . . Rm

R1 a11 . . . a1j . . . a1n b1 1 . . . 0...

......

......

......

Ri ai1 . . . aij . . . ain bi 0. . . 0

......

......

......

...Rm am1 . . . amj . . . amn bm 0 . . . 1

OCL = Original constraint label for the row

Some notes: In this version, the columns of the variables x1, . . . , xn

in the original system are left as they are throughout the computation;pivot operations are carried out only on the other columns of the ini-

446 Ch. 4. Numerical Linear Algebra

tial tableau. So these other columns constitute the working area in thisversion, and we will record these only separately in subsequent steps,calling it the working tableau. The working tableau in the initialstage is Tableau 2.

Initial working tableau: Tableau 2BV Updated Memory matrix PC OCL

RHS R1 . . . Rm

b1 1 . . . 0 R1...

......

...

bi 0. . . 0 Ri

......

......

bm 0 . . . 1 Rm

In the GJ elimination method, rows 1 to m can be selected as pivotrows in any order, and the method does not need row interchanges. Inthe G elimination method however, we perform a row interchange inevery step to make sure that all rows which have already served as pivotrows are at the top of the tableau, and those in which pivot steps haveyet to be carried out are below them at the bottom. In this version,like pivot steps, row interchanges are only carried out on the workingtableau, and never on the original tableau. If some row interchangeshave been carried out so far, in the current working tableau rows 1to m correspond to the original constraints Rw1, . . . , Rwm respectively,where (w1, . . . , wm) is a permutation of (1, . . . , m).

Current working tableau: Tableau 3BV Updated Updated mem. matrix PC OCL

RHS R1 . . . Rm

b1 β11 . . . β1m Rw1

......

......

bi βi1. . . βim Rwi

......

......

bm βm1 . . . βmm Rwm

4.11 Memory Matrix Versions 447

At some general stage suppose the current working tableau is asgiven in Tableau 3. The matrix β = (βij : i, j = 1, . . . , m) in thecurrent working tableau is the updated memory matrix at this stage.In this version we do not have the current tableau consisting of theupdated columns of all the variables, which would have been obtainedif all the pivot steps have been carried out on the original tableau asin the versions discussed in Sections 1.16, 1.20. Let us use the follow-ing notation for the updated columns of the variables in the currenttableau:

OCL Current tableau*x1 x2 xj xn

R1 a11 a12 . . . a1j . . . a1n

R2 a21 a22 . . . a2j . . . a2n...

......

......

Ri ai1 ai2 . . . aij . . . ain...

......

......

Rm am1 am2 . . . amj . . . amn

*Not computed in this version

But all the entries in this current tableau can be computed usingthe updated memory matrix β and the original tableau. We show theformulas for computing them now.

Let A = (aij : i = 1 to m, j = 1 to n) denote the current tableauwhich is not computed in this version. Ai., A.j denote the ith row,jth column of A. A.j is the updated column of the variable xj in thecurrent tableau.

From Section 1.12 we know that the ith row in the current tableau isthe linear combination of rows in the original tableau with coefficientsfrom the row of the updated memory matrix corresponding to the OCLRi (this may not be the ith row in the updated memory matrix if rowinterchanges have been performed during the algorithm). So, let

p be such that wp = i in the current working tableau. Then

ith row in current tableau = Ai. = (ai1, . . . , ain) =

448 Ch. 4. Numerical Linear Algebra

= βwp.A = (βwp1, . . . , βwpn)

a11 . . . a1n...

...am1 . . . amn

Using this formula any row vector in the current tableau can becomputed from β and A as needed.

From the results in Section 4.7, we know that the updated columnof any variable xj in the current tableau with rows rearrranged as in thecurrent working tableau = (aw1j , . . . , awmj) is obtained by multiplyingthe original column of xj on the left by β, i.e., it is = βA.j.

Using these formulas, any row vector in the current tableau, or theupdated column of any variable can be computed using β, A. We willuse these formulae in the following steps.

Step 2: The general pivot step: Select one of the rows in whicha pivot step has not been performed, as the row for performing the nextpivot step. Suppose it is row wt in original tableau corresponding to rowr in the current working tableau. This updated row is (awt1, . . . , awtn)= βr.A.

Case 1: If awt1 = . . . = awtn = 0, and the updated RHS constantin this row = bwt = 0 also, this row represents a redundant constraint.The row βr., the rth row of the updated memory matrix is the evidencefor this redundancy conclusion.

So, no pivot step will be carried out in this row in this case. Selectone of the remaining rows for performing the next pivot step and repeatthis Step 2 with it.

Case 2: If awt1 = . . . = awtn = 0, and the updated RHS constantin this row = bwt �= 0, this row represents an inconsistent equation, sothe original system has no solution. The row βr., the rth row of theupdated memory matrix is the evidence for this infeasibility conclusion(i.e., br times a feasible solution for the alternate system correspondingto the original system discussed in Section 1.17).—

4.11 Memory Matrix Versions 449

If the infeasibility conclusion is all that is needed, terminate themethod. At any rate no pivot step will be carried out in this row. If itis required to find a modification of the RHS constants vector b in theoriginal system to make it feasible as discussed in Section 1.22, selectone of the remaining rows to perform the next pivot step and repeatthis Step 2 with it.

Case 3: Suppose at least one of awt1, . . . , awtn is �= 0. In this casewe will perform a pivot step.

If GJ elimination method is being used, select this row, row wt inthe original tableau corresponding to row r in current working tableauas the pivot row for this pivot step. Select an s such that awts �= 0 asthe pivot element, and the associated variable xs as the basic variablein this pivot row, i.e., the entering variable in this pivot step.

When doing hand computation, we want to avoid getting difficultfractions, so we look for a pivot element of ±1 or the smallest availableinteger as far as possible. But while solving the problem on a com-puter, we try to select the pivot element to minimize roundoff erroraccumulation, for this it is better to select the pivot element to be anelement of maximum absolute value, i.e., satisfying |awts| = maximum{|awtj | : j = 1 to n}. The pivot column (PC) for this pivot step is theupdated column of the entering variable with rows rearranged as in thecurrent working tableau = βA.s. Enter the PC on the current workingtableau, mark the pivot element in it, and perform the GJ pivot step.

If G elimination method is being used to solve the problem on acomputer, we select s such that awts is the maximum absolute valueelement among awt1, . . . , awtn, and select the associated variable xs asthe entering variable for this pivot step. Then the pivot column (PC)for this pivot step is the updated column of the entering variable xs

with rows rearranged as in the current working tableau = βA.s. Enterthe PC on the current working tableau. Select the pivot element to bean element of maximum absolute value in the PC among all elementsin it in rows r to m (these are the rows in which pivot steps have notbeen carried out so far). In the working tableau interchange the rowcontaining the pivot element with row r, then make it the pivot rowfor the pivot step. Then perform the G pivot step.—

450 Ch. 4. Numerical Linear Algebra

If there are no more rows in which pivot steps have to be carried out,with the new updated working tableau go to Step 3 if the infeasibilityconclusion has not been reached so far, or to Step 4 if the infeasibilityconclusion has occured during the algorithm and it is desired to carryout infeasibility analysis.

If there are some more rows in which pivot steps have to be carriedout, select one of those to perform the next pivot step in the newworking tableau, and repeat this Step 2 with it.

Step 3: Reading the Solution: In the final working tableauevery row which has no basic variable recorded in it corresponds to aredundant constraint in the original system, and can be deleted withoutchanging the set of feasible solutions of the original system.

By making all the nonbasic variables 0, and equating each basicvariable in the final working tableau to the updated RHS constant inits row, we get a basic solution of the original system if the GJ methodis used (if the G method is used instead, a back substitution step isneeded to get the values of the basic variables).

The parametric representation of the general solution of the systemcan be obtained by computing the updated column of each of the non-basic variables using the formulas discussed above, and then using theformula for the general solution discussed in Sections 1.16, 1.20.

Step 4: Infeasibility analysis: Here the original system is in-feasible. It can be made feasible by modifying some of the data in it.One possible modification is to change the RHS constants vector b asdiscussed in Section 1.22.

For this, if any row interchanges have been carried out during thealgorithm, rearrange the rows in the final working tableau accordingto their order in the original tableau using the information in the OCLcolumn. For each i such that row i corresponds to an infeasible con-straint (i.e., updated constraint of the form 0 = bi �= 0) change theoriginal RHS constant bi to b′i = bi − bi where bi is the updated RHSconstant in this row; this modification has the effect of converting bi inthe working tableau to 0. With this change each infeasible constraintbecomes a redundant constraint, and the system becomes feasible. A—

4.11 Memo ry Ma trix Versions 451

basic solution, and the general solution of the modified system can nowbe obtained from the final working tableau as in Step 3.

END

We will now provide some numerical examples. In all these exam-ples we will use the following notation:

A, b = Coefficient matrix and RHS constants vector in theoriginal system

Ai., A.j = The ith row, jth column of A

β, βi. = The current working tableau, its ith row

A, Ai., A.j = The updated coefficient matrix, and its ithrow, jth column

b = (bi) = Updated RHS constants vector

OCL = Original constraint label

PR, PC; BV = Pivot row, pivot column for the currentpivot step, selected basic variable in that row.

Example 1: This problem is from Example 1 in Section 1.16solved there by the version of the GJ elimination method discussedthere. The original system with the OCL is:

x1 x2 x3 x4 x5 RHS OCL−1 0 1 1 2 10 R1

1 −1 2 −1 1 5 R2

1 −2 5 −1 4 20 R3

2 0 0 −2 1 25 R4

3 −2 5 −3 5 45 R5

5 −2 5 −5 6 70 R6

We will apply the memory matrix version of the GJ eliminationmethod to find a feasible solution of this system. The initial workingtableau is:

452 Ch. 4. Numerical Linear Algebra

BV Memory matrix β PC OCLb R1 R2 R3 R4 R5 R6 x3

10 1 0 0 0 0 0 1 R1 PR5 0 1 0 0 0 0 2 R2

20 0 0 1 0 0 0 5 R3

25 0 0 0 1 0 0 0 R4

45 0 0 0 0 1 0 5 R5

70 0 0 0 0 0 1 5 R6

We select x3 as the first entering variable and perform a pivot stepwith row 1 as the PR, leading to the next working tableau.

BV Memory matrix β PC OCLb R1 R2 R3 R4 R5 R6 x2

x3 10 1 0 0 0 0 0 0 R1

−15 −2 1 0 0 0 0 −1 R2 PR−30 −5 0 1 0 0 0 −2 R3

25 0 0 0 1 0 0 0 R4

−5 −5 0 0 0 1 0 −2 R5

20 −5 0 0 0 0 1 −2 R6

Selecting row 2 to perform the next pivot step we find A2. = β2.A= (−2, 1, 0, 0, 0, 0)A = (3,−1, 0,−3,−3). We select the 2nd entry init, −1, as the pivot element, x2 as the entering variable. Its updatedcolumn is βA.2 = β(0,−1,−2, 0,−2,−2)T = A.2 which is the PC en-tered on the working tableau. Performing this pivot step leads to thenext working tableau.

BV Memory matrix β PC OCLb R1 R2 R3 R4 R5 R6 x5

x3 10 1 0 0 0 0 0 2 R1

x2 15 2 −1 0 0 0 0 3 R2

0 −1 −2 1 0 0 0 0 R3

25 0 0 0 1 0 0 1 R4 PR25 −1 −2 0 0 1 0 1 R5

50 −1 −2 0 0 0 1 2 R6

4.11 Memory Matrix Versions 453

Selecting row 3 to perform the next piovot step, we find that A3.

= β3.A = (0, 0, 0, 0, 0), a row vector of zeros. Also, b3 = 0. Hence the3rd constraint in the original system can be treated as a redundantconstraint, so we cannot perform a pivot step in it. Moving to row 4,we find that A4. = β4.A = (2, 0, 0,−2, 1) and we select the element 1 init as the pivot element and x5 as the entering variable for the next pivotstep. The PC, the updated column of x5 is βA.5 = β(2, 1, 4, 1, 5, 6)T =(2, 3, 0, 1, 1, 2)T which is entered on the working tableau. Performingthis pivot step leads to the next working tableau.

Final working tableauBV Memory matrix β PC OCL

b R1 R2 R3 R4 R5 R6 x2

x3 −40 1 0 0 −2 0 0 R1

x2 −60 2 −1 0 −3 0 0 R2

0 −1 −2 1 0 0 0 R3

x5 25 0 0 0 1 0 0 R4

0 −1 −2 0 −1 1 0 R5

0 −1 −2 0 −2 0 1 R6

Only rows 5, 6 remain for performing pivot steps, but we now findthat both sides of their updated rows are 0, so both these rows representredundant constraints in the original tableau, so no pivot steps needto be performed in them. Hence the above working tableau is the finalworking tableau. From it we see that a basic solution for the originalsystem is given by

Basic variables

x3 = −40x2 = −60x5 = 25

nonbasic variables

{x1 = 0x4 = 0

which is x = (0,−60,−40, 0, 25)T .To get the general solution of the system we find the updated col-

umn of the nonbasic variables x1, x4 which are A.1 = βA.1 = (−5,−9, 0, 2, 0, 0)T ,

454 Ch. 4 . Numerical Linear Algebra

and A. 4 = βA. 4 = (5, 9, 0,−2, 0, 0)T . Hence as explained in Section 1.16,the general solution of the system is given by

x3 = −4 + 5x1 − 5x4

x2 = −60 + 9x1 − 9x4

x5 = 25 − 2x1 + 2x4

x1 = x1, at user’s choice

x4 = x4, at user’s choice

or rearranging the variables in natural order, it is

x =

x1

−60 + 9x1 − 9x4

−40 + 5x1 − 5x4

x4

25 − 2x1 + 2x4

The user can select any real values for the free variables x1, x4 andplug them in the above to get a solution of the original system. Givingthem values 0 leads to the basic solution mentioned above.

From the final working tableau we also find

Evidence of redundancy for R3 is (−1,−2, 1, 0, 0, 0)

Evidence of redundancy for R5 is (−1,−2, 0,−1, 1, 0)

Evidence of redundancy for R6 is (−1,−2, 0,−2, 0, 1)

For the original constraint R3 this means that the linear combina-tion −R1 −R2+R3 is the 0=0 equation, or equivalently, that constraintR3 is the linear combination R1 + R2.

Similarly R5 is R1 +2R2 +R4 and R6 is R1 +2R2 +2R4 all of whichcan be verified easily from the original tableau.

Example 2: This problem is from Example 3 in Section 1.16.The original system with the OCLs is

4.11 Memory Matrix Versions 455

x1 x2 x3 x4 x5 RHS OCL2 1 −1 −2 0 4 R1

4 1 5 6 4 9 R2

6 2 4 4 4 12 R3

10 3 9 10 8 22 R4

The initial working tableau is

BV Memory matrix β PC OCLb R1 R2 R3 R4 x2

4 1 0 0 0 1 R1 PR9 0 1 0 0 1 R2

12 0 0 1 0 2 R3

22 0 0 0 1 3 R4

We select row 1 to perform the 1st pivot step, and select x2 as theentering variable in it. The PC, original column of the entering variablex2 is entered on the working tableau. Performing the pivot step leadsto the next working tableau.

BV Memory matrix β PC OCLb R1 R2 R3 R4 x1

x2 4 1 0 0 0 2 R1

5 −1 1 0 0 2 R2 PR4 −2 0 1 0 2 R3

10 −3 0 0 1 4 R4

Selecting row 2 for the next pivot step, we find A2. = β2.A =(2, 0, 6, 8, 4) and we choose 2 in it as the pivot element. The enter-ing variable is x1, its updated column, βA.1 = (2, 2, 2, 4)T , is the PCentered on the working tableau. Performing the pivot step leads to thenext working tableau.

456 Ch. 4. Numerical Linear Algebra

Final working tableauBV Memory matrix β PC OCL

b R1 R2 R3 R4

x2 −1 2 −1 0 0 R1

x1 5/2 −1/2 1/2 0 0 R2

−1 −1 −1 1 0 R3

0 −1 −2 0 1 R4

We verify that the updated row 3 is A3. = β3.A = 0, and theupdated RHS constant in it is −1 �= 0. So row 3 represents an infeasibleconstraint, and the original system has no feasible solution.

Evidence of infeasibility for R3 is (−1,−2, 0, 1)

This evidence shows that the linear combination of original con-straints −R1 − R2 + R3 is the infeasible equation 0 = −1.

This also implies that the linear combination (−R1−R2+R3)/(−1)= R1 + R2 − R3 is the fundamental infeasible equation 0 = 1. Hence(1, 1,−1, 0) is a solution of the alternate system corresponding to theoriginal system, discussed in Section 1.17.

We also verify that the updated row 4 is β4.A = 0, and since b4 =0 row 4 represents a redundant constraint. The evidence for this is(−1,−2, 0, 1) which implies that −R1 − 2R2 + R4 is the 0=0 equation,i.e., R4 is the linear combination R1 + 2R2. So no pivot steps will beperformed in rows 3, 4; hence the current working tableau is the finalone.

We will now perform infeasibility analysis (Section 1.22) on thisinfeasible system. From the final working tableau, we see that theoriginal system can be made feasible by subtracting b3 = −1 from theoriginal b3 = 12, i.e., changing b3 to b′3 = 13. This changes the originalRHS constants vector b from (4, 9, 12, 22)T to (4, 9, 13, 22)T .

If the same pivot steps as above are carried out on the modifiedoriginal tableau the final working tableau will be

4.11 Memo ry Ma trix Versions 457

Final working tableauBV Memory matrix β PC

b R1 R2 R3 R4

x2 −1 2 −1 0 0 R1

x1 5/2 −1/2 1/2 0 0 R2

0 −1 −1 1 0 R3

0 −1 −2 0 1 R4

The only change here is that b3 has now become 0, so row 3 repre-sents a redundant constraint in the modified problem, hence the mod-ified problem is feasible. A basic solution for it, from the final workingtableau, is given by x2 = −1, x1 = 5/2, x3 = x4 = 0; i.e., it isx = (5/2,−1, 0, 0)T .

Example 3: Here we will solve the problem in Example 1 fromSection 1.20 with the memory matrix version of the G eliminationmethod using the partial pivoting strategy for pivot element selection.

The original tableauBV x1 x2 x3 x4 x5 x6 RHS OCL

1 −2 1 1 1 −1 −4 R1

2 2 0 1 −1 −1 10 R2

2 −4 −2 0 −2 −2 −8 R3

0 −6 −2 −1 −1 −1 −18 R4

0 0 0 4 1 1 20 R5

1 4 −1 0 −2 0 14 R6

The Initial working tableauBV Memory matrix β PC OCL

b R1 R2 R3 R4 R5 R6 x1

−4 1 0 0 0 0 0 1 R1

10 0 1 0 0 0 0 2 R2

−8 0 0 1 0 0 0 2 R3 RI to row 1−18 0 0 0 1 0 0 0 R4

20 0 0 0 0 1 0 0 R5

14 0 0 0 0 0 1 1 R6

458 Ch. 4. Numerical Linear Algebra

Starting with row 1, we select x1 with an entry of 1 in it as theentering variable. We enter its column as the PC. An entry of maximumabsolute value in it is 2 in R3. So, we interchange R3 and R1 leadingto the following, in which we perform the G pivot step leading to the2nd working tableau.

BV Memory matrix β PC OCLb R1 R2 R3 R4 R5 R6 x1

−8 0 0 1 0 0 0 2 R3 PR10 0 1 0 0 0 0 2 R2

−4 1 0 0 0 0 0 1 R1

−18 0 0 0 1 0 0 0 R4

20 0 0 0 0 1 0 0 R5

14 0 0 0 0 0 1 1 R6

The 2nd working tableau is

BV Memory matrix β PC OCLb R1 R2 R3 R4 R5 R6 x3

x1 −8 0 0 1 0 0 0 −2 R3

18 0 1 −1 0 0 0 2 R2

0 1 0 −1/2 0 0 0 2 R1 RI to row 2−18 0 0 0 1 0 0 −2 R4

20 0 0 0 0 1 0 0 R5

18 0 0 −1/2 0 0 1 0 R6

Looking at row 2, we see that A2. = β2.A = (0, 6, 2, 1, 1, 1) and weselect x3 with a nonzero entry of 2 in it as the entering variable. ThePC = A.3 = βA.3 = (−2, 2, 2,−2, 0, 0)T . To illustrate row interchangeswe select the 3rd entry in it as the pivot element, thus requiring theinterchange of rows 2 and 3 in the working tableau leading to

4.11 Memory Matrix Versions 459

BV Memory matrix β PC OCLb R1 R2 R3 R4 R5 R6 x3

x1 −8 0 0 1 0 0 0 −2 R3

0 1 0 −1/2 0 0 0 2 R1 PR18 0 1 −1 0 0 0 2 R2

−18 0 0 0 1 0 0 −2 R4

20 0 0 0 0 1 0 0 R5

18 0 0 −1/2 0 0 1 0 R6

Performing the G pivot step leads to the 3rd working tableau.

BV Memory matrix β PC OCLb R1 R2 R3 R4 R5 R6 x2

x1 −8 0 0 1 0 0 0 −4 R3

x3 0 1 0 −1/2 0 0 0 0 R1

18 −1 1 −1/2 0 0 0 6 R2 PR−18 1 0 −1/2 1 0 0 −6 R4

20 0 0 0 0 1 0 0 R5

18 0 0 −1/2 0 0 1 6 R6

Looking at row 3, the updated row is A3. = β3.A = (0, 6, 0, 0,−1, 1)and we select the entering variable to be x2 with an entry of 6 in it.The PC is βA.2 = (−4, 0, 6,−6, 0, 6)T . We make the 6 in row 3 in thePC as the pivot element and row 3 as the PR, so no row interchangeis needed in this step. Performing this G pivot step leads to the 4thworking tableau.

BV Memory matrix β PC OCLb R1 R2 R3 R4 R5 R6 x4

x1 −8 0 0 1 0 0 0 0 R3

x3 0 1 0 −1/2 0 0 0 1 R1

x2 18 −1 1 −1/2 0 0 0 0 R2

0 0 1 −1 1 0 0 0 R4

20 0 0 0 0 1 0 4 R5 PR0 1 −1 0 0 0 1 0 R6

460 Ch. 4. Numerical Linear Algebra

Looking at row 4 we find A4. = β4.A = 0 and b4 = 0 too. So row 4represents a redundant constraint and no pivot step will be performedin it. Eveidence for this redundancy is (0, 1,−1, 1, 0, 0) which impliesthat R4 = −R2 + R3.

Now looking at row 5, the updated row is β5.A = (0, 0, 0, 4, 1, 1) andwe select x4 with an entry of 4 in it as the entering variable. The PCis βA.4 = (0, 1, 0, 0, 4, 0)T . So we keep the 4 in it as the pivot elementand row 5 as the PR. Hence no row interchanges now, the G pivot stepleads to the 5th working tableau.

Final working tableau

BV Memory matrix β PC OCLb R1 R2 R3 R4 R5 R6

x1 −8 0 0 1 0 0 0 R3

x3 0 1 0 −1/2 0 0 0 R1

x2 18 −1 1 −1/2 0 0 0 R2

0 0 1 −1 1 0 0 R4

x4 20 0 0 0 0 1 0 R5 PR0 1 −1 0 0 0 1 R6

The only remaining row is row 6. The updated row 6 is β6.A = 0.Also, the updated RHS constant in this row is 0, hence this row repre-sents a redundant constraint, the evidence for this is (1,−1, 0, 0, 0, 1)which shows that R6 = −R1 + R2.

From the final working tableau we compute βA.1, βA.3, βA.2, βA.4

(the updated columns of x1, x3, x2, x4) and then obtain the followingupdated system by setting the nonbasic variables x5, x6 both at 0.

x1 x3 x2 x4 RHS2 −2 −4 0 −80 1 0 1 00 0 6 0 180 0 0 0 00 0 0 4 200 0 0 0 0

4.12 Orthogonal Sets & Matrices 461

Eliminating the redundant 4th and 6th constraints here leads to thetriangular system

x1 x3 x2 x4 RHS2 −2 −4 0 −80 1 0 1 00 0 6 0 180 0 0 4 20

from which we obtain the basic solution x = (−1/2, 3,−5/2, 5, 0, 6)T

for the original system by back substitution.

4.12 Orthogonal and Orthonormal Sets of

Vectors, Gram-Schmidt Orthogonal-

ization, QR-Factorization, Orthogo-

nal Matrices, Orthonormal Bases

Orthogonal Sets of Vectors

A set of vectors {a1, . . . , ap } in R n is said to be an orthogonal set

of vectors if every pair of distinct vectors from the set is orthogonal,i.e., if the dot product < ai, aj >= 0 for all i �= j.

Example 1: The set of n unit vectors in R n is an orthogonal

set. For example, for n = 3, these are

I. 1 =

100

, I. 2 =

010

, I. 3 =

001

and clearly, < I.i, I.j >= 0 for all i �= j. As another example, considerthe set of vectors {a1, a2, a3, a4} from R3 where

462 Ch. 4. Numerical Linear Algebra

a1 =

000

, a2 =

622

, a3 =

−242

, a4 =

−1−4

7

and verify that the dot product < ai, aj >= 0 for all i �= j, so that theset {a1, a2, a3, a4} is an orthogonal set.

Result 4.12.1: Linear independence of an orthogonal set ofvectors: If Γ = {a1, . . . , ap} is an orthogonal set of nonzero vectors,then Γ is linearly independent.

To see this, suppose scalars α1, . . . , αp exist such that

α1a1 + . . . + αpa

p = 0

For any 1 ≤ i ≤ p take the dot product of both sides of this equationwith ai. This gives

0 = < α1a1 + . . . + αpa

p, ai > = α1 < a1, ai > + . . . + αi−1 < ai−1, ai >

+αi < ai, ai > + αi+1 < ai+1, ai > + . . . + αp < ap, ai >

= αi||ai||2, since < aj, ai > = 0 for all j �= i by orthogonality.

Since ai �= 0 for all i, we have ||ai|| > 0, hence the above equa-tion only holds if αi = 0 for all i; i.e., the only linear combination ofvectors in Γ that is 0 is the 0-linear combination. Hence Γ is linearlyindependent.

Normalizing an Orthogonal Set of Vectors, Or-thonormal Sets

If a �= 0 is any vector in Rn, ||a|| �= 0, and hence we can multiply aby the scalar 1/||a||. The process of multiplying the nonzero vector aby 1/||a|| is called normalizing a. It yields the normalized vectorb = (1/||a||)a satisfying ||b|| = 1.

4.12 Orthogonal Sets & Matrices 463

The zero vector 0 cannot be normalized because ||0|| = 0. Everynonzero vector can be normalized.

As an example, consider a = (−3, 0, 4, 0) ∈ R 4.

||a|| =√

9 + 0 + 16 + 0 = 5.

So normalizing a leads to b = (1/5)a = (−3/5, 0, 4/5, 0) whoseEuclidean norm is 1.

A set of vectors {a1, . . . , ap } in R n is said to be an orthonormal

set of vectors if it is an orthogonal set in which each vector has norm1. For example the following sets of vectors in R

3 are orthonormal sets.I. 1 =

100

, I. 2 =

010

, I. 3 =

001

1√44

622

,

1√24

−242

,

1√66

−1−4

7

An orthogonal set of vectors can be converted into an orthonormalset if all the vectors in it are nonzero, by normalizing each vector in it.

Example 2: Convert the following sets of vectors into orthonor-mal sets.

(i):

1−1

10

,

−1101

,

1101

This set cannot be converted into an orthonormal set by normaliz-ing, because it is not an orthogonal set to start with (the dot productof the 2nd and the 3rd vectors is 1 �= 0).

(ii):

1−1

10

,

−1101

,

1100

464 Ch. 4. Numerical Linear Algebra

This set is an orthogonal set. Dividing each vector in the set by itsnorm leads to the orthonormal set

1√3

1−1

10

,

1√3

−1101

,

1√2

1100

(iii):

1−1

10

,

−1101

,

1100

,

0000

This is an orthogonal set, but since it contains the 0-vector, it can-not be converted into an orthonormal set by normalizing each vector.

Orthogonal, Orthonormal Bases

A basis for a set of vectors, or for the linear hull of a set of vectors,is said to be an orthogonal basis if that basis is an orthogonal set,and an orthonormal basis if it is an orthonormal set.

As an example, the set of n unit vectors in Rn is an orthogo-nal, and also orthonormal basis for Rn. For n = 3, this basis is{(1, 0, 0), (0, 1, 0), (0, 0, 1)}.

Representation of Vectors in Terms of anOrthonormal Basis

Result 4.12.2: Representation of a vector in terms of anorthonormal basis: Let {a1, . . . , am} be an orthonormal basis for agiven set of vectors Γ. Let a be a vector in the linear hull of Γ. Thenas defined in Section 4.2, the representation of a in terms of the basis{a1, . . . , am} is the unique vector {α1, . . . , αm} satisfying

a = α1a1 + . . . + αmam.

Then, (α1, . . . , αm) = (< a1, a >, . . . , < am, a >).

4.12 Orthogonal Sets & Matrices 465

To see this, we have for any i = 1 to m, < ai, a > = < ai, α1a1 +

. . .+ αmam > = α1 < ai, a1 > + . . .+ αm < ai, am > = αi < ai, ai >= αi; because < ai, aj > = 0 for all j �= i and < ai, ai > = 1 by theorthonormality of the set {a1, . . . , am }.

Example 3: As an example, consider the set Γ = {a1, . . . , a5 }where

a1 =

100000

, a2 =

0−2/3

2/31/3

00

, a3 =

01/21/2

0−1/2−1/2

,

a4 =

1−1

31

−1−1

, a5 =

1−3

52

−1−1

.

{a1, a2, a3} is an orthonormal basis for this set. Consider the vector

a = a1 − 6a2 − 4a3 + a4 − a5 = (1, 4,−8,−3, 2, 2)T

in the linear hull of Γ. Then from Result 4.12.2, the representation ofa in terms of the orthonormal basis {a1, a2, a3} is (< a1, a >, < a2, a >, < a3, a >) = (1,−9,−4). In fact we verify that a = a1 − 9a2 − 4a3.

The Gram-Schmidt Orthogonalization Procedure

Any linearly independent set of vectors, ∆, can be converted intoan orthonormal set of vectors with the same linear hull as ∆ usingthis procedure. The procedure is named after J. P. Gram of Denmark,and E. Schmidt of Germany who developed it in the late 19th andearly 20th centuries. However, the same procedure was used by otherresearchers much earlier.

466 Ch. 4. Numerical Linear Algebra

Let ∆ = {a1, . . . , ak} be a linearly independent set of vectors inRn say. The Gram-Schmidt procedure converts this into an or-thonormal set of vectors {q1, . . . , qk} generating the vectors q1, . . . , qk

sequentially in this specific order; that’s why often the vectors q1, . . . , qk

are referred to as the Gram-Schmidt sequence. The procedure is:

q1 = (1/ν1)a1

and for r = 2 to k

qr = (1/νr)(ar −

r−1∑i=1

< qi, ar > qi)

where ν1 = ||a1||, νr = ||(ar −∑r−1i=1 < qi, ar > qi)|| for r = 2 to k.

For some r between 2 to k, if (ar −∑r−1i=1 < qi, ar > qi) becomes 0,

we have a linear dependence relation (because this equation gives ar

as a linear combination of {a1, . . . , ar−1}) which implies that the set∆ is linearly dependent, and the procedure cannot continue further.So, if ∆ is linearly independent as originally assumed, the proceduregenerates the complete Gram-Schmidt sequence.

From the formula for qr, it is clear that ||qr|| = 1 for all r = 1 to kwhen the entire sequence is obtained. Also, the fact that < qj, qr >= 0for all j ≤ r − 1 can be established inductively. For example,

< q2, q1 > = < (1/ν2)(a2− < q1, a2 > a1), q1 >

= (1/ν2)(< a2, q1 > − < q1, a2 >< a1, q1 >)

= (1/ν2) < a2, q1 > (1− < a1, q1 >) = 0

because < a1, q1 > = 0.It can also be verified that the Gram-Schmidt sequence q1, . . . , qk

satisfies the following property: for all r = 1 to k

linear hull of {q1, . . . , qr} = linear hull of {a1, . . . , ar}.

The Gram-Schmidt procedure given above is mathematically cor-rect, but if implemented in a straightforward way using floating point

4.12 Orthogonal Sets & Matrices 467

arithmetic, the sequence produced may be far from being an orthogo-nal set because of roundoff errors. Numerical analysts have developedmodified versions of the Gram-Schmidt algorithm using product formimplementations that are much more stable. These are outside thescope of this book, the reader is referred to C. Meyer [4.2] for details.

We will provide a numerical example of Gram-Schmidt orthogonal-ization, after discussing the QR-factorization of a matrix with a linearlyindependent set of column vectors.

The QR-Factorization (or Decomposition) of aMatrix

Let A = (aij) be an m × n matrix of rank n, i.e., the set ofcolumn vectors of A, {A.1, . . . , A.n} is linearly independent. Since{A.1, . . . , A.n} is linearly independent, we can convert it into an or-thonormal set of n column vectors {Q.1, . . . , Q.n} using the Gram-Schmidt orthogonalization discussed above. This procedure obtainsthese vectors sequentially using the formulas

Q.1 = (1/ν1)A.1

Q.r = (1/νr)(A.r −r−1∑i=1

< Q.i, A.r > Q.i), r = 2, . . . , k

where ν1 = ||A.1||, and for r = 2 to k, νr = ||A.r −∑r−1i=1 < Q.i, A.r >

Q.i||. Let

Q = (Q.1... . . .

...Q.n)

be the matrix with Q.1, . . . , Q.n as its column vectors. The above equa-tions can be written as

A.1 = ν1Q.1

A.r = < Q.1, A.r > Q.1+ < Q.r−1, A.r > Q.r−1 + νrQ.r, r = 2, . . . , n

In matrix notation, these equations state that

468 Ch. 4 . Numerical Linear Algebra

A = QR

where

R =

ν1 < Q. 1, A. 2 > < Q. 1, A. 3 > . . . < Q. 1, A.n >0 ν2 < Q. 2, A. 3 > . . . < Q. 2, A.n >0 0 ν3 . . . < Q. 3, A.n >...

......

...0 0 0 . . . νn

.

So, R is a square upper triangular matrix of order n. The factoriza-tion of A as QR is called the QR-factorization (sometimes also calledQR-decomposition) for A, and it is uniquely determined by A.

A matrix has a QR-factorization iff its set of column vectors islinearly independent. If the set of column vectors of Am× n is linearlyindependent, it can be expressed as Qm× nRn × n where the columns ofQ form an orthonormal set, and R is upper triangular with positivediagonal entries.

The QR factorization for Am× n is called the rectangular QR-factorization or decomposition if m > n; and the square QR-factorization or decomposition if m = n.

Example 4: Find the QR-factorization of

A =

−1 0 00 1 01 1 1

The column vectors of A are

A.1 =

−101

, A.2 =

011

, A.3 =

001

Applying the Gram-Schmidt procedure on the set of column vectorsof A , we get the orthonormal set of vectors

4.12 Orthogonal Sets & Matrices 469

Q.1 = (1/||A.1||)A.1 = (1/√

2)

−101

Q.2 = (A.2− < Q.1, A.2 > Q.1)/(its norm)

= (√

(2/3))

1/21

1/2

Q.3 = (A.3− < Q.1, A.3 > Q.1− < Q.2, A.3 > Q.2)/(its norm)

=√

3

1/3−1/3

1/3

.

Calculating R by the formula given above, the QR-factorization ofA has

Q =

−1/√

2 1/√

6 1/√

3

0√

2/√

3 −1/√

3

1/√

2 1/√

6 1/√

3

R =

√2 1/

√2 1/

√2

0√

3/√

2 1/√

6

0 0 1/√

3

.

The QR-factorization is an extremely useful tool in numerical com-putation for computing the eigen values of matrices (discussed in Chap-ter 6) and for many other applications.

Orthogonal Matrices

An orthogonal matrix is a square matrix whose columns forman orthonormal set. Since the columns of a square matrix form an or-thonormal set iff its rows form an orthonormal set, the orthogonality ofa square matrix can be defined either in terms of orthonormal columnsor orthonormal rows.

Thus a square matrix An×n is orthogonal iff

470 Ch. 4. Numerical Linear Algebra

||A.j|| = ||Ai.|| = 1 for all i, j = 1 to n< A.i, A.j >=< Ai., Aj. >= 0 for all i �= j.

i.e., iff AT A = AAT = the unit matrix I.Examples of orthogonal matrices are all permutation matrices, for

instance

0 1 00 0 11 0 0

Another example is

1/√

2 1/√

3 −1/√

6

−1/√

2 1/√

3 −1/√

6

0 1/√

3 2/√

6

Therefore a square matrix A is orthogonal iff A−1 = AT . Orthogo-nal matrices satisfy the following properties.

1 The product of orthogonal matrices is orthogonal.

2 The inverse of an orthogonal matrix is orthogonal.

3 The determinant of an orthogonal matrix is ±1.

4 If A is an orthogonal matrix of order n, and x is a column vectorin Rn, then ||Ax|| = ||x||.

Orthogonal matrices play a very important role in the problem ofdiagonalizing square matrices discussed in Chapters 5, 6; and in manyother applications.

4.13 Additional Exercises

1. Consider the system of linear equations Ax = b where A is amatrix of order m × n.

If the system has a unique solution, what does it imply aboutrank(A)?

4.13 Additional Excersises 471

If the system is known to have a unique solution, and m = n, thenshow that the system has a unique solution for all b ∈ Rm.

If the system is known to have a unique solution, and m > n showthat the system has either unique solution, or no solution for all b ∈ Rm.

If the system is known to have a unique solution, and m > n, showthat the system Ax = 0 has no nonzero solution.

2. The set of vectors {a1, . . . , am, am+1} in Rn is linearly dependent.Does this mean that am+1 can be expressed as a linear combination of{a1, . . . , am}? Either provide an argument why it must be true, or acounterexample.

How does the answer to the above question change if it is also giventhat {a1, . . . , am} is linearly independent? Why?

3. Let x be a solution to the system Ax = b.Also, suppose y is a solution for the homogeneous system Ay = 0.Then show that x+λy is a solution for Ax = b for all real values

of λ.

4. Let a1 = (1, 1,−1)T , a2 = (−1, 1, 1)T , a3 = (p, q, r)T .Express the conditions that p, q, r have to satisfy for a3 to be a linear

combination, affine combination, convex combination, of {a1, a2}.

5. Show that the set of vectors {(a, b), (c, d)} in R2 is linearlyindependent iff ad − bc �= 0.

6. Discuss how the rank of the following matrix varies as a, b takeall real values.

−1 1 1 12 −3 −1 2a 11 5 −8b −5 −1 3

.

7. (From H. G. Campbell, Linear Algebra With Applications, Pren-tice Hall, 1980). Show that the rank of the following matrix takes onlytwo different values as a varies. For what values of a is the rank 3?

472 Ch. 4. Numerical Linear Algebra

2 4 22 1 21 0 a

.

8. A, B are orthogonal matrices of the same order.What are their inverses? What are the values of their determinants?Show that AB is also an orthogonal matrix.If A, B are also symmetric, show that AB = BA.

4.14 References

[4.1] P. E. Gill, W. Murray, and M. H. Wright, Numerical Linear Alge-bra and Optimization, Volume 1, Addison Wesley, Redwood City, CA,1991.[4.2] C. P. Meyer, Matrix Analysis and Applied Linear Algebra, SIAM,Philadelphia, PA, 2000.

Index

For each index entry we providethe section number where it isdefined or discussed first. Click-ing on the light blue section num-ber takes you to the page wherethis entry is defined or discussed.

Alternate system 4.11Augmentation scheme 4.2

Basic vectors 4.2Basis 4.2

Dantzig 4.11Dense matrix 4.6Dropping basic vector 4.4Duality theorem 4.10

Elementary matrices 4.7Entering variable 4.11Entering vector 4.4Eta column 4.7Evidence of infeasibility 4.11Evidence of redundancy 4.11

Fill-in 4.7

Gram-Schmidt orthogonaliza-tion 4.10

Infeasibility analysis 4.11Integer programming 4.10Introducing nonbasic vectors 4.4

Linear dependence 4.1Linear dependence relation 4.1Linear independence 4.1Linear inequality as Nonlinear

eq. 4.10Linear optimization 4.11Linear programming 4.11

Matrix factorizations 4.7Matrix inversion 4.6, 4.7Maximal linear independence 4.2Memory matrix 4.3Memory matrix version of “GJ”,

“G” 4.11Minimal spanning subset 4.2

nonbasic vectors 4.2Nonsingularity 4.5

Orthogonal matrices 4.10Orthogonal sets 4.10Orthonormal basis 4.10Orthonormal sets 4.10

473

474 Ch. 4. Numerical Linear Algebra

Pivot matrices 4.7Product form of inverse 4.7

QR-decomposition 4.10QR-factorization 4.10

Rank 4.2Representation 4.2Round-off errors 4.7Row reduction method 4.3, 4.5

Slack variables 4.10Spanning subset 4.2Sparse matrix 4.6String computation 4.7

Left to right 4.7Right to left 4.7

unit basis 4.2Updated column 4.3Updating the inverse 4.8

Working tableau 4.11Initial 4.11Current or updated 4.11

Index for Ch. 4 475

List of algorithms

1. Algorithm 1: Checking linear dependence of a set of vectors us-ing the memory matrix

2. Algorithm 2: Checking linear dependence of a set of vectors byrow reduction on the system of equations in the linear dependence re-lation

3. Algorithm to compute the inverse of a square matrix A using GJpivot steps

4. Algorithm to compute the inverse of a square matrix in productform

5. Algorithm to Compute the Inverse of B Obtained By ReplacingB.r in B By d, From B− 1

6. Algorithm to determine which possibility occurs for a linear op-timization problem subject to linear equations

7. Memory matrix versions of GJ, G elimination methods for solv-ing a system of linear equations.

Historical note on:

1. Rank.

Definition of:

1. Basis

2. Representation of a nonbasic vector in terms of the basis.

476 Ch. 4. Numerical Linear Algebra

List of Results

1. Result 4.2.1: Maximal linear independence and minimal span-ning property imply each other

2. Result 4.2.2: Rank of a set of vectors

3. Result 4.2.3: Ranks of subsets of R n

4. Result 4.2.4: Basic vectors span each nonbasic vector uniquely

5. Result 4.5.1: Row rank and column rank are the same

6. Result 4.5.2: Rank and nonsingularity

7. Results on the General System of Linear Equations Ax = b

8. Result 4.12.1: Linear independence of an orthogonal set of vec-tors

9. Result 4.12.2: Representation of a vector in terms of an or-thonormal basis.

Theorem

1. Duality theorem of linear algebra.

Contents

5 Quadratic Forms, Positive, Negative (Semi) Definite-ness 4775.1 Expressing a Quadratic Function in n Variables Using

Matrix Notation . . . . . . . . . . . . . . . . . . . . . . 4775.2 Convex, Concave Functions; Positive (Negative) (Semi)

Definiteness; Indefiniteness . . . . . . . . . . . . . . . . 4805.3 Algorithm to Check Positive (Semi) Definiteness Using

G Pivot Steps . . . . . . . . . . . . . . . . . . . . . . . 4845.4 Diagonalization of Quadratic Forms and Square Matrices 497

i

ii

Chapter 5

Quadratic Forms, Positive,Negative (Semi) Definiteness

This is Chapter 5 of “Sophomore Level Self-Teaching Web-book for Computational and Algorithmic Linear Algebra andn-Dimensional Geometry” by Katta G. Murty.

5.1 Expressing a Quadratic Function in n

Variables Using Matrix Notation

As defined in Section 1.7, a linear function of x = (x1, . . . , xn)T is afunction of the form c1x1 + . . . + cnxn where c1, . . . , cn are constantscalled the coefficients of the variables in this function. For example,when n = 4, f1(x) = −2x2 + 5x4 is a linear function of (x1, x2, x3, x4)

T

with the coefficient vector (0,−2, 0, 5).

An affine function of x = (x1, . . . , xn)T is a constant plus a linearfunction, i.e., a function of the form f2(x) = c0 + c1x1 + . . .+ cnxn. So,if f2(x) is an affine function, f2(x) − f2(0) is a linear function. As anexample, −10 − 2x2 + 5x4 is an affine function of (x1, x2, x3, x4)

T .

A quadratic form in the variables x = (x1, . . . , xn)T consists of thesecond degree terms of a second degree polynomial in these variables,i.e., it is a function of the form

477

478 Ch. 5. Quadratic Forms

Q(x) =n∑

i=1

qiix2i +

n∑i=1

n∑j=i+1

qijxixj

where the qij are the coefficients of the terms in the quadratic form.Define a square matrix D = (dij) of order n where

dii = qii for i = 1 to n

and dij, dji are arbitrary real numbers satisfying

dij + dji = qij for j > i.

As an example, if q12 = −10, we could take (d12 = d21 = −5), or(d12 = 100, d21 = −110), or (d12 = −4, d21 = −6), etc. Then

Dx =

d11x1 + d12x2 + . . . + d1nxn

d21x1 + d22x2 + . . . + d2nxn...

dn1x1 + dn2x2 + . . . + dnnxn

So

xT Dx = x1(d11x1 + . . . + d1nxn) + . . . + xn(dn1x1 + . . . + dnnxn)

=n∑

i=1

diix2i +

n∑i=1

n∑j=i+1

(dij + dji)xixj

=n∑

i=1

qiix2i +

n∑i=1

n∑j=i+1

qijxixj

= Q(x)

So, for any matrix D = (dij) satisfying the conditions stated above,we have xT Dx = Q(x). Now define D = (dij) by

dii = qii i = 1, . . . , n

dij = dji = (1/2)qij j > i

5.1 Expressing in Matrix Notation 479

then D is a symmetric matrix and Q(x) = xT Dx. D is known asthe symmetric coefficient matrix defining the quadratic formQ(x).

Clearly for all D = (dij) satisfying the conditions stated above,

we have Q(x) = xT Dx = xT DT x = xT (D+DT

2)x and D+DT

2= D, the

symmetric coefficient matrix defining the quadratic form Q(x).

As an example consider

n = 3, x = (x1, x2, x3)T , Q(x) = 81x2

1 − 7x22 + 5x1x2 − 6x1x3 + 18x2x3

Then the following square matrices satisfy the conditions statedabove for Q(x).

D1 =

81 −10 10015 −7 10

−106 8 0

, D2 =

81 200 −1006−195 −7 2181000 −200 0

,

D3 =

81 2 −23 −7 15

−4 3 0

, D =

81 5/2 −35/2 −7 9−3 9 0

It can be verified that xT D1x = xT D2x = xT D3x = xT Dx = Q(x),and that

D =D1 + DT

1

2=

D2 + DT2

2=

D3 + DT3

2

Hence a general quadratic form in n variables x = (x1, . . . , xn)T

can be represented in matrix notation as xT Mx where M = (mij) isa square matrix of order n. If M is not symmetric, it can be replacedin the above formula by (M + MT )/2 without changing the quadraticform, and this (M +MT )/2 is known as the symmetric matrix definingthis quadratic form.

A quadratic function in variables x = (x1, . . . , xn)T is a functionwhich is the sum of a quadratic form in x and an affine function in x;i.e., it is of the form xT Dx+ cx+ c0 for some square matrix D of ordern, row vector c ∈ Rn and constant term c0.

480 Ch. 5 . Q uadratic Fo rms

Exercises

5.1.1: Express the following functions in matrix notation with asymmetric coefficient matrix: (i): −6x2

1 +7x22 −14x2

4 −12x1x2 +20x1x3

+6x1x4 −7x2x3 + 8x3x4 −9x1 + 6x2 − 13x3 + 100 (ii): x21 − x2

2 + x23

−18x1x3 + 12x2x3 −7x1 + 18x2 (iii): 4x21 + 3x2

2 − 8x1x2 (iv):6x1 + 8x2 − 11x3 + 6.

5.1.2: Express the quadratic form xT Dx+cx as a sum of individualterms in it for the following data:

(i) : D =

3 −6 9 810 −2 0 1213 −11 0 3−4 −9 9 1

, c =

1−9

07

.

(ii) : D =

3 −4 8−4 9 9−8 −9 2

, c =

006

.

(iii) : D =

1 1 1 1 11 1 1 1 11 1 1 1 11 1 1 1 11 1 1 1 1

, c = 0.

5.2 Convex, Concave Functions; Positive

(Negative) (Semi) Definiteness; Indef-

initeness

Consider the real valued function f(x) defined over x ∈ Rn, or oversome convex subset of Rn. It is said to be a convex function if forevery x, y for which it is defined, and 0 ≤ α ≤ 1, we have

f(αx + (1 − α)y) ≤ αf(x) + (1 − α)f(y)

5.2 Positive (Negative) (Semi) Definiteness 481

This inequality defining the convexity of the function f(x) is calledJensen’s inequality after the Danish mathematician who first definedit. This inequality is easy to visualize when n = 1. It says that if youjoin two points on the graph of the function by a chord, then thefunction itself lies underneath the chord on the interval joining thesepoints. See Figure 5.1.

x1x

x2

.

.

.

.

..

f(x)

chord

αf(x1 + (1 - α)f(x2))

f(x1)

f(x2)

αx1+ (1 - α)x2

Figure 5.1: A convex function lies beneath any chord

The convex function f(x) is said to be a strictly convex functionif the above inequality holds as a strict inequality for every x �= y and0 < α < 1.

A real valued function g(x) defined over Rn or a convex subset ofRn is said to be a concave function if −g(x) is a convex functionas defined above, i.e., if for every x, y for which it is defined, and0 ≤ α ≤ 1, we have

g(αx + (1 − α)y) ≥ αg(x) + (1 − α)g(y)

482 Ch. 5. Quadratic Forms

x1x

x2

.

.

.

.

..

g(x)

chordαg(x1 + (1 - α)g(x2))

g(x1)

g(x2)

αx1+ (1 - α)x2

Figure 5.2: A concave function stays above any chord

The concave function g(x) is said to be a strictly concave func-tion if −g(x) is a strictly convex function, i.e., if the above inequalityholds as a strict inequality for every x �= y and 0 < α < 1.

The properties of convex and concave functions are of great impor-tance in optimization theory. From the definition it can be verified thata function f(x1) of a single variable x1 is

Convex iff its slope is nondecreasing, i.e., iff its 2nd deriva-tive is ≥ 0 when it is twice continuously differentiable

Concave iff its slope is nonincreasing, i.e., iff its 2nd deriva-tive is ≤ 0 when it is twice continuously differentiable.

As examples, some convex functions of a single variable x1 are:

x21, x

41, e

−x1,− log(x1)(in the region x1 > 0).

5.2 Positive (Negative) (Semi) Definiteness 483

Among functions of several variables, linear and affine functions areboth convex and concave, but they are not strictly convex or strictlyconcave. For classifying quadratic functions into the convex and con-cave classes, we need the following definitions.

A square matrix M of order n, whether symmetric or not, is saidto be

Positive semidefinite (PSD) iff yTMy ≥ 0 for all y ∈ Rn

Positive definite (PD) iff yTMy > 0 for all y ∈ Rn,y �= 0

Negative semidefinite (NSD) iff yTMy ≤ 0 for all y ∈ Rn

Negative definite (ND) iff yTMy < 0 for all y ∈ Rn,y �= 0

Indefinite if it is neither PSD nor NSD,i.e., iff there are points x, y ∈Rn satisfying xT Mx > 0, andyT My < 0.

We have the following results.

Result 5.2.1: Conditions for a quadratic function to be(strictly) convex: The quadratic function f(x) = xT Dx + cx + c0

defined over Rn is a convex function over Rn iff the matrix D is PSD;and a strictly convex function iff the matrix D is PD.

To see this, take any two points x, y ∈ Rn, and a 0 < α < 1. It canbe verified that

αf(x) + (1 − α)f(y) − f(αx + (1 − α)y) = α(1 − α)(x − y)TD(x − y)

by expanding the terms on both sides. So Jensen’s inequality for theconvexity of f(x) holds iff

(x − y)TD(x − y) ≥ 0 for all x, y ∈ Rn

484 Ch. 5 . Quadratic Fo rms

i.e., iff z T Dz ≥ 0 for all z ∈ R

n, i.e., iff D is PSD. Likewise for strictconvexity of f(x) we need

(x − y)TD(x − y) > 0 for all x �= y ∈ R n

i.e., iff z T Dz > 0 for all z ∈ R

n, z �= 0, i.e., iff D is PD.

Result 5.2.2: Conditions for a quadratic function to be(strictly) concave: The quadratic function f(x) = xT Dx + cx + c0

defined over R n is a concave function over R

n iff the matrix D is NSD;and a strictly concave function iff the matrix D is ND.

This follows by applying Result 5.2.1 to −f(x).

Checking whether a given quadratic function is strictly convex, orconvex, is a task of great importance in optimization, statistics, andmany other sciences. From Result 5.2.1, this task is equivalent tochecking whether a given square matrix is PD, PSD. We discuss efficientalgorithms for PD, PSD checking based on Gaussian pivot steps in thenext section.

5.3 Algorithm to Check Positive (Semi)

Definiteness Using G Pivot Steps

Submatrices of a Matrix

Let A be a matrix of order m × n. Let S ⊂ {1, . . . , m} and T ⊂{1, . . . , n}. If we delete all the rows of A not in S, and all the columnsnot in T , what remains is a smaller matrix denoted by AS×T known asthe submatrix of A corresponding to the subset of rows S andcolumns T . As an example, consider the following matrix

A =

6 0 −7 8 0 −9−25 −1 2 14 −15 −4

0 3 −2 −7 0 1011 16 17 −5 4 0

.

5.3 Algorithm to Check Positive (Semi) Definiteness 485

Let S1 = {1, 4}, S2 = {2, 3}, T1 = {1, 3, 4, 6}, T2 = {2, 5, 6}. Thenthe submatrices AS1×T1 , AS2×T2 are

AS1×T1 =

(6 −7 8 −9

11 17 −5 0

), AS2×T2 =

( −1 −15 −43 0 10

).

Principal Submatrices of a Square Matrix

Square matrices have special submatrices called principal subma-trices. Let M be a square matrix of order n, and S ⊂ {1, . . . , n} withr elements in it. Then the submatrix MS×S obtained by deleting fromM all rows and all columns not in the set S is known as the principalsubmatrix of the square matrix M corresponding to the sub-set S, it is a square matrix of order r. The determinant of the principalsubmatrix of M corresponding to the subset S is called the principalminor of M corresponding to the subset S. As an example, let

M =

4 2 3 51 22 12 −1

−4 −2 −5 60 −3 16 −7

.

Let S = {2, 4}. Then the principal submatrix of M correspondingto S is

MS×S =

(22 −1−3 −7

).

and the principal minor of M corresponding to S is determinant(MS×S)which is −157.

In this same example, consider the singleton subset S1 = {3}. Thenthe principal submatrix of M corresponding to S1 is (−5) of order 1,i.e., the third diagonal element of M . In the same way, all diagonalelements of a square matrix are its principal submatrices of order 1.

486 Ch. 5. Quadratic Forms

Some Results on PSD, PD, NSD, ND Matrices

Result 5.3.1: Properties shared by all principal submatri-ces: If M is a square matrix of order n which has any of the propertiesPSD, PD, NSD, ND; then all principal submatrices of M have the sameproperty.

To see this, consider the subset S ⊂ {1, 2} as an example, and theprincipal submatrix MS×S of order 2. Let y = (y1, y2, 0, . . . , 0)T ∈ Rn

with yi = 0 for all i �∈ S. Since y3 to yn are all 0 in y, we verify thatyTMy = (y1, y2)MS×S(y1, y2)

T . So, for all y1, y2, if

yTMy ≥ 0 so is (y1, y2)MS×S(y1, y2)T

yTMy > 0 for all (y1, y2) �= 0, so is (y1, y2)MS×S(y1, y2)T

yTMy ≤ 0 so is (y1, y2)MS×S(y1, y2)T

yTMy < 0 for all (y1, y2) �= 0 so is (y1, y2)MS×S(y1, y2)T .

Hence, if M is PSD, PD, NSD, ND, MS×S has the same property.A similar argumant applies to all principal submatrices of M .

Result 5.3.2: Conditions for a 1 × 1 matrix: A 1 × 1 matrix(a) is

PSD iff a ≥ 0

PD iff a > 0

NSD iff a ≤ 0

ND iff a < 0.

The quadratic form defined by (a) is ax21; and it is ≥ 0 for all x1 iff

a ≥ 0; it is > 0 for all x1 �= 0 iff a > 0; etc. Hence this result followsfrom the definitions.

Result 5.3.3: Conditions satisfied by diagonal entries: LetM = (mij) be a square matrix. If M is a

5.3 Algorithm to Check Positive (Semi) Definiteness 487

PSD matrix, all its diagonal entries must be ≥ 0

PD matrix, all its diagonal entries must be > 0

NSD matrix, all its diagonal entries must be ≤ 0

ND matrix, all its diagonal entries must be < 0.

Since all diagonal entries of a square matrix are its 1 × 1 principalsubmatrices, this result follows from Results 5.3.1 and 5.3.2.

Result 5.3.4: Conditions satisfied by the row and columnof a 0-diagonal entry in a PSD matrix: Let M = (mij) be a squarePSD matrix. If one of its diagonal entries, say mpp = 0, then for all jwe must have mpj + mjp = 0.

Suppose m11 = 0, and m12 + m21 = α �= 0. Let the principalsubmatrix of M corresponding to the subset {1, 2} be

M =

(m11 m12

m21 m22

)=

(0 m12

m21 m22

)

The quadratic form defined by M is (y1, y2)M(y1, y2)T = m22y

22 +

αy1y2. By fixing

y1 =−1 − m22

α, y2 = 1

we see that this quadratic form has value −1, so M is not PSD, andby Result 5.3.1 M cannot be PSD, a contradiction. Hence α must be0 in this case. The result follows from the same argument.

Result 5.3.5: Conditions satisfied by the row and column ofa 0-diagonal entry in a symmetric PSD matrix: Let M = (mij)be a square matrix. Symmetrize M , i.e., let D = (dij) = (M + MT )/2.M is PSD iff D is, because the quadratic forms defined by M and Dare the same. Also, since D is symmetric, we have dij = dji for all i, j.Hence, if D is PSD, and a diagonal entry in D, say dii = 0, then all

488 Ch. 5 . Q uadratic Fo rms

the entries in the ith row and the ith column of D must be zero, i.e.,Di. = 0 and D.i = 0.

This follows from Result 5.3.4.

Example 1: Consider the following matrices.

M1 =

10 3 1−2 0 0

1 0 4

, M2 =

100 0 20 10 32 3 −1

M3 =

10 3 −33 10 63 −6 0

, M4 =

2 0 −70 0 0

10 0 6

The matrix M1 is not symmetric, and in it m22 = 0 but m12+m21 =3− 2 = 1 �= 0, hence this matrix violates the condition in Result 5.3.4,and is not PSD.

In the matrix M2, the third diagonal entry is −1, hence by Result5.3.3 this matrix is not PSD.

In the matrix M3, m33 = 0, and m13 + m31 = m23 + m32 = 0, hencethis matrix satisfies the condition in Result 5.3.4.

The matrix M4 is symmetric, its second diagonal entry is 0, andits 2nd row and 2nd column are both zero vectors. Hence this matrixsatisfies the condition in Result 5.3.5.

Result 5.3.6: Conditions satisfied by a principal submatrixobtained after a G pivot step: Let D = (dij) be a symmetricmatrix of order n with its first diagonal entry d11 �= 0. Perform aGaussian pivot step on D with d11 as the pivot element, and let D bethe resulting matrix; i.e. transform

D =

d11 . . . d1n

d21 . . . d2n...

...dn1 . . . dnn

into D =

d11 d12 . . . d1n

0 d22 . . . d2n...

......

0 dn2 . . . dnn

5.3 Algorithm to Check Positive (Semi) Definiteness 489

Let D1 be the matrix of order (n− 1)× (n− 1) obtained by deletingrow 1 and column 1 from D. Then D1 is symmetric too, and D is PD(PSD) iff d11 > 0 and D1 is PD (PSD).

Since the original matrix D is symmetric, we have dij = dji for alli, j. Using this and the formula coming from the Gaussian pivot step,that for i, j = 2 to n, dij = dij − di1(d1j/d11) it can be verified thatdij = dji, hence D1 is symmetric.

The other part of this result can be seen from the following.

xT Dx = d11[x21 + 2(d12/d11)x1x2 + . . . + 2(d1n/d11)x1xn

+(1/d11)n∑

i=2

n∑j=2

dijxixj ]

Let

θ = (d12/d11)x2 + . . . + (d1n/d11)xn

δ = (1/d11)n∑

i=2

n∑j=2

dijxixj

Then xT Dx = d11[x21 +2x1θ + δ] = d11[(x1 + θ)2 +(δ− θ2)], because

x21 + 2x1θ = (x1 + θ)2 − θ2. In mathematical literature this argument

is called completing the square argument.After rearranging the terms it can be verified that

δ − θ2 = (x2, . . . , xn)D1(x2, . . . , xn)T . So,

xT Dx = d11(x1 + θ)2 + (x2, . . . , xn)D1(x2, . . . , xn)T

From this it is clear that D is is PD (PSD) iff d11 > 0 and D1 isPD (PSD).

Based on these results, we now describe algorithms to check whethera given square matrix is PD, PSD. To check if a square matrix is ND,NSD, apply the following algorithms to check if its negative is PD,PSD.

490 Ch. 5 . Q uadratic Fo rms

Algorithm to Check if a Given Square Matrix Mof Order n is PD

BEGIN

Step 1: First symmetrize M , i.e., compute D = (M +M T )/2. The

algorithm works on D.If any of the diagonal entries in D are ≤ 0, D and M are not PD

by Result 5.3.3, terminate.

Step 2: Start performing Gaussian pivot steps on D using thediagonal elements as the pivot elements in the order 1 to n. At anystage of this process, if the current matrix has an entry ≤ 0 in its maindiagonal, terminate with the conclusion that D, M are not PD. If allthe pivot steps are completed and all the diagonal entries are > 0,terminate with the conclusion that D, M are PD.

END

Example 2: Check whether the following matrix M is PD.

M =

3 1 2 2−1 2 0 2

0 4 4 5/30 −2 −13/3 6

Symmetrizing, we get

D =

3 0 1 10 2 2 01 2 4 −4/31 0 −4/3 6

All the entries in the principal diagonal of D are > 0. So, we applythe algorithm, and obtain the following results. The pivot elements areboxed in the following. PR, PC indicate pivot row, pivot column foreach G pivot step.

5.3 Algorithm to Check Positive ( Semi) D efiniteness 491

PC↓PR 3 0 1 1

0 2 2 01 2 4 −4/31 0 −4/3 6

PC↓3 0 1 1

PR 0 2 2 00 2 11/3 −5/30 0 −5/3 17/3

PC↓3 0 1 10 2 2 0

PR 0 0 5/3 −5/3

0 0 −5/3 17/33 0 1 10 2 2 00 0 5/3 −5/30 0 0 4

The algorithm terminates now. Since all the diagonal entries in allthe tableaus are > 0, D and hence M are PD.

Example 3: Check whether the following matrix M is PD.

M =

1 0 2 00 2 4 02 4 4 50 0 5 3

The matrix M is already symmetric and its diagonal entries are> 0. So, we apply the algorithm on it, and obtain the following results.The pivot elements are boxed in the following. PR, PC indicate pivotrow, pivot column for each G pivot step.

492 Ch. 5. Quadratic Forms

PC↓PR 1 0 2 0

0 2 4 02 4 4 50 0 5 31 0 2 00 2 4 00 4 0 50 0 5 3

Since the third diagonal entry in this tableau is 0, we terminatewith the conclusion that this matrix M is not PD.

A square matrix which is not PD, could be PSD. We discuss thealgorithm based on G pivot steps, for checking whether a given squarematrix is PSD, next.

Algorithm to Check if a Given Square Matrix Mof Order n is PSD

BEGIN

Initial step: First symmetrize M , i.e., compute D = (M +MT )/2.The algorithm works on D. Apply the following General step on thematrix D.

General Step: If any of the diagonal entries in the matrix are< 0, terminate with the conclusion that the original matrix is not PSD(Results 5.3.3, 5.3.6). Otherwise continue.

Check if the matrix has any 0-diagonal entries. For each 0-diagonalentry check whether its row and column in the matrix are both com-pletely 0-vectors, if not the original matrix is not PSD (Results 5.3.5,5.3.6), terminate. If this condition is satisfied, delete the 0-row vectorand the 0-column vector of each 0-diagonal entry in the matrix.

If the remaining matrix is of order 1 × 1, its entry will be > 0,terminate with the conclusion that the original matrix is PSD.—

5.3 Algorithm to Check Positive ( Semi) D efiniteness 493

If the remaining matrix is of order ≥ 2, its first diagonal entrywill be positive, perform a G pivot step on the matrix with this firstdiagonal element as the pivot element. After this pivot step, delete row1, column 1 of the resulting matrix.

Now apply the same general step on the remining matrix, and repeatthe same way.

END

Example 4: Check whether the following matrix M is PSD.

M =

0 −2 −3 −4 52 3 3 0 03 3 3 0 04 0 0 8 4

−5 0 0 4 2

Symmetrizing, we get

D =

0 0 0 0 00 3 3 0 00 3 3 0 00 0 0 8 40 0 0 4 2

The first diagonal entry in D is 0, and in fact D1., D.1 are both0, so we eliminate them and apply the general step on the remainingmatrix. Since there is a 0-diagonal entry in D, the original matrix Mis not PD, but it may possibly be PSD.

PC↓PR 3 3 0 0

3 3 0 00 0 8 40 0 4 23 3 0 00 0 0 00 0 8 40 0 4 2

494 Ch. 5 . Q uadratic Fo rms

Eliminating row 1 and column 1 in the matrix resulting after thefirst G step leads to the matrix

0 0 00 8 40 4 2

The diagonal entries in this matrix are all ≥ 0, and the first diagonalentry is 0. Correspondingly, row 1 and column 1 are both 0-vectors, sowe delete them, and apply the general step on the remaining matrix.

PC↓PR 8 4

4 28 40 0

.

Eliminating row 1 and column 1 in the matrix resulting after thisG pivot step leads to the 1× 1 matrix (0). Since the entry in it is ≥ 0,we terminate with the conclusion that the original matrix M is PSDbut not PD.

Example 5: Check whether the following matrix M is PSD.

M =

1 0 1 00 2 4 01 4 1 50 0 5 3

The matrix M is already symmetric, and its diagonal entries are all> 0. So, we apply the general step on it.

5.3 Algorithm to Check Positive ( Semi) D efiniteness 495

PC↓PR 1 0 1 0

0 2 4 01 4 1 50 0 5 31 0 1 00 2 4 00 4 0 50 0 5 3

Eliminating row 1 and column 1 in the matrix resulting after thefirst G step leads to the matrix

2 4 04 0 50 5 3

The 2nd diagonal entry in this matrix is 0, but its 2nd row and 2ndcolumn are not 0-vectors. So, we terminate with the conclusion thatthe original matrix M is not PSD.

Exercises

5.3.1: Find all the conditions that p, q > 0 have to satisfy for thefollowing matrix to be PD.

1−p2

2p1−pqp+q

1−pqp+q

1−q2

2q

.

5.3.2: Show that the following matrices are PD.

3 1 2 2−1 2 0 2

0 4 4 5/30 −2 −13/3 6

,

2 −11 3 49 2 −39 10

−1 41 5 12−2 −10 −12 2

,

496 Ch. 5. Quadratic Forms

2 −1 1−1 2 −1

1 −1 2

,

2 0 00 3 00 0 6

.

5.3.3: Show that the following matrices are not PD.

1 0 2 00 2 4 02 4 4 50 0 5 3

,

1 −1 −1 −1−1 1 −1 −1−1 −1 1 −1−1 −1 −1 1

,

10 −2 −1−2 0 1−1 1 5

,

2 0 30 2 13 1 −1

,

2 4 50 2 60 0 2

.

5.3.4: Show that the following matrices are PSD.

0 −2 −3 −4 52 3 3 0 03 3 3 0 04 0 0 8 4

−5 0 0 4 2

,

2 2 22 2 22 2 2

,

2 −2 1−2 4 2

1 2 6

.

5.3.3: Show that the following matrices are not PSD.

1 0 2 00 2 4 02 4 4 50 0 5 3

,

0 1 11 2 11 1 4

.

5.3.5: Let A be a square matrix of order n. Show that xT Ax is aconvex function in x iff its minimum value over Rn is 0.

5.3.6: Consider the square matrix of order n with all its diagonalentries = p, and all its off-diagonal entries = q. Determine all the valuesof p, q for which this matrix is PD.

5.4 Diagonalization 497

5.3.7: A square matrix A = (aij) of order n is said to be skew-symmetric if A + AT = 0. Show that the quadratic form xT Ax is 0 forall x ∈ Rn iff A is skew-symmetric.

Also, if A is a skew symmetric matrix of order n, and B is anymatrix of order n, show that xT (A + B)x = xT Bx for all x.

5.3.8: Let A be a square matrix of order n and f(x) = xT Ax. Forx, y ∈ Rn show that f(x+ y)− f(x)− f(y) is a bilinear function of x, yas defined in Section 1.7.

5.3.9: Find the range of values of α for which the quadratic formx2

1 + 4x1x2 + 6x1x3 + αx22 + αx2

3 is a strictly convex function.

5.3.10: Find the range of values of α for which the following matrixis PD.

1 4 20 5 − α 8 − 4α0 0 8 − α

.

5.4 Diagonalization of Quadratic Forms

and Square Matrices

Optimization deals with problems of the form: find a y = (y1, . . . , yn)T ∈Rn that minimizes a given real valued function g(y) of y. This problembecomes easier to solve if g(y) is separable, i.e., if it is the sum of nfunctions each involving one variable only, as in

g(y) = g1(y1) + . . . + gn(yn).

In this case, minimizing g(y) can be achieved by minimizing eachgj(yj) separately for j = 1 to n; and the problem of minimizing afunction is much easier if it involves only one variable.

Consider the problem of minimizing a quadratic function

f(x) = cx + (1/2)xT Mx

498 Ch. 5. Quadratic Forms

where M = (mij) is an n × n symmetric PD matrix. If some mij �= 0for i �= j, the function involves the product term xixj with a nonzerocoefficient, and it is not separable. To make this function separable,we need to convert the matrix M into a diagonal matrix. For this wecan try to apply a linear transformation of the variables x with the aimof achieving separability in the space of new variables. Consider thetransformation

x = Py

where P is an n × n nonsingular matrix, and the new variables arey = (y1, . . . , yn)T . Then, in terms of the new variables y, the functionis

F (y) = f(x = Py) = cPy + (1/2)yTP T MPy.

F (y) is separable if P TMP is a diagonal matrix. If P T MP =diag(d11, . . . , dnn), then let cP = c = (c1, . . . , cn). In this case F (y) =∑n

j=1(cjyj + djjy2j ). The point minimizing F (y) is y = (y1, . . . , yn)T

where yj is the minimizer of cjyj+djjy2j . Once y is found, the minimizer

of the original function f(x) is x = P y.The condition

P TMP =

d11 0 . . . 00 d22 . . . 0...

.... . .

...0 0 . . . dnn

requires

(P.i)TMP.j = 0 for all i �= j.

WhenM is a PD symmetric matrix, the set of column vectors{P.1, . . . , P.n} of a matrix P satisfying the above condition is said to bea conjugate set of vectors WRT M . Optimization methods calledconjugate direction methods or conjugate gradient methodsare based on using this type of transformations of variables.

Given a symmetric matrix M , this problem of finding a nonsingularmatrix P satisfying P T MP is a diagonal matrix is called the problem

5.4 Diagonalization 499

of diagonalizing a quadratic form. Conjugate direction methodsin nonlinear optimization have very efficient special routines for findingsuch a matrix P , but a discussion of these methods is outside the scopeof this book.

In mathematics they often discuss other matrix diagonalizationproblems, which we now explain.

The orthogonal diagonalization problem is the same as theabove, with the additional requirement that the matrix P must bean orthogonal matrix; i.e., it should also satisfy P T = P−1. Hencethe orthogonal diagonalization problem is: given a square symmetricmatrix M of order n, find an orthogonal matrix P of order n satisfying:

P−1MP is a diagonal matrix D (this is also written some-times as MP = PD).

The general matrix diagonalization problem discussed in math-ematics is: given a square matrix M of order n, find a nonsingularsquare matrix P of order n such that P−1MP is a diagonal matrix.

Solving either the orthogonal diagonalization problem, or the ma-trix diagonalization problem involves finding the eigenvalues and eigen-vectors of a square matrix, which are discussed in the next chapter.

Index

For each index entry we providethe section number where it isdefined or discussed first.Clickingon the light blue section numbertakes you to the page where thisentry is defined or discussed.

Completing the square 5.3Concave function 5.2Conjugacy 5.4Convex function 5.2

Diagonalization of quadraticforms 5.4

Indefinite 5.2

Jensen’s inequality 5.2

Matrix diagonalization 5.4

ND 5.2NSD 5.2

Orthogonal diagonalization 5.4

PD 5.2Principal submatrices 5.3PSD 5.2

Quadratic form 5.1Quadratic function 5.1

Strictly concave function 5.2Strictly convex function 5.2Symmetric coefficient matrix 5.1

500

Index for Ch. 5 501

List of algorithms

1. Algorithm to check if a given square matrix M of order n is PD

2. Algorithm to check if a given square matrix M of order n is PSD.

List of Results

1. Result 5.2.1: Conditions for a quadratic function to be (strictly)convex

2. Result 5.2.2: Conditions for a quadratic function to be (strictly)concave

3. Result 5.3.1: Properties shared by all principal submatrices

4. Result 5.3.2: Conditions for a 1 × 1 matrix

5. Result 5.3.3: Conditions satisfied by diagonal entries

6. Result 5.3.4: Conditions satisfied by the row and column of a0-diagonal entry in a PSD matrix

7. Result 5.3.5: Conditions satisfied by the row and column of a0-diagonal entry in a symmetric PSD matrix

502 Ch. 5. Quadratic Forms

8. Result 5.3.6: Conditions satisfied by a principal submatrix ob-tained after a pivot step.

Contents

6 Eigen Values, Eigen Vectors, and Matrix Diagonaliza-tion 5036.1 Definitions, Some Applications . . . . . . . . . . . . . . 5036.2 Relationship of Eigen Values and Eigen Vectors to the

Diagonalizability of a Matrix . . . . . . . . . . . . . . . 5156.3 An Illustrative Application of Eigen Values, Eigen Vec-

tors, and Diagonalization in Differential Equations . . . 5236.4 How to Compute Eigen Values ? . . . . . . . . . . . . . 5266.5 References . . . . . . . . . . . . . . . . . . . . . . . . . 527

i

ii

Chapter 6

Eigen Values, Eigen Vectors,and Matrix Diagonalization

This is Chapter 6 of “Sophomore Level Self-Teaching Web-book for Computational and Algorithmic Linear Algebra andn-Dimensional Geometry” by Katta G. Murty.

6.1 Definitions, Some Applications

Eigen values and eigen vectors are only defined for square matrices;hence in this chapter we will consider square matrices only.

Let A = (aij) be a square matrix of order n. Let I denote theunit matrix of order n. A scalar λ is said to be an eigen value (orcharacteristic value, or proper value, or latent value) of squarematrix A if the matrix A − λI is singular, i.e., if the determinantdet(A − λI) = 0.

Let P (λ) = det(A − λI). Expansion of det(A − λI) produces thepolynomial in λ, P (λ) which is known as the characteristic polyno-mial for the square matrix A. The degree of P (λ) is n = order of thematrix A, and the leading term in it is (−1)nλn.

The eigen values of A are the solutions of the characteristic equa-tion for A

P (λ) = 0

503

504 Ch. 6. Eigen Values

i.e., they are the roots of the characteristic polynomial for A.The set of all eigen values of A, usually denoted by the symbol

σ(A), is called the spectrum of A.Since the characteristic polynomial is of degree n, it has exactly

n roots by the Fundamental Theorem of Algebra. But some of theseroots may be complex numbers (i.e., a number of the form a + ib,where a, b are real numbers and i =

√−1), and some of the roots maybe repeated.

Since all the entries aij in the matrix A are real numbers, all thecoefficients in the characteristic polynomial are real numbers. By thefundamental theorem of algebra this implies that if λ = a + ib is aroot of the characteristic polynomial, then so is its complex conjugateλ = a − ib. Thus the complex eigen values of A occur in complexconjugate pairs.

So, if λ1 is an eigen value of A, then λ = λ1 is a solution of P (λ) = 0.From theory of equations, this implies that (λ1 − λ) is a factor of thecharacteristic polynomial P (λ). In the same way we conclude that ifλ1, λ2, . . . , λn are the various eigen values of A, then the characteristicpolynomial

P (λ) = (λ1 − λ)(λ2 − λ) . . . (λn − λ).

However, as mentioned earlier, some of the eigen values of A maybe complex numbers, and some of the eigen values may be equal toeach other. If λ1 is an eigen value of A and r ≥ 1 is the positive integersatisfying the property that (λ1 − λ)r is a factor of the characteristicpolynomial P (λ), but (λ1 − λ)r+1 is not, then we say that r is thealgebraic multiplicity of the eigen value λ1 of A. If r = 1, λ1 iscalled a simple eigen value of A; if r > 1, λ1 is called a multipleeigen value of A with algebraic multiplicity r.

Thus if λ1, λ2, . . . , λk are all the distinct eigen values of A includ-ing any complex eigen values, with algebraic multiplicities r1, . . . , rk

respectively, then the characteristic polynomial

P (λ) = (λ1 − λ)r1(λ2 − λ)r2 . . . (λk − λ)rk

and r1 + . . . + rk = n. In this case the spectrum of A, σ(A) ={λ1, . . . , λk}.

6.1 Definitions 505

If λ is an eigen value of A, since A−λI is singular, the homogeneoussystem of equations

(A − λI)x = 0 or equivalently Ax = λx

must have a nonzero solution x ∈ Rn. A column vector x �= 0 satisfyingthis equation is called an eigen vector of A corresponding to its eigenvalue λ; and the pair (λ, x) is called an eigen pair for A. Notice thatthe above homogeneous system may have many solutions x, the set ofall solutions of it is the nullspace of the matrix (A − λI) defined inSection 3.10 and denoted by N(A − λI). This nullspace N(A − λI)is called the eigen space of A corresponding to its eigen value λ.Therefore, the set of all eigen vectors of A associated with its eigenvalue λ is {x ∈ Rn : x �= 0, and x ∈ N(A− λI)}, the set of all nonzerovectors in the corresponding eigen space.

The dimension of N(A − λI) (as defined in Section 1.23, this isequal to the number of nonbasic variables in the final tableau when theabove homogeneous system (A− λI)x = 0 is solved by the GJ methodor the G elimination method) is called the geometric multiplicity ofthe eigen value λ of A. We state the following result on multiplicitieswithout proof (for proofs see C. Meyer [6.1]). called the geometricmultiplicity of the eigen value λ of A.

Result 6.1.1: Multiplicity Inequality: For simple eigen valuesλ of A (i.e., those with algebraic multiplicity of 1) the dimension ofthe eigen space N(A − λI) is 1; i.e., this eigen space is a straight linethrough the origin.

If λ is an eigen value of A with algebraic multiplicity > 1, we willalways have

geometric multiplicity of λ ≤ algebriac multiplicity of λ

i.e., the dimension of N(A − λI) ≤ algebriac multiplicity of λ.

Result 6.1.2: Linear Independence of Eigen vectors Asso-ciated With Distinct Eigen values: Let λ1, . . . , λh be some distincteigen values of A; and {(λ1, x

1), (λ2, x2), . . . , (λh, x

h)} a set of eigen

506 Ch. 6. Eigen Values

pairs for A. Then the set of eigen vectors {x1, . . . , xh} is linearly in-dependent.

Suppose all the eigen values of A are real, and σ(A) = {λ1, . . . , λk}is this set of distinct eigen values. If, for all i = 1 to k, geometricmultiplicity of λi is equal to its algebraic multiplicity; then it is possibleto get a complete set of n eigen vectors for A which is linearlyindependent.

If geometric multiplicity of λi is < its algebriac multiplicity for atleast one i, then A does not have a complete set of eigen vectors which islinearly independent. Such matrices are called deficient or defectivematrices.

Example 1: Let

A =

(6 −54 −3

). Then A − λI =

(6 − λ −5

4 −3 − λ

).

Therefore the characteristic polynomial of A, P (λ) = determinant(A−λI) = (6−λ)(−3−λ)+20 = λ2 −3λ+2 = (2−λ)(1−λ). Hencethe characteristic equation for A is

(2 − λ)(1 − λ) = 0

for which the roots are 2 and 1. So, the eigen values of A are 2 and 1respectively, both are simple with algebraic multiplicity of one.

To get the eigen space of an eigen value λi we need to solve thehomogeneous system (A − λiI)x = 0. For λ = 1 this system in de-tached coefficient form is the one in the top tableau given below. Wesolve it using the GJ method discussed in Section 1.23. Pivot elementsare enclosed in a box; PR, PC denote the pivot row, pivot column re-spectively, and BV denotes the basic variable selected in the row. RCdenotes redundant constraint to be eliminated.

6.1 Definitions 507

BV x1 x2

5 −5 0 PR4 −4 0

PC↑x1 1 −1 0

0 0 0 RCFinal Tableau

x1 1 −1 0

Since there is one nonbasic variable in the final tableau, the dimen-sion of the eigen space is one. The general solution of the system isx = (α, α)T for any α real. So, for any α �= 0 (α, α)T is an eigen vectorcorresponding to the eigen value 1. Taking α = 1 leads to one eigenvector x1 = (1, 1)T .

For the eigen value λ = 2, the eigen space is determined by thefollowing system, which we also solve by the GJ method.

BV x1 x2

4 −5 0 PR4 −5 0

PC↑x1 1 −5/4 0

0 0 0 RCFinal Tableau

x1 1 −5/4 0

So, the general solution of this system is x = (5α/4, α)T whichis an eigen vector corresponding to the eigen value 2 for any α �= 0.In particular, taking α = 1, yields x2 = (5/4, 1)T as an eigen vectorcorresponding to 2.

So, {x1, x2} = {(1, 1)T , (5/4, 1)T} is a complete set of eigen vectorsfor A.

Example 2: Let

A =

(0 −11 0

). Then A − λI =

( −λ −11 −λ

).

508 Ch. 6. Eigen Values

Therefore the characteristic polynomial of A, P (λ) = determinant(A − λI) = λ2 + 1. Hence the characteristic equation for A is

λ2 + 1 = 0

for which the solutions are the imaginary numbers λ1 = i =√−1 and

λ2 = −i. Thus the eigen values of this matrix A are the complexconjugate pair +i and −i.

It can also be verified that the eigen vectors corresponding to

λ1 = +i are (α,−αi)T

λ2 = −i are (α, αi)T

for any nonzero complex number α. Thus the eigen vectors of A areboth complex vectors in this example, even though the matrix A isreal.

Example 3: Let

A =

0 0 −110 1 20 0 1

. Then A − λI =

λ 0 −110 1 − λ 20 0 1 − λ

.

Therefore the characteristic polynomial of A, P (λ) = determinant(A − λI) = −λ(1 − λ)2. Hence the characteristic equation for A is

−λ(1 − λ)2 = 0

for which the roots are 0 and 1, with 1 being a double root. So, theeigen values of A are 0 and 1 , 0 is simple with algebraic multiplicityone, and 1 is multiple with algebraic multiplicity 2.

To get the eigen space of eigen value 0 we need to solve the homo-geneous system Ax = 0. We solve it using the GJ method discussedin Section 1.23. Pivot elements are enclosed in a box; PR, PC denotethe pivot row, pivot column respectively, and BV denotes the basicvariable selected in the row. RC denotes redundant constraint to beeliminated.

6.1 Definitions 509

BV x1 x2 x3

0 0 −1 0 PR10 1 2 00 0 1 0

PC↑x3 0 0 1 0

10 1 0 0 PR0 0 0 0 RC

PC↑Final Tableau

x3 0 0 1 0x2 10 1 0 0

The general solution of this system is x = (α,−10α, 0)T for any αreal. Taking α = 1, an eigen vector corresponding to the eigen value 0is x1 = (1,−10, 0)T .

For λ = 1, the eigen space is determined by the following system,which we also solve using the GJ method of Section 1.23.

BV x1 x2 x3

−1 0 −1 0 PR10 0 2 00 0 0 0 RC

PC↑x1 1 0 1 0

0 0 −8 0 PRPC↑

Final Tableaux1 1 0 0 0x3 0 0 1 0

The general solution of this system is x = (0, α, 0)T for any α real;so an eigen vector corresponding to the eigen value 1 is (0, 1, 0)T . Theeigen space corresponding to this eigenvalue is a straight line, a onedimensional object.

The algebraic multiplicity of the eigen value 1 is 2, but its geometric

510 Ch. 6. Eigen Values

multiplicity, the dimension of the eigen space is only 1.

This example illustrates the fact that the geometric multiplicity ofan eigen value, the dimension of its eigen space, can be strictly lessthan its algebraic multiplicity.

Example 4: Let

A =

0 1 11 0 11 1 0

. Then A − λI =

−λ 1 11 −λ 11 1 −λ

.

Therefore the characteristic polynomial of A, P (λ) = determinant(A− λI) = (λ + 1)2(2− λ). Hence the characteristic equation for A is

(λ + 1)2(2 − λ) = 0

for which the roots are −1 and 2, with −1 being a double root. So, theeigen values of A are −1 and 2; 2 is simple with algebraic multiplicityone, and −1 is multiple with algebraic multiplicity 2.

The eigen space corresponding to eigen value −1 is determined bythe following homogeneous system which we solve using the GJ methoddiscussed in Section 1.23. Pivot elements are enclosed in a box; PR, PCdenote the pivot row, pivot column respectively, and BV denotes thebasic variable selected in the row. RC denotes redundant constraint tobe eliminated.

BV x1 x2 x3

1 1 1 0 PR1 1 1 01 1 1 0

PC↑x3 1 1 1 0

0 0 0 0 RC0 0 0 0 RC

Final Tableaux3 1 1 1 0

6.1 Definitions 511

The general solution of this system is x = (α, β,−(α+β))T for anyα, β real. So, this eigen space has dimension 2, it is the subspace whichis the linear hull of the two vectors (1, 0,−1)T and (0, 1,−1)T . The setof these two vectors is a basis for this eigen space.

Thus for the matrix A in this example, both the algebraic andgeometric multiplicities of its eigen value −1 are two.

The eigen vector corresponding to the eigen value 2 is obtained fromthe following ststem.

BV x1 x2 x3

−2 1 1 0 PR1 −2 1 01 1 −2 0

PC↑x3 −2 1 1 0

3 −3 0 0 PR−3 3 0 0

PC↑x3 0 −1 1 0x1 1 −1 0 0 PR

0 0 0 0 RCFinal Tableau

x3 0 −1 1 0x1 1 −1 0 0

The general solution of this system is x = (α, α, α)T for any α real;so an eigen vector corresponding to the eigen value 2 is (1, 1, 1)T .

Example 5: Eigen Values of a Diagonal Matrix:Let

A =

2 0 00 −7 00 0 9

. Then A − λI =

2 − λ 0 00 −7 − λ 00 0 9 − λ

.

Therefore the characteristic polynomial of A, P (λ) = determinant(A − λI) = (2 − λ)(−7 − λ)(9 − λ). Hence the characteristic equation

512 Ch. 6. Eigen Values

for A is

(2 − λ)(−7 − λ)(9 − λ) = 0

for which the roots are 2, −7, and 9. So, the eigen values of A are 2,−7 and 9; the diagonal entries in the diagonal matrix A.

The eigen space corresponding to eigen value 2 is the solution setof

x1 x2 x3

0 0 0 00 −9 0 00 0 7 0

The general solution of this system is x = (α, 0, 0)T for real α.Hence an eigen vector corresponding to the eigen value 2 is (1, 0, 0)T ,the first unit vector.

In the same way it can be verified that for any diagonal matrix, itseigen values are its diagonal entries; and an eigen vector correspondingto its ith diagonal entry is the ith unit vector.

Example 6 : Eigen Va lues of an Upp er TriangularMatrix: Let

A =

-1 -10 160 2 -180 0 -3

. Then A−λI =

−1 − λ −10 160 2 − λ -180 0 −3 − λ

.

Therefore the characteristic polynomial of A, P (λ) = determinant(A−λI) = (−1−λ)(2−λ)(−3−λ). Hence the characteristic equationfor A is

(−1 − λ)(2 − λ)(−3 − λ) = 0

Its roots, the eigen values of the upper triangular matrix A, areexactly its diagonal entries.

The eigen space corresponding to eigen value −1 is the set of solu-tions of

6.1 Definitions 513

x1 x2 x3

0 −10 16 00 3 −18 00 0 −2 0

The general solution of this system is x = (α, 0, 0)T for α real.Hence an eigen vector corresponding to −1, the first diagonal entry ofA, is (1, 0, 0)T , the first unit vector.

In the same way it can be verified that an eigen vector correspondingto the ith diagonal entry of A is the ith unit vector, for all i.

Similarly, for any upper or lower triangular matrix, its eigen valuesare its diagonal entries, and its eigen vectors are the unit vectors.

Basis for the Eigen Space of a Square Matrix ACorresponding to an Eigen Value λ

Let λ be an eigen value of a square matrix A of order n. Then theeigen space corresponding to λ is N(A− λI), the set of all solutions ofthe system

(A − λI)x = 0

A basis for this eigen space, as defined in Sections 1.23, 4.2, is amaximal linearly independent set of vectors in N(A − λI), it has theproperty that every point in this eigen space can be expressed as itslinear combination in a unique manner.

Example 7: Consider the matrix

A =

0 1 11 0 11 1 0

from Example 4, and its eigen value −1. The eigen space correspondingto this eigen value is the set of all solutions of

514 Ch. 6. Eigen Values

x1 x2 x3

1 1 1 01 1 1 01 1 1 0

All the constraints in this system are the same as x1 + x2 + x3 = 0.So, a general solution of this system is (α, β,−α − β)T , where α, βare real valued parameters. A basis for this eigen space is the set{(1, 0,−1)T , (0, 1,−1)T}, the set of two vectors obtained by settingα = 1, β = 0, and α = 0, β = 1 respectively in the formula for thegeneral solution.

Results on Eigen Values and Eigen Vectors

We now discuss some important results on eigen values and eigenvectors, without proofs.

Result 6.1.3: Linear independence of the set of eigen vec-tors corresponding to distinct eigen values: Let (λ1, x

1), . . . , (λk, xk)

be eigen pairs for a square matrix A, where the eigen values λ1, . . . , λk

are distinct. Then {x1, . . . , xk} is a linearly independent set.

Result 6.1.4: The union of bases of eigen spaces corre-sponding to distinct eigen values is a basis: Let {λ1, . . . , λk} bea set of distinct eigen values for a square matrix A. For i = 1 to k, letBi be a basis for the eigen space corresponding to λi. Then B = ∪k

i=1Bi

is a linearly independent set.

A Complete Set of Eigen Vectors

A complete set of eigen vectors for a square matrix A of ordern is any linearly independent set of n eigen vectors for A. Not allsquare matrices have complete sets of eigen vectors. Square matricesthat do not have a complete set of eigen vectors are called deficientor defective matrices.

Let A be a square matrix of order n which has distinct real eigen

6.2 Matrix Diagonalization 515

values λ1, . . . , λn. Let xi be an eigen vector corresponding to λi for i= 1 to n. Then by Result 6.1.3, {x1, . . . , xn} is a complete set of eigenvectors for A. Thus any square matrix with all eigen values real anddistinct has a complete set of eigen vectors.

Let A be a square matrix of order n with distinct real eigen valuesλ1, . . . , λk, where for i = 1 to k, both the algebraic and geometricmultiplicities of λi are equal to ri, satisfying r1 + . . . + rk = n. Thenthe eigen space corresponding to λi has a basis Bi consisting of ri

vectors, and B = ∪ki=1Bi is linearly independent by Result 6.1.4, and is

a basis for Rn. In this case B is a complete set of eigen vectors for A.Thus if all eigen values of A are real, and for each of them its geometricmultiplicity is its algebraic multiplicity, then A has a complete set ofeigen values.

Thus if a square matrix A has an eigen value for which the geometricmultiplicity is < its algebraic multiplicity, then A is deficient in eigenvectors; i.e., it does not have a complete set of eigen vectors.

6.2 Relationship of Eigen Values and Eigen

Vectors to the Diagonalizability of a

Matrix

Let A be a square matrix of order n. In Section 5.4 we mentioned thatthe matrix diagonalization problem for matrix A is to find a squarenonsingular matrix P of order n such that P−1AP is a diagonal matrixD = diag(d1, . . . , dn), say. In this case we say that the matrix Pdiagonalizes A, i.e.,

P−1AP =

d1 0 . . . 00 d2 . . . 0...

......

0 0 . . . dn

= D = diag(d1, . . . , dn)

Then multiplying the above equation on both sides by P , we have

AP = PD.

516 Ch. 6. Eigen Values

P.1, . . . , P.n are the column vectors of P . Since D = diag(d1, . . . , dn),

we have PD = (d1P.1... . . .

...dnP.n), and by the definition of matrix mul-

tiplication AP = (AP.1... . . .

...AP.n). So, the above equation AP = PDimplies

(AP.1... . . .

...AP.n) = (d1P.1... . . .

...dnP.n)

i.e., AP.j = djP.j j = 1, . . . , n.

Since P is invertible, P.j �= 0 for all j. So, the above equationAP.j = djP.j, i.e., (A−djI)P.j = 0 implies that (dj , P.j) are eigen pairsfor A for j = 1 to n.

Thus if the matrix P diagonalizes the square matrix A into diag(d1,. . . , dn) then d1, . . . , dn are the eigen values of A; and P.j is an eigenvector corresponding to dj for each j = 1 to n.

Conversely, suppose a square matrix A of order n has a complete setof eigen vectors {P.1, . . . , P.n} with corresponding eigen values λ1, . . . , λn.Let D = diag(λ1, . . . , λn). Since (λj, P.j) is an eigen pair for A for j =1 to n; we have AP.j = λjP.j for j = 1 to n, i.e.

(AP.1... . . .

...AP.n) = (λ1P.1... . . .

...AP.n) = DP

i.e., AP = DP where P is the square matrix with the eigen vectorsP.1, . . . , P.n as the column vectors. Since {P.1, . . . , P.n} is a completeset of eigen vectors, it is linearly independent, so P is invertible. So,AP = PD implies P−1AP = D. Thus P diagonalizes A.

Thus the square matrix A of order n is diagonalizable iff it has acomplete set of eigen vectors; and a matrix which diagonalizes A isthe matrix whose column vectors are a basis for Rn consisting of eigenvectors of A.

Thus the task of diagonalizing a square matrix A is equivalent tofinding its eigen values and a complete set of eigen vectors for it.

We now provide some examples. In this chapter we number allthe examples serially. So, continuing the numbering from the previoussection, we number the next example as Example 8.

Example 8: Consider the matrix

6.2 Matrix Diagonalization 517

A =

0 1 11 0 11 1 0

discussed in Examples 4, 7. From Examples 4, 7, we know that itseigen values are−1 and 2, and that {(1, 0,−1)T , (0, 1,−1)T , (1, 1, 1)T}is a complete set of eigen vectors for A. So

P =

1 0 10 1 1

−1 −1 1

is a matrix that diagonalizes A. In fact P−1AP = diag(−1,−1, 2).

Example 9: Let

A =

0 0 −110 1 20 0 1

In Example 3 we determined that its eigen values are 0 and 1, with1 having algebraic multiplicity 2. But the geometric multiplicity of 1 is1 < its algebraic multiplicity. So, this matrix does not have a completeset of eigen vectors, and hence it is not diagonalizable.

Example 10: Let

A =

(6 −54 −3

)

the matrix considered in Example 1. From Example 1 we know thatits eigen values are 2, 1 and a complete set of eigen vectors for it is{(1, 4/5)T , (1, 1)T}. So, the matrix

P =

(1 1

4/5 1

)

518 Ch. 6. Eigen Values

diagonalizes A, and P−1AP = diag(2, 1).

We will now present some more important results, most of themwithout proofs.

Result 6.2.1: Results on Eigen Values and Eigen Vectorsfor Symmetric Matrices:

(i): All eigen values are real: If A is a square symmetric matrixof order n, all its eigen values are real numbers, i.e., it has no complexeigen values.

(ii): Eigen vectors corresponding to distinct eigen valuesare orthogonal: If A is a square symmetric matrix of order n, eigenvectors corresponding to distinct eigen values of A are orthogonal.

We will provide a proof of this result because it is so simple. SinceA is symmetric, A = AT . Let λ1 �= λ2 be eigen values associated witheigen vectors v1, v2 respectively. So, Av1 = λ1v

1 and Av2 = λ2v2. So

λ1(v1)T v2 = (λ1v

1)T v2 = (Av1)T v2 = (v1)T AT v2 = (v1)T Av2 = λ2(v1)T v2

Hence, λ1(v1)T v2 = λ2(v

1)T v2, i.e., (λ1 − λ2)(v1)T v2 = 0. Since

λ1 �= λ2, λ1 − λ2 �= 0. Hence (v1)T v2 must be zero; i.e., v1 and v2 areorthogonal.

(iii): Special eigen properties of symmetric PD, PSD, ND,NSD, Indefinite matrices: A square symmetric matrix is

Positive definite iff all its eigen values are > 0Positive semidefinite iff all its eigen values are ≥ 0Negative definite iff all its eigen values are < 0Negative semidefinite iff all its eigen values are ≤ 0Indefinite iff it has both a positive and a negative eigen value.

See Section 5.2 for definitions of these various classes of matrices.

6.2 Matrix Diagonalization 519

(iv): Geometric and algebraic multiplicities are equal: IfA is a square symmetric matrix of order n, for every one of its eigenvalues, its geometric multiplicity is equal to its algebraic multiplicity.Hence symmetric matrices can always be diagonalized.

(v): Orthonormal set of eigen vectors: If A is a square ma-trix of order n, it has an orthonormal set of n eigen vectors iff it issymmetric.

Orthogonal Diagonalization of a Square Matrix

As discussed in Section 5.4, orthogonal diagonalization of asquare matrix A deals with the problem of finding an orthogonal matrixP such that

P−1AP = a diagonal matrix diag(λ1, . . . , λn) say.

When P is orthogonal P−1 = P T . The orthogonal matrix P is saidto orthogonally diagonalize the square matrix A if P−1AP = P TAPis a diagonal matrix.

Result 6.2.2: Orthogonally diagonalizable iff symmetric: Asquare matrix can be orthogonally diagonolized iff it is symmetric.

To see this, suppose the orthogonal matrix P orthogonally diago-nalizes the square matrix A. So, P−1 = P T and P−1AP = a diagonalmatrix D. Since D is diagonal, DT = D. Now, P−1AP = D impliesA = PDP−1 = PDP T . So,

AT = (PDP T )T = PDTP T = PDP T = A

i.e., A is symmetric.To orthogonally diagonalize a square symmetric matrix A of order n,

one needs to find its eigen values. If they are all distinct, let (λi, P.i) bethe various eigen pairs where each P.i is normalized to satisfy ||P.i|| = 1.Then by Result 6.2.1 (ii), {P.1, . . . , P.n} is an orthonormal set of eigenvectors for A. Let P be the matrix with P.1, . . . , P.n as its columnvectors, it orthogonally diagonalizes A.

520 Ch. 6. Eigen Values

If the eigen values of the symmetric matrix A are not all distinct,find a basis for the eigenspace of each eigen value. Apply the Gram-Schmidt process to each of these bases to obtain an orthonormal basisfor each eigen space. Then form the square matrix P whose columns arethe vectors in these various orthonormal bases. Then P orthogonallydiagonalizes A.

Result 6.2.3: Similar Matrices Have the Same Eigen Spec-trum: Two square matrices A, B of order n are said to be similar ifthere exists a nonsingular matrix P of order n satisfying B = P−1AP .In this case we say that B is obtained by a similarity transforma-tion on A. Since B = P−1AP implies A = PBP−1, if B can be ob-tained by a similarity transformation on A, then A can be obtained bya similarity transformation on B. When B = P−1AP , det(B − λI) =det(P−1AP−λI) = det(P−1(AP−λP )) = det(P−1)det(A−λI)det(P )=det(A − λI). Thus similar matrices all have the same charateristicpolynomial, so they all have the same eigen values with the same mul-tiplicities. However, their eigen vectors may be different.

Result 6.2.4: Eigen values of powers of a matrix: Supposethe square matrix A is diagonalizable, and let P−1AP = diag(λ1, . . . , λn).Then for any positive integer s,

As = P−1(diag(λs1, . . . , λ

sn))P

Since λ1, . . . , λn are real numbers, computing λs1, . . . , λ

sn is much

easier than computing As because of the complexity of matrix multi-plication.

So, calculating higher powers As of matrix A for large values of scan be carried out efficiently if it can be diagonalized.

Result 6.2.5: Spectral Theorem for Diagonalizable Ma-trices: Let A be a square matrix of order n with spectrum σ(A) ={λ1, . . . , λk}. Suppose A is diagonalizable, and for i = 1 to k, let ri =algebraic multiplicity of λi = geometric multiplicity of λi. Then thereexist square matrices of order n, G1, . . . , Gk satisfying

6.2 Matrix Diagonalization 521

i) GiGj = 0 for all i �= jii) G1 + . . . + Gk = Iiii) λ1G1 + . . . + λkGk = Aiv) G2

i = Gi for all iv) Rank(Gi) = ri for all i.

Let Xi be the matrix of order n × ri whose column vectors form a

basis for the eigen space N(A − λiI). Let P = (X1...X2

... . . ....Xk). Then

P is nonsingular by Result 6.1.2. Partition P−1 as follows

P−1 =

Y1

. . .Y2

. . ....

. . .Yk

where Yi is an ri × n matrix. Then Gi = XiYi for i = 1 to k satisfiesall the above properties. All these properties follow from the followingfacts

PP−1 = P−1P = I

A = PDP−1 = (X1...X2

... . . ....Xk)

λ1Ir1 0 . . . 00 λ2Ir2 . . . 0...

......

0 0 . . . λkIrk

Y1

. . .Y2

. . ....

. . .Yk

where Iriis the unit matrix of order ri for i = 1 to k.

The decomposition of A into λ1G1 + . . . + λkGk is known as thespectral decomposition of A, and the Gis are called the spectralprojectors associated with A.

522 Ch. 6. Eigen Values

Example 11: Consider the matrix

A =

0 1 11 0 11 1 0

discussed in Example 8. Its eigen values are λ1 = −1 with an algebraicand geometric multiplicity of two, and λ2 = 2 with an algebraic andgeometric multiplicity of one. A basis for the eigen space correpondingto −1 is {(1, 0,−1)T , (0, 1,−1)T}; and an eigen vector correspondingto 2 is (1, 1, 1)T . So, using the notation in Result 6.2.5, we have

X1 =

1 00 1

−1 −1

, X2 =

111

, P =

1 0... 1

0 1... 1

−1 −1... 1

.

We compute P−1 and obtain

P−1 =

2/3 −1/3 −1/3−1/3 2/3 −1/3

. . . . . . . . .1/3 1/3 1/3

,

Y1 =

(2/3 −1/3 −1/3

−1/3 2/3 −1/3

),

Y2 =(

1/3 1/3 1/3).

So

G1 = X1Y1 =

2/3 −1/3 −1/3−1/3 2/3 −1/3−1/3 −1/3 2/3

G2 = X2Y2 =

1/3 1/3 1/31/3 1/3 1/31/3 1/3 1/3

.

It can be verified that rank(G1) = 2 = algebraic multiplicity of theeigen value −1, and rank(G2) = 1 = algebraic multiplicity of the eigenvalue 2. Also, we verify that

G1 + G2 = I = unit matrix of order 3

2G1 + G2 = A, the spectral decomposition of A.

6.3 An Application 523

G1, G2 are the spectral projectors associated with A.

Result 6.2.6: Eigen pairs for A and A− 1 If (λ, x) is an eigenpair for a nonsingular square matrix A, then (1/λ, x) is an eigen pairfor A−1.

By hypothesis we have Ax = λx, and since A is nonsingular, wehave λ �= 0. Multiplying both sides of the equation Ax = λx on theleft by (1/λ)A−1 we get (A−1 − (1/λ)I)x = 0. Therefore ((1/λ), x) isan eigen pair for A−1.

Result 6.2.7: Sylvester’s Law of Inertia: Let A be a squaresymmetric matrix of order n. Let C be any nonsingular square matrixof order n, and B = CT AC. Then we say that B is obtained fromA by a congruent transformation, and that A, B are congruent.Since C is nonsingular, B = CTAC implies A = (CT )−1BC−1 =(C−1)T B(C−1), so if B can be obtained by a congruent transformationon A, then A can be obtained by a congruent transformation on B.

The inertia of a real symmetric matrix is defined to be the triple(ρ, ν, ζ) where

ρ, ν, ζ are the numbers of positive, negative, and zero eigenvalues of the matrix respectively.

Sylvester’s law states that two symmetric matrices are congruent iffthey have the same inertia.

6.3 An Illustrative Application of Eigen

Values, Eigen Vectors, and Diagonal-

ization in Differential Equations

Eigen values, eigen vectors, and diagonalization of square matrices findmany applications in all branches of science. Here we provide one il-lustrative application in differential equations, which are equations in-volving functions and their derivatives. Mathematical models involving

524 Ch. 6. Eigen Values

differential equations appear very often in studies in chemical engi-neering and other branches of engineering, physics, chemistry, biology,pharmacy, and other sciences.

Let u1(t), . . . , un(t) be n real valued differentiable functions of a sin-gle real variable t; and let u′

i(t) denote the derivative of ui(t). Supposethe formulas for these functions are unknown, but we know that thesefunctions satisfy the following equations.

u′1(t) = a11u1(t) + . . . + a1nun(t)

......

u′n(t) = an1u1(t) + . . . + annun(t)

and ui(0) = ci i = 1, . . . , n

where A = (aij) is a given square matrix of order n, and c = (c1, . . . , cn)T

is a given vector of function values corresponding to t = 0.A system like this is known as a homogeneous system of first

order linear differential equations with constant coefficients.Solving this system means to find the formulas for the functions u1(t),. . . , un(t) satisfying these equations. Let

A =

a11 . . . a1n...

...an1 . . . ann

, u(t) =

u1(t)...

un(t)

, u′(t) =

u′1(t)...

u′n(t)

.

Then the system can be written in matrix form as

u′(t) = Au(t), u(0) = c.

For simplicity, we only consider the case where the matrix A isdiagonalizable. Let the spectrum of A be σ(A) = {λ1, . . . , λk}, and letthe spectral projector associated with λi be Gi for i = 1 to k. Then itcan be shown that the unique solution of this system is

u(t) = eλ1tv1 + . . . + eλktvk

6.3 An Application 525

where vi = Gic for i = 1 to k.

Example 12: Consider the following linear differential equa-tion involving three functions u1(t), u2(t), u3(t) and their derivativesu′

1(t), u′2(t), u

′3(t).

u′1(t) = u2(t) +u3(t)

u′2(t) = u1(t) +u3(t)

u′3(t) = u1(t) +u2(t)

(u1(0), u2(0), u3(0))T = (12,−15, 18)T

Letting u(t) = (u1(t), u2(t), u3(t))T , u′(t) = (u′

1(t), u′2(t), u

′3(t))

T ,c = (12,−15, 18)T , the system in matrix notation is

u′(t) = Au(t)

u(0) = c

where

A =

0 1 11 0 11 1 0

the matrix discussed in Example 8, 11. Its eigen values are λ1 = −1and λ2 = 2, and the associated spectral projectors are

G1 =

2/3 −1/3 −1/3−1/3 2/3 −1/3−1/3 −1/3 2/3

, G2 =

1/3 1/3 1/31/3 1/3 1/31/3 1/3 1/3

from Example 11. So, in the notation given above, we have

v1 = G1c =

2/3 −1/3 −1/3−1/3 2/3 −1/3−1/3 −1/3 2/3

12−15

18

=

7−20

13

v2 = G2c =

1/3 1/3 1/31/3 1/3 1/31/3 1/3 1/3

12−15

18

=

555

.

526 Ch. 6. Eigen Values

So, from the above, we know that the unique solution of this systemis

u(t) =

u1(t)u2(t)u3(t)

= e−t

7−20

13

+ e2t

555

=

7e−t + 5e2t

−20e−t + 5e2t

13e−t + 5e2t

.

i.e., u1(t) = 7e−t + 5e2t, u2(t) = −20e−t + 5e2t, u3(t) = 13e−t + 5e2t.It can be verified that these functions do satisfy the given differentialequations.

6.4 How to Compute Eigen Values ?

Computing the eigen values of a matrix A of order n using the maindefinition boils down to finding the roots of a polynomial equation ofdegree n. Finding the roots of a polynomial equation of degree ≥ 5 isnot a simple task, it has to be carried out using iterative methods.Themost practical eigen value computation method is the QR iterationalgorithm, an iterative method that computes the QR factors (see Sec-tion 4.12) of a matrix in every iteration. Providing a mathematicaldescription of this algorithm, and the details of the work needed toconvert it into a practical implementation, are beyond the scope of thisbook. For details see the references given below.

Exercises

6.4.1: Obtain the characteristic polynomial of the following ma-trices. Using it obtain their eigen values, and then the correspondingeigen vectors and eigen spaces.

(3 28 3

),

(3 1/3

31/3 5

),

5 1 26 5 36 2 6

,

3 1 22 4 43 3 8

.

6.4.2: Compute eigen values and eigen vectors of the followingmatrices.

6.4 Computation 527

1 0 1 00 1 0 11 0 1 00 1 0 1

,

0 0 1 00 0 0 11 0 0 00 1 0 0

,

2 −1 10 3 −12 1 3

.

6.4.3: Let A be a square matrix of order n in which all diagonalentries are = α, and all off-diagonal entries are = β. Show that it hasan eigen value α−β with algebraic multiplicity n−1, and α+(n−1)βas the only other eigen value.

6.4.4: If λ is an eigen value of A associated with the eigen vectorx, show that λk is an eigen value of Ak associated with the same eigenvector x for all positive integers k.

6.4.5: If A, B are square matrices of order n, show that the eigenvalues of AB, BA are the same.

6.5 References

[6.1] C. P. Meyer, Matrix Analysis and Applied Linear Algebra, SIAM,Philadelphia, PA, 2000.[6.2] J. H. Wilkinson, The Algebraic Eigenvalue Problem, Oxford Uni-versity Press, 1965.

Index

For each index entry we providethe section number where it isdefined or discussed first. Click-ing on the light blue section num-ber takes you to the page wherethis entry is defined or discussed.

Algebraic multiplicity 6.1

Characteristic equation 6.1Characteristic polynomial 6.1Characteristic value 6.1Congruent transformation 6.2

Deficient matrices 6.1

Eigen pair 6.1Eigen space 6.1Eigen value 6.1Eigen vector 6.1

Geometric multiplicity 6.1

Latent value 6.1Linear differential equation 6.3

Matrix diagonalization 6.2Multiple eigen value 6.1Multiplicity inequality 6.1

Orthogonal diagonalization 6.2

Proper value 6.1

Similar matrices 6.2Simple eigen value 6.1Spectral decomposition 6.2Spectral projectors 6.2Spectral theorem 6.2Spectrum 6.1Sylvester’s law 6.2Symmetric matrices 6.2

528

Index for Ch. 6 529

List of Results

1. Result 6.1.1: Multiplicity inequality

2. Result 6.1.2: Linear independence of eigen values associated withdistinct eigen values

3. Result 6.1.3: Linear independence of the set of eigen vectorscorresponding to distinct eigen values

4. Result 6.1.4: The union of bases of eigen spaces correspondingto distinct eigen values is a basis

5. Result 6.2.1: Results on Eigen Values and Eigen Vectors forSymmetric Matrices

(i): All eigen values are real

(ii): Eigen vectors corresponding to distinct eigen valuesare orthogonal

(iii): Special eigen properties of symmetric PD, PSD, ND,NSD, Indefinite matrices

(iv): Geometric and algebraic multiplicities are equal

(v): Orthonormal set of eigen vectors

6. Result 6.2.2: Orthogonally diagonalizable iff symmetric

7. Result 6.2.3: Similar Matrices Have the Same Eigen Spectrum

8. Result 6.2.4: Eigen values of powers of a matrix

530 Ch. 6. Eigen Values

9. Result 6.2.5: Spectral Theorem for Diagonalizable Matrices

10. Result 6.2.6: Eigen pairs for A and A− 1

11. Result 6.2.7: Sylvester’s Law of Inertia.

Contents

7 Software Systems for Linear Algebra Problems 5317.1 Difference Between Solving a Small Problem By Hand

and Scientific Computing Using a Computer . . . . . . 5317.2 How to Enter Vectors and Matrices in MATLAB . . . . 5337.3 Illustrative Numerical Examples . . . . . . . . . . . . . 5387.4 References . . . . . . . . . . . . . . . . . . . . . . . . . 552

i

ii

Chapter 7

Software Systems for LinearAlgebra Problems

This is Chapter 7 of “Sophomore Level Self-Teaching Web-book for Computational and Algorithmic Linear Algebra andn-Dimensional Geometry” by Katta G. Murty.

Nowadays there are several commercially available and highly pop-ular software systems for solving the problems discussed in previouschapters. Among them the best known are: MATLAB, EXCEL, MATH-EMATICA, MAPLE, and MACAULAY. We will provide illustarativenumerical examples of how to solve these problems using MATLAB(this name is derived from Matrix Laboratory) in this chapter. Forusing the other software systems, see the references cited on them.

7.1 Difference Between Solving a Small

Problem By Hand and Scientific Com-

puting Using a Computer

When we solve small numerical problems by hand, we always use exactarithmetic, so all the numbers in the various stages of the work areexactly what they are supposed to be (for example, a fraction like “1/3”

531

532 Ch. 7. Software Systems

is maintained as it is). Hand computation is not feasible for large sizeproblems encountered in applications. Such large problems are usuallysolved using a software package on a digital computer.

Digital computation uses finite precision arithmetic, not exactarithmetic; for example a fraction like “1/3” is rounded to 0.3333 or0.3 . . . 3 depending on the number of significant digits of precisionmaintained, introducing a small rounding error. These roundingerrors abound in scientific computation, and they accumulate duringthe course of computation. So results obtained in digital computationusually contain some rounding errors, and may only be approximatelycorrect. These errors make it nontrivial to implement many of thealgorithms discussed in earlier chapters to get answers of reasonableprecision. That’s why detailed descriptions of numerical implementa-tions of algorithms are very different and much more complicated thantheir mathematical descriptions given in earlier chapters.

As an example consider the GJ pivotal method for solving a systemof linear equations discussed in Section 1.16. Under exact arithmetic,we know that a redundant constraint in the system corresponds to a“0 = 0” equation in the final tableau and vice versa. However, underfinite precision arithmetic some of the “0” entries in it may becomenonzero numbers of hopefully small absolute value. So, what is actu-ally a “0 = 0” equation in the final tableau, may in reality become anequation with nonzero entries of small absolute value, and it is verydifficult to decide whether it is a redundant equation or not. In com-puter implementations, one difficult question faced at every stage iswhether a nonzero entry of small absolute value is actually a nonzeroentry, or a zero entry that has become nonzero due to rounding erroraccumulation. A practical rule that is often used selects a small pos-itive tolerance, and replaces any entry in the tableau whose absolutevalue is less than this tolerance by zero. Under such a rule, we can onlyconclude that a “0 = 0” equation in the final tableau is redundant toworking precision.

In the same way, when a square matrix has a determinant of verysmall absolute value, using finite precision arithmetic it is very difficultto decide whether it is actually nonsingular or singular, and softwaresystems will conclude that this matrix is singular to working precision.

7.2 Vectors and Matrices 533

Similarly while the definitions of linear independence or dependenceof a set of vectors and its rank defined in Chapter 4 are conceptuallyprecise and mathematically unambiguous under exact arithmetic, usingfinite precision arithmetic they can only be determined correct to theworking precision.

Numerical analysts have developed a variety of techniques to reducethe effects of rounding errors, some of these like partial pivoting orcomplete pivoting for pivot element selection are discussed very brieflyin Chapter 1. As a detailed discussion of these techniques is beyondthe scope of this book, the interested reader should refer to books inthat area (for example, see Gill, Murray, Wright[4.1]).

We will see the effects of rounding errors in some of the illustrativenumerical examples given below.

7.2 How to Enter Vectors and Matrices

in MATLAB

First you need to begin a MATLAB session. It should be noted thatthere are many versions of MATLAB in the market, and input prompts,and error and warning messages may vary from version to version. Also,different computer systems may have different commands for initiatinga MATLAB session on their system.

After getting the MATLAB prompt, if you want to enter a rowvector, a = (1, 6, 7,−9) say, type

a = [1 6 7 −9] or a = [1, 6, 7, −9]

with a blank space or a comma (,) between entries, and MATLABresponds with

a = 1 6 7 −9.

To enter the same vector as a column vector, type

a = [1, 6, 7, −9]’ or a = [1 6 7 −9]’ or a = [1; 6; 7; −9]

and MATLAB responds with

534 Ch. 7. Software Systems

a = 167

−9

To supress the system’s response, a semicolon (;) is placed as thelast character of the expression. For example, after typing a = [1, 6,7, −9]; MATLAB responds with a new line awaiting a new command.

MATLAB also lets one place several expressions on one line, a linebeing terminated by pressing the Enter button on the computer. In thiscase, each expression is separated by either a comma (,) or a semicolon(;). The comma results in the system echoing the output; the semicolonsupresses it. For example, by typing

a = [1, 2, 6, 9], c = [2, −3, 1, −4]

the system responds with

a = 1 2 6 9c = 2 −3 1 −4.

Matrices are entered into MATLAB row by row, with rows sepa-rated by either semicolons, or by pressing Enter button on the com-puter. For example, to enter the matrix

A =

(10 −2 40 8 −7

)

type A = [10, −2, 4; 0, 8, −7]

or typeA = [ 10 −2 4

0 8 −7 ]using Enter to indicate theend of 1st row

or typeA = [ 10 −2 4; . . .

0 8 −7 ]the . . . called ellipses isa continuation of a MAT-LAB expression to next line.Used to create more read-able code.

7.2 Vectors and Matrices 535

MATLAB has very convenient ways to address the entries of a ma-trix. For example, to display the (2, 3) entry in the above matrix Atype A(2, 3), to display 2nd column of A type A(:, 2), and todisplay 2nd row of A type A(2, :).

Some of the commands used for special matrices are:

eye(n) identity matrix of order ndiag([a1 a2 . . . an]) diagonal matrix with diagonal entries

a1, . . . , an

zeros(m, n) zero matrix of order m × nzero(n) zero matrix of order n × n

A′ transpose of matrix A

If B is a square matrix defined in MATLAB, diag(B) returns thediagonal elements of B. To see the contents of a vector or matrix Adefined earlier, just type A and MATLAB responds by displayingit.

The arithmetic operators to perform addition, subtraction, multi-plication, division, and exponentiation are: +, −, *, /, ˆ respec-tively. As an example, suppose the vectors x = (1, 0, 0), y = (0, 1, 0),z = (0, 0, 1) have been defined earlier. If you type 2 ∗x+3 ∗ y − 6 ∗ zMATLAB responds with

ans = 2 3 −6.

If you type w = 2 ∗ x + 3 ∗ y − 6 ∗ z MATLAB responds with

w = 2 3 −6.

If you type an arithmetical expression that is not defined, for ex-ample x+y′ with x, y as defined above (i.e., the sum of a row vectorand a column vector), MATLAB yields the warning

???Error using =⇒ +Matrix dimensions must agree.

Other arithmetical operations on matrices and vectors can be de-fined in MATLAB in the same way. Here are some MATLAB com-mands that we will use, and the outputs they produce. For a morecomplete list of MATLAB commands, see the MATLAB references

536 Ch. 7. Software Systems

given at the end of this chapter. Let A, B be two matrices, and x, y betwo vectors.

length(x) or size(x) the number of the elements in the vector xsize(A) m n where m × n is the order of A

dot(x, y) dot product of x and y (whether each is eithera row or a column) provided they have same no.of elements. This can also be obtained by typingx ∗ y provided x is a row vector & y is a col.vector

norm(x) the Eucledean norm ||x|| of the vector xdet(A) determinant of A, if it is a square matrixrank(A) rank of the matrix Ainv(A) inverse of matrix A, if A is an invertible square

matrixA ∗ B matrix product AB if it is defined.A\b solves the system of linear equations with A as

the coefficient matrix, and b as the RHS constantsvector

null(A) outputs 0-vector if system Ax = 0 has nononzero solution, or an orthonormal basis for thesubspace which is the set of all solutions of Ax =0 obtained by a method called singular valuedecomposition not discussed in the book if Ax =0 has nonzero solutions.

rref(A) outputs the reduced row echelon form of matrixA.

rrefmovie(A) shows all the operations that MATLAB performsto obtain the RREF of the matrix A.

null(A,′ r′) same as null(A) except that in the 2nd case itproduces the basis for the set of solutions of Ax =0 obtained by the GJ or the G elimination methodas discussed in Section 1.23.

eig(A) produces the eigen values of the square matrix A.

7.2 Vectors and Matrices 537

[V, D] = eig(A) produces D = diagonal matrix of eigen values ofA, and V = matrix whose column vectors are cor-responding eigen vectors.

all(eig(A + A′) > 0) for a square matrix A, this checks whether alleigenvalues of (A + A′) are strictly positive. Theoutput is 1 if the answer is yes, in this case A is apositive definite matrix; or 0 if the answer is no.In the latter case, using the command describedabove, one can generate all the eigenvalues andassociated eigen vectors of (A + A′). The eigenvector x associated with a nonpositive eigen valueof (A + A′) satisfies xT Ax ≤ 0.

all(eig(A + A′) >= 0) for a square matrix A, this checks whether alleigenvalues of (A+A′) are nonnegative. The out-put is 1 if the answer is yes, in this case A is apositive semidefinite matrix; or 0 if the answer isno and A is not PSD.

chol(A) for a symmetric square matrix A, this commandoutputs the Cholesky factor of A (Cholesky fac-torization is not discussed in this book) if A is PD.If A is not PD, it outputs an error message thatMatrix must be positive definite to use chol. So,this command can also be used to check if a givensquare symmetric matrix is PD.

When a system of linear equations Ax = b has more than onesolution, there are two ways of obtaining an expression of the generalsolution of the system using MATLAB. One is to get the RREF (acanonical tableau WRT a basic vector) of the system using the rref-movie command (see Examples 2, 3 given below), and then constructan expression of the general solution of the system using it as explainedin Section 1.8. The other is to get one solution, x say, using the A\bcommand; and then a basis for the null space of the coefficient ma-trix A using the null(A) command. If this basis is {x1, . . . , xs}, thenthe general solution of the system is x + α1x

1 + . . . + αsxs where

α1, . . . , αs are parameters that take real values.

538 Ch. 7. Software Systems

7.3 Illustrative Numerical Examples

1. Solving a Square System of Linear Equations:Suppose we want to find a solution for the following system and checkwhether it is unique. We provide the MATLAB session for doing this.The coefficient matrix is called A, and the RHS constants vector iscalled b, and the solution vector is called x.

2x1 + x2 − 3x3 − x4 = 1

−x1 − 2x2 + 4x3 + 5x4 = 6

7x1 − 6x4 = 7

−3x1 − 8x2 − 9x3 + 3x4 = 9

Type A = [2, 1, −3, −1; −1, −2, 4, 5; 7, 0, 0, −6; −3, −8, −9,3]

Response A =

2 1 −3 −1−1 −2 4 5

7 0 0 −6−3 −8 −9 3

Type b = [1, 6, 7, 9]’

Response b = 1679

Type x = A\bResponse x = 1.7843

−1.50620.04910.9151

The solution vector x is provided by MATLAB, this indicates thatthis is the unique solution, and that the coefficient matrix A is nonsin-gular.

7.3 Examples 539

In MATLAB the final canonical tableau is called RREF (reducedrow echelon form). The command: rref(A, b), outputs the RREF for

the augmented matrix (A...b) of the system. Here is the session to obtain

that for this example.

Type d =rref([Ab])

Response d =1.0000 0 0 0 1.7843

0 1.0000 0 0 −1.50630 0 1.0000 0 0.04910 0 0 1.0000 0.9151

2. Another Square System of Linear Equations:The ststem is given below. We call the coefficient matrix A, the RHSconstants vector b, and the solution vector x.

2x1 + x2 − 3x3 + x4 = 4

−x1 − 2x2 + 4x3 + 5x4 = −3

2x1 − 2x2 + 2x3 + 12x4 = 6

x1 − x2 + x3 + 6x4 = 3

The MATLAB session is carried out exactly as above, but this timeMATLAB gives the following final response.

Warning: Matrix is singular to working precision

x = InfInfInfInf

This indicates that the coefficient matrix A is singular, and thatthe syatem has no solution.

In MATLAB the final canonical tableau is called RREF (reducedrow echelon form). The command: rref(A, b), outputs the RREF for

the augmented matrix (A...b) of the system. When rank(A

...b) = 1 +

540 Ch. 7. Software Systems

rank(A), as in this case, MATLAB performs also a final pivot step inthe column vector B, which we do not want. In this case the command:

rrefmovie(A...b)

helps you to observe all the operations that MATLAB performs to

obtain the RREF of the augmented matrix (A...b). The matrix before

the final pivot step in the column of b is the RREF to our system. Forthis example, here is that output (here we did all the computations infractional form).

1 0 −2/3 7/3 7/30 1 −5/3 −11/3 −2/30 0 0 0 −20 0 0 0 0

The rows in this final tableau do not directly correspond to rows(constraints) in the original problem, because of row interchanges per-formed during the algorithm. To find out which original constraint eachrow in the final tableau corresponds to, you can use the row interchangeinformation that is displayed in the rrefmovie output.

3. A Rectangular System of Linear Equations:The system is given below. The symbols A, b, x refer to the coefficientmatrix, RHS constants vector, solution vector respectively.

x1 x2 x3 x4 x5 x6 x7 b2 1 0 0 −1 −3 4 30 −2 1 2 0 4 3 −24 0 1 2 −2 −2 11 4

−1 2 3 1 −2 0 0 55 1 5 5 −5 −1 18 10

The session is carried out exactly as above, and MATLAB producedthe following final response.

Warning: Rank deficient, rank = 3 tol = 3.3697e-014.

7.3 Examples 541

x = 0.76471.47060.9412

0000

Here since the system has a solution, the final canonical tableau,RREF can be produced with the command: rref(A, b). Here it is:

1 0 0 0.2941 −0.3529 −0.7059 2.4118 0.76470 1 0 −0.5882 0.2941 −1.5882 −0.8235 1.47060 0 1 0.8235 −0.5882 0.8235 1.3529 0.94120 0 0 0 0 0 0 00 0 0 0 0 0 0 0

As in the above example, to trace the correspondence of rows inthe RREF to the constraints in the original statement of the prob-lem, one has to take into account the row interchanges performed byMATLAB in the process of getting this RREF by using the command:rrefmovie(A, b).

4. Another Rectangular System of Linear Equa-tions: The system is given below. The symbols A, b, x refer to thecoefficient matrix, RHS constants vector, solution vector respectively.

x1 x2 x3 x4 x5 x6 x7 b1 0 −1 1 −2 1 −3 −40 −1 2 3 1 −1 2 21 −2 3 7 0 −1 1 −2

−3 1 2 −1 0 0 2 3−2 0 3 3 −1 0 1 3

The canonical tableau for this system, RREF, produced using thecommand: rrefmovie(A, b) is given below.

542 Ch. 7. Software Systems

x1 x2 x3 x4 x5 x6 x7 b1 0 0 6 −7 3 −8 −120 1 0 7 −11 5 −12 −170 0 1 5 −5 2 −5 −80 0 0 0 0 0 0 10 0 0 0 0 0 0 3

The last two rows in the canonical tableau represent inconsistentequations, “0 = α” for some α �= 0; so this system has no solution.To trace the correspondence of rows in the RREF to the constraintsin the original statement of the problem, one has to take into accountthe row interchanges performed by MATLAB in the process of gettingthis RREF.

5. A Homogeneous System of Linear Equations:Calling the coefficient matrix, RHS constants vector, solution vector asA, b, x respectively, here is the session.

Type A = [1,1,0; 0,1,1; 1, 0, 1; 2, 2, 2]

Response A =

1 1 00 1 11 0 12 2 2

Type b = zeros(4, 1) (Note: this command generates a 0-matrixof order 4 × 1)

Response b = 0000

Type x = A\bResponse x = 0

00

7.3 Examples 543

Type rank(A)

Response 3

Since rank of the coefficient mstrix is 3, from the results in Sections4.5 and 1.22, we conclude that this system has no nonzero solution.

6. Another Homogeneous System of Linear Equa-tions: Calling the coefficient matrix, RHS constants vector, solutionvector as A, b, x respectively, here is the session.

Type A = [1,−1,0, 2, 3, −2, 1; 0,1, −2, 1, −1, 3, −2; 1, 0, −2, 3,2, 1, −1; −1, 2, 1, −2, 1, 3, 2]

Response A =

1 −1 0 2 3 −2 10 1 −2 1 -1 3 −21 0 −2 3 2 1 −1

−1 2 1 −2 1 3 2

Type b = zeros(4, 1) (Note: this command generates a 0-matrixof order 4 × 1)

Response b = 0000

Type d =rref([Ab])

Response d =

1 0 0 7/3 16/3 −1/3 7/3 00 1 0 1/3 7/3 5/3 4/3 00 0 1 −1/3 5/3 −2/3 5/3 00 0 0 0 0 0 0 0

From the RREF we can see that this homogeneous system hasnonzero solutions. From the RREF a basic set of nonzero solutionsfor the system can be constructed as discussed in Section 1.23.

7. Checking Linear Independence: The most convenient

544 Ch. 7. Software Systems

way to check linear independence of a given set of vectors (either all rowvectors or all column vectors) using MATLAB is to write each of thesevectors as a column vector of a matrix, A say. Then check whether thehomogeneous system of equations Ax = 0 has a nonzero solution.If this system has no nonzero solution, the set is linearly independent.If x is a nonzero solution of the system, the set is linearly dependent;and x is the vector of coefficients in a linear dependence relation for itin the order in which the vectors are entered as column vectors in thematrix A.

As an example, consider the set of 4 column vectors, {A.1.A.2, A.3, A.4}of the matrix A in Example 1. Here is the session for checking its linearindependence; we suppress the display of the matrix in this session.

Type A = [2, 1, −3, −1; −1, −2, 4, 5; 7, 0,0, −6; −3, −8, −9, 3];

null(A)

Response ans = Empty matrix: 4-by-0.

This indicates that the system Ax = 0 has no nonzero solution,i.e., the set of column vectors of A is linearly independent.

8. Checking Linear Independence, Another Ex-ample: Consider the set of 4 column vectors, {A. 1.A. 2, A. 3, A. 4 } ofthe matrix A in Example 2. Here is the session for checking its linearindependence; we suppress the display of the matrix in this session.

Type A = [2, 1, −3, 1; −1, −2, 4, 5; 2, −2, 2, 12; 1, −1, 1, 6];

null(A)

Response ans =

−0.7818 −0.3411−0.2469 0.9111−0.5436 0.1382

0.1797 0.1857

Each of the column vectors in the output above is a nonzero solutionof Ax = 0 and these two vectors together form an orthonormal basisfor the subspace which is the set of all solutions of this homogeneoussystem, obtained by using a method called singular value decompo-sition, which is not discussed in the book. If you do not require an

7.3 Examples 545

orthonormal basis, but want the basis obtained by the GJ or G elim-ination methods as discussed in Section 1.22, change the commandnull(A) to null(A,′ r′), then the output is (each column vector is anonzero solution, together they form a basis for the subspace which isthe set of all solutions of Ax = 0):

Response ans =

0.6667 −2.33331.6667 3.66671.0000 0

0 1.0000.

Each of the column vectors in these outputs is the vector of coeffi-cients in a linear dependence relation for the set of column vectors ofthe matrix A in this example. For instance the first column vector inthe output under the command null(A) yields the linear dependencerelation

−0.7818A. 1 − 0.2469A. 2 − 0.5436A. 3 + 0.1797A. 4 = 0.

9. Matrix Inversion: Calling the matrix A, here is the ses-sion.

Type A = [1,2,1, −1; −1, 1, 1, 2; 0, −1, 2, 1; 2, 2, −1, 0]

Response A =

1 2 1 −1−1 1 1 2

0 −1 2 12 2 −1 0

Type inv(A)

Response ans = −0.1071 −0.2500 0.3929 0.42860.2500 0.2500 −0.2500 0.00000.2857 0.0000 0.2857 −0.1429

−0.3214 0.2500 0.1786 0.2857

10. Another Matrix Inversion: Calling the matrix A,here is the session.

Type A = [0, 1, 1, 2; 1, 0, 2, 1; 2, 1, 0, 1; 3, 2, 3, 4]

546 Ch. 7. Software Systems

Response A =

0 1 1 21 0 2 12 1 0 13 2 3 4

Type inv(A)

Response Warning: Matrix is close to singular or badly scaled.Results may be inaccurate. RCOND = 4.336809e− 018.

ans = 1.0e + 016∗0.1801 0.1801 0.1801 −0.1801

−1.2610 −1.2610 −1.2610 1.2610−0.5404 −0.5404 −0.5404 0.5404

0.9007 0.9007 0.9007 -0.9007

According to the warning the input matrix A in this example ap-pears to be singular, but reaching this exact conclusion is made difficultdue to rounding errors. Of course a singular matrix does not have aninverse, but an inverse is obtained because rounding errors have al-tered the outcome of singularity. The warning message indicates thatthe outputted inverse is probably not accurate due to the singularityof the original matrix. Most likely, an inverse is still computed dueto roundoff error introduced into the computation by computer arith-metic.

11. Nearest Point to x On a Given Straight Line:Find the nearest point, x∗, (in terms of the Euclidean distance), to thegiven point x, on the straight line L given in parametric form: L ={x : x = a + λc, where a = (2, 1, 3, 3)T , c = (0, 1,−1, 2)T , and λ is thereal valued parameter}.

We denote the point x by x1 to avoid confusion, and we call thenearest point xn. Here is the session.

Type a = [2, 1, 3, 3];

c = [0, 1,−1, 2]′;x1 = [1,−1, 2, 0]′;lambda = c′ ∗ (x1 − a)/[c(1, 1)2 + c(2, 1)2 + c(3, 1)2 + c(4, 1)2]

Response lambda = −1.1667

7.3 Examples 547

Type xn = 1 + lambda ∗ c

Response xn =

2.0000−0.1667

4.16670.6667

Type dist = [[(x1(1, 1) − xn(1, 1)]2 + [(x1(2, 1) − xn(2, 1)]2+[(x1(3, 1) − xn(3, 1)]2 + [(x1(4, 1) − xn(4, 1)]2](1/2)

Response dist = 2.6141“dist” is the Euclidean distance between x1 and the nearest point

to x1 on the given straight line. All these formulae are from Section3.16.

12. Nearest Point to x On a Given Hyperplane:Find the nearest point, x∗, (in terms of the Euclidean distance), tothe given point x, on the hyperplane H = {x : ax = a0 where a =(1,−2, 3,−4), a0 = −13}.

Again we denote x by x1, and the nearest point x∗ by xn to avoidconfusion. Here is the session.

Type a = [1,−2, 3,−4]; a0 = −13; x1 = [1,−1, 2, 0]′;xn = x1+a′∗ [−(a∗x1−a0)/(a(1, 1)2+a(1, 2)2+a(1, 3)2+a(1, 4)2)]

Response xn =

0.26670.4667

−0.20002.9333

Type dist = [[(x1(1, 1) − xn(1, 1)]2 + [(x1(2, 1) − xn(2, 1)]2+[(x1(3, 1) − xn(3, 1)]2 + [(x1(4, 1) − xn(4, 1)]2](1/2)

Response dist = 4.0166“dist” is the Euclidean distance between x1 and the nearest point

to x1 on the given hyperplane. All these formulae are from Section3.16.

548 Ch. 7. Software Systems

13. Nearest Point to x In a Given Affine Space:Find the nearest point, x∗, (in terms of the Euclidean distance), to thegiven point x = (1, 1, 0)T , in the affine space F = {x : Ax = b whereA, b are given below}.

A =

(1 1 −12 1 1

), b =

(29

).

Here is the portion of the session after reading in A, b, x1 where weare calling the given point x as x1. The nearest point to x1 in the givenaffine space will be denoted by xn.

Type xn = x1 − A′ ∗ inv(A ∗ A′) ∗ (A ∗ x1 − b)

Response xn =

2.71431.42862.1429

The Euclidean distance between x1 and xn using the same com-mand as in the above examples is 2.7775.

14. Checking PD, PSD: Check whether the following ma-trices A, B, C are PD, PSD, or neither.

A =

3 1 2 2−1 2 0 2

0 4 4 5/30 2 −13/3 6

, B =

8 2 1 04 7 −1 12 1 1 22 −2 1 1

C =

0 −2 −3 −4 52 3 3 0 03 3 3 0 04 0 0 8 4

−5 0 0 4 2

Here are the sessions after reading in the matrices.

7.3 Examples 549

Type all(eig(A + A′) > 0

Response ans = 0. This means that A is not PD.

We show how the same thing is done using the chol cammand.

Type chol(A + A’)

Response ??? Error using =⇒ chol. Matrix must be positivedefinite.

We check that B is not PD using same procedures. Now to checkwhether B is PSD, we do the following.

Type all(eig(B + B′) >= 0)

Response 1. This means that B is PSD.

Since C has a 0 diagonal element, it is clearly not PD. To checkwhether it is PSD, we use the command [V, D] = eig(Cs) commandwhere Cs = C + C ′. We get

D =

0 0 0 0 00 12 0 0 00 0 0 0 00 0 0 20 00 0 0 0 0

Since all the eigenvalues are nonnegative C is PSD.

15. Eigen Values and Eigen Vectors: Calling the matrixA, here is the session.

Type A = [−3, 1, −3; 20, 3, 10; 2, −2, 4]

Response A =−3 1 −320 3 102 −2 4

Type eig(A)

550 Ch. 7. Software Systems

Response ans =−2.0000

3.00003.0000

[V, D] = eig(A)

Response V =0.4082 −0.4472 0.4472

−0.8165 0.0000 0.0000−0.4082 0.8944 −0.8944

Response D =−2.0000 0 0

0 3.0000 00 0 3.0000

16. Matrix Diagonalization: Diagonalize the followingmatrices A, B.

A =

1 3 3−3 −5 −3

3 3 1

, B =

2 4 3−4 −6 −3

3 3 1

.

The command [V, D] = eig(A) gives the following output.

V =0.5774 0 −0.7459

−0.5774 −0.7071 0.08530.5774 0.7071 0.6606

D =1 0 00 −2 00 0 −2

−2 is an eigenvalue of A with algebraic multiplicity 2. Also, sincetwo eigenvectors (which are not scalar multiples of each other) areassociated with this eigenvalue, its geometric multiplicity is also 2. So,A has a complete set of eigenvectors which is linearly independent. So,A can be diagonalized. In fact by the results discussed in Chapter 6,the matrix V whose column vectors are the distinct eigenvectors of Adiagonalizes A, i.e., V −1AV = D.

7.3 Examples 551

The command [V, D] = eig(B) gives the following output.

V =−0.5774 0.7071 0.7071

0.5774 −0.7071 −0.7071−0.5774 0.0000 0.0000

D =1 0 00 −2 00 0 −2

Since the two eigenvectors associated with the eigenvalue −2 arethe same, it indicates that the geometric multiplicity of the eigenvalue−2 is 1, while its algebraic multiplicity is 2. So, B does not have acomplete set of eigenvectors, and hence it is not diagonalizable.

17. Orthogonal Diagonalization: Orthogonally diago-nalize the following matrix A.

A =

5 2 9 −62 5 −6 99 −6 5 2

−6 9 2 5

.

Since A is symmetric, it is possible to orthogonally diagonalize Aby the results in Chapter 6. The command [V, D] = eig(A) producedthe following output:

V =

0.5 0.5 −0.5 0.50.5 0.5 0.5 −0.50.5 −0.5 −0.5 −0.50.5 −0.5 0.5 0.5

D =

10 0 0 00 4 0 00 0 18 00 0 0 −12

552 Ch. 7. Software Systems

By the results in Chapter 6, the matrix V whose column vectorsform a complete set of eigenvectors for A orthogonally diagonalize A,i.e., V T AV = D.

7.4 References

References on MATLAB

[7.1] M. Golubitsky and M. Dellnitz, Linear Algebra and DifferentialEquations Using MATLAB, Brooks/Cole Publishing Co., NY, 1999.[7.2] E. B. Magrab and others, An Engineer’s Guide to MATLAB,Prentice Hall, Upper Saddle River, NJ, 2000.7.3] C. F. Van Loan, Introduction to Scientific Computing, MATLABCurriculum Series, Prentice Hall, Upper Saddle River, NJ, 1997.[7.4] The Student Edition of MATLAB, Users Guide, The MathWorksInc., Natick, MA.

References on Other Software Systems

[7.5] B. W. Char, et. al. First Leaves: A Tutorial Introduction toMaple V, Springer Verlag, NY, 1992.[7.6] J. H. Davenport, Computer Algebra: Systems and Algorithms forAlgebraic Computation, Academic Press, San diago, 1993.[7.7] D. Eugene, Schaum’s Outline of Theory and Problems of Mathe-matica, McGraw-Hill, NY, 2001.[7.8] M. B. Monagan, et. al., Maple V Programming Guide, Springer,NY, 1998.[7.9] Microsoft Corp., Microsoft Excel User’s Guide, Version 5.0, Red-mond, WA, 1993.[7.10] S. Wolfram, The Mathematica Book, 3rd ed., Wolfram Media,Cambridge University Press, 1996.