csc5160 topics in algorithms tutorial 1

CSC5160 Topics in Algorithms

Tutorial 1

Jan 25 2007Jerry Le

[email protected]

mailto:[email protected]

Outline

• Basic Linear Algebra – a quick review Vector space, Spanning set Linear independency Matrix: operations, rank, inverse,

eigenvalue, determinant

• Introduction to Linear Programming

Linear AlgebraVector space, spanning set

• Vector space- a set V that is closed under the ‘+’ and ‘•’ operations

- e.g.

VBV and ABAV, Then If A,B

R

RR

2102

210

32

,,

0,,

aaaxaxaaP

zyx

z

y

x

P ,

Note: A vector space does not necessarily contain “vectors”!

Linear AlgebraVector space, spanning set

• Spanning set- Definition: the spanning set of a nonempty set S is the set of all linear combinations of vectors in S

- Definition: a subspace of a vector space is itself a vector space

- Lemma: The span of a subspace is itself a vector space

- e.g.

SssccscscSspan nnnn

,,,,)( 1111 and R

2

1

1,

1

1R is span its , set consider

Linear AlgebraLinear independency

• Linear independency- Definition: A subset of a vector space is linearly independent if none of its elements can be represented as a linear combination of the others.

- e.g.

2

2,

1

1,

1

1

linearly independent

linearly dependent

Linear AlgebraLinear independency

• Linear independency- in other words:

- Basis

Definition: A basis of vector space V is a linearly independent subset whose span is V

ia

VaVaaaa

VVV

i

nnn

n

,0

0,,,

,,,

1121

21

are

satisfy that only the iff

tindependenlinearly are vectors The

R

22

2

2,

1

1,

1

1RR of basis a Not is it but , spans

Linear AlgebraMatrix

• Operations

- sums and scalar products

- multiplication

• Rank of a matrixDefinition: the rank of a matrix A is the maximal number of linearly independent rows (or columns) of A

?

333231

232221

131211

232221

131211

bbb

bbb

bbb

aaa

aaa


• Rank of a matrix (cnt.)- Properties (consider an m-by-n matrix A) 1. the rank of A is at most min(m,n) 2. if the rank is n (m), A has full column (row) rank 3. if A is a square matrix (m=n), A is invertible iff it has rank n (Invertible: there exists B such that AB=In) 4. if B is an n-by-k matrix with rank n, then AB has the same rank as A 5. if B is an k-by-m matrix with rank m, then BA has the same rank as A …


• Rank of a matrix (cnt.)- Computation

by Gauss elimination (or Gauss-Jordan reduction)

3212

11213

8112

1100

12/12/10

8112

1100

3010

2001rank: 3


• Invertible matrix- an n-by-n matrix A is invertible iff

1. there exists an n-by-n matrix B such that AB=BA=In

OR 2. A has rank n

OR 3. the equation Ax=0 has only trivial solution x=0

OR 4. the equation Ax=b has exactly one solution in Rn

OR 5. the rows (columns) of A are linearly independent

OR 6. the rows (columns) of A form a basis of Rn

OR 5. the transpose AT is invertible

OR …


• Eigenvalue & Eigenvector- consider an m-by-n matrix A as linear transformation:

- Definition: for an n-by-n matrix A, if there exist an non-zero vector x and a scalar λ, such that

Ax=λx

then λ is the eigenvalue of A, and x is the eigenvector of A.

- The eigenvalue (λ) “visualizes” the effect that a linear transformation (A) produces on a vector (x).

yAxxFyxF )(: mn RR


• Eigenvalue & Eigenvector (cnt.)- how to compute?

Ax=λx Ax=λInx (A-λIn)x=0

(A-λIn)x=0 must have non-trivial solutions

i.e. B=(A-λIn)x is NOT invertible

• Determinant- how to determine if an (n-by-n) matrix (A) is invertible?

by Gauss elimination


• Determinant (cnt.)- Gauss elimination: by linear row (column) operations

If AFJMX=0, then A is not invertibleso, can we define some function det(A), which is equal to AFJMX, or at least equal to λAFJMX?

'

0000

000

00

0

A

X

NM

LKJ

IHGF

EDCBA

yxwvu

tsrqp

onmlk

jihgf

edcba

A ji

i

ji

operations row/column


• Determinant (cnt.)- Tracing every step of a specific process of Gauss elimination, we get

nspermutationn PaaaA

ihg

fed

cba

AegdhcfgdibfheiaA

dc

baAbcadA

)(,)2(,2)1(,1)det(

),()()()det(

,)det(

Generally,


• Determinant (cnt.)- determinant as size functions

(x1,y1)

(x2,y2)

21

21

yy

xxA

det(A) is exactly the area of the parallelogram!

More generally, the absolute value of det(A) is the volume of the box formed by A’s column vectors V1,V2,…Vn in n-dimensional space

Outline

• Basic Linear Algebra – a quick review

• Introduction to Linear Programming Definition Example Basic feasible solution Geometry of LP

Linear programmingDefinition

• General optimization

• Linear programming

- when all f(x),gi(x),hj(x) are linear (general form)

nR

x

pjxh

mixg

xf

j

i

where

to subject

minimize

,,1,0)(

,,1,0)(

)(

inequality constraints

equality constraints

objective function

Linear programmingExample

• Example: the Diet problemFood (n kinds of food)

Nutrient … requirement

Fat

Vitamin aij ri

Protein

amount

boughtxj

Cost cj

m kin

ds o

f nu

trient


• Example: the Diet problem- question: how to buy the food such that the cost is minimized, while the minimal nutrient requirement is satisfied?

0

min

x

rAx

cx

s.t.

nutrient th-i of trequiremen minimal r

j food of unit one of cost the :c

bought food th-j of amount x

food th-j of unit one in nutrient th-i of amount a

i

j

j

ij

:

:

:

Canonical form of LP (without equality constraints in the general form)


• Example: the Diet problem- what if the amount of nutrient must be exactly the same with the requirement?

0

min

x

rAx

cx

s.t.

nutrient th-i of trequiremen r

j food of unit one of cost the :c

bought food th-j of amount x

food th-j of unit one in nutrient th-i of amount a

i

j

j

ij

:

:

:

Standard form of LP (without inequality constraints in the general form)


• Equivalent of the three forms of LP- the general form, canonical form (without equality constraints) and standard form (without inequality constraints) are equivalent- One form can be transformed into another

Therefore, we can simply consider standard form without loss of generality for all LPs

0

is

bsAxbAx

bAx

bAxbAx

:constaintsequality into sconstraint inequality

:sconstraint inequality into sconstraintequality

Linear programmingBasic feasible solution

• Basic feasible solutions- consider the standard form of LP:

nR

cbxnmA

x

bAx

cx

,,

0

min

matrix, an is where

s.t.

How to find the solution?1. solve the equation system Ax=b (find basic solutions)2. pick the basic feasible solutions satisfying x≥0, out of the basic solutions3. find the solution that minimize cx


• Basic feasible solutionshow to find the basic solution of Ax=b?• If A is invertible, then x has unique solution

• If A is not invertible, assuming A has rank m 1. select a collection of m linearly independent column vectors of A 2. set the components of x that are not corresponding to the selected column vectors to zero 3. solve the m resulting equations to get the remaining component of x


• Basic feasible solutions - how to find the basic solution of Ax=b? (cnt.)

6

3

2

4

1000130

0100100

0010001

0001111

7

6

5

4

3

2

1

x

x

x

x

x

x

x

(0,0,0,4,2,3,6)feasible

6

3

2

4

1000130

0100100

0010001

0001111

7

6

5

4

3

2

1

x

x

x

x

x

x

x

(0,4,0,0,2,3,-6)not feasible (because x7<0)

Linear programmingGeometry of LP

• Geometry of LP- consider the LP

0,

12

max

yx

xy

yx

yx

s.t.

x

y

x+2y=1

y=x

feasible region

objective

Linear programmingGeometry of LP

• Geometry of LPin Rn space

- the linear constraints form a

closed convex polytope

- optimal solution appears on

the boundaries

- local optimal is global optimal

- Simplex algorithm: starting from a vertex, search along the edges until reaching the vertex of the optimal solution

csc5160 topics in algorithms tutorial 1

Documents

rank n

n matrix b

n invertible

matrix cnt

linear transformation

square matrix

vector x

vector space lemma