linear spacesweb.eecs.umich.edu/~fessler/course/600/l/l02.pdf · 2004. 10. 4. · vector spaces in...

Chapter 2

Linear Spaces

Contents

Field of Scalars . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . 2.2

Vector Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . 2.3

Subspaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . 2.5

Sum of subsets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . 2.5

Linear combinations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . 2.6

Linear independence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . 2.7

Basis and Dimension . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . 2.7

Convexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . 2.8

Normed linear spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . 2.9

The`p andLp spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . 2.10

Topological concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . 2.12

Open sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . 2.13

Closed sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . 2.14

Bounded sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . 2.15

Convergence of sequences . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . 2.16

Series . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . 2.17

Cauchy sequences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . 2.18

Banach spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . 2.19

Complete subsets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . 2.19

Transformations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . 2.21

Linear transformations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . 2.21

Continuity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . 2.22

Compactness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . 2.23

Upper semicontinuous functions . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . 2.26

Quotient Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . 2.27

Denseness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . 2.27

Separability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . 2.27

Schauder basis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . 2.27

Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . 2.28

2.1

2.2 c© J. Fessler, October 4, 2004, 12:44 (student version)

In systems analysis,linear spacesare ubiquitous.

Why? Linear systems/models are easier to analyze; many systems, particularly in signal processing, are deliberately designed tobe linear; linear models are a useful starting point (approximation) for more complicated nonlinear cases.

Formal definitions of a vector space use the concept of afield of scalars, so we first review that.

Field of Scalars(from Applied Linear Algebra, Noble and Daniel, 2nd ed.)

A field of scalarsF is a collection of elementsα, β, γ, . . . along with an “addition” and a “multiplication” operator.

To every pair of scalarsα, β in F , there must correspond a scalarα + β in F , called thesumof α andβ, such that• Addition is commutative:α + β = β + α• Addition is associative:α + (β + γ) = (α + β) + γ• There exists a unique element0 ∈ F , calledzero, for whichα + 0 = α, ∀α ∈ F• For everyα ∈ F , there corresponds a unique scalar(−α) ∈ F for whichα + (−α) = 0.

To every pair of scalarsα, β in F , there must correspond a scalarαβ in F , called theproduct of α andβ, such that• Multiplication is commutative:αβ = βα• Multiplication is associative:α(βγ) = (αβ)γ• Multiplication distributes over addition:α(β + γ) = αβ + αγ• There exists a unique element1 ∈ F , calledone, or unity , or theidentity element, for which1α = α, ∀α ∈ F• For every nonzeroα ∈ F , there corresponds a unique scalarα−1 ∈ F , called theinverseof α for whichαα−1 = 1.

Simple facts for fields:• 0 + 0 = 0 (useα = 0 in the definition of 0)• −0 = 0 Proof. For anyα, by the associative property(α + 0) + (−0) = α + (0 + (−0)) henceα + (−0) = α. Hence, since

the zero element is unique,−0 = 0.

Example. The set of rational numbersQ (with the usual definition of addition and multiplication) is a field.

The only fields that we will need are the field of real numbersR and the field of complex numbersC.Therefore, hereafter we will useF when describing results that hold for eitherR or C.

c© J. Fessler, October 4, 2004, 12:44 (student version) 2.3

Vector Spaces

In simple words, a vector space is a space that is closed undervector addition and under scalar multiplication.

Definition. A vector spaceor linear spaceconsists of the following four entities.

1. A fieldF of scalars.

2. A setX of elements calledvectors.

3. An operation calledvector addition that associates asumx + y ∈ X with each pair of vectorsx,y ∈ X such that• Addition is commutative:x + y = y + x

• Addition is associative:x + (y + z) = (x + y) + z

• There exists an element0 ∈ X , called thezero vector, for whichx + 0 = x, ∀x ∈ X• For everyx ∈ X , there corresponds a unique vector(−x) ∈ X for whichx + (−x) = 0.

4. An operation calledmultiplication by a scalar that associates with each scalarα ∈ F and vectorx ∈ X a vectorαx ∈ X ,called theproduct of α andx, such that:• Associative:α(βx) = (αβ)x• Distributiveα(x + y) = αx + αy

• Distributive(α + β)x = αx + βx

• If 1 is the identify element ofF , then1x = x. ∀x ∈ X .• 0x = 0 for anyx ∈ X .

The requirement thatx + y ∈ X andαx ∈ X is sometimes called theclosure property.

Simple facts for vector spaces:• 0 is unique.• (−1)x = −x for x ∈ X .

Proof.x + (−1)x = 1x + (−1)x = (1 + (−1))x = 0x = 0.• α0 = 0 for α ∈ F .

Proof.α0 = α0 + 0 = α0 + [α0 + (−α0)] = [α0 + α0] + (−α0) = α(0 + 0) + (−α0) = α0 + (−α0) = 0.• x + y = x + z impliesy = z (cancellation law)• αx = αy andα 6= 0 impliesx = y (cancellation law)• αx = βx andx 6= 0 impliesα = β (cancellation law)• α(x − y) = αx − αy (distributive law)• (α − β)x = αx − βx (distributive law)• −αx = α(−x) = −(αx)


What are some examples? (Linear algebra classes focus on finite-dimensional examples.)

Important Vector Spaces• Euclidean spaceor n-tuple space: X = Rn. If x ∈ X , thenx = (a1, a2, . . . , an) whereai ∈ R and we use ordinary addition

and multiplication:x + y = (a1 + b1, a2 + b2, . . . , an + bn) andαx = (αa1, . . . , αan).(As a special case, the set of real numbersR (with ordinary addition and multiplication) is a trivial vector space.)

• X = L1

[R2]. The set of functionsf : R2 → R that areabsolutely (Lebesgue) integrable:

∫∞−∞

∫∞−∞ |f(x, y)|dx dy < ∞,

with the usual pointwise definition of addition and scalar multiplication.To show show thatf, g ∈ L1

[R2]

impliesf + g ∈ L1

[R2], one can apply the triangle inequality:|f + g| ≤ |f | + |g|.

• The set of functions on the planeR2 that are zero outside of the unit square.• The set of solutions to a homogeneous linear system of equationsAx = 0.• C[a, b]: the space of real-valued, continuous functions defined on the interval[a, b].• The space ofband-limited signals.• Many more in Luenberger...

Example. For1 ≤ p < ∞, define the following infinite-dimensional1 space:

X = Rp[a, b] =

{f : [a, b] → R : f is Riemann integrable and

∫ 1

0

|f(t)|p dt < ∞}

,

(with the usual pointwise definitions of addition of functions and multiplication of functions by a scalar). To show thatthis spaceis a vector space, the only nontrivial work is verifyingclosure.

Clearly if f ∈ X thenαf ∈ X since∫ 1

0|αf(t)|p dt = |α|p

∫ 1

0|f(t)|p dt < ∞, soX is closed under scalar multiplication.

To show thatf +g ∈ X if f, g ∈ X , i.e., closure under addition, requires a bit more work. Note thatsince|a+b| ≤ 2max{|a|, |b|},it follows for p ≥ 1 that

|a + b|p ≤ 2p max{|a|p, |b|p} ≤ 2p[|a|p + |b|p].Hence iff, g ∈ X :

∫ 1

0

|f(t) + g(t)|p dt ≤∫ 1

0

[2p|f(t)|p + 2p|f(t)|p] dt ≤ 2p

∫ 1

0

|f(t)|p dt +2p

∫ 1

0

|f(t)|p dt < ∞ + ∞ = ∞.

showing closure under addition.

Example. Can X = (0,∞) with F = R be a vector space? ??

Cartesian product

We can make a “larger” vector space from two vector spacesX andY (having a common fieldF) by forming theCartesianproduct of X andY, denotedX × Y, which is the collection of ordered pairs(x,y) wherex ∈ X andy ∈ Y.

X × Y = {(x,y) : x ∈ X andy ∈ Y} .

To be a vector space we must define vector addition and scalar multiplication operations, which we define component-wise:• (x1,y1) + (x2,y2) = (x1 + x2,y1 + y2)• α(x,y) = (αx, αy), ∀α ∈ F .

Fact. With these definitions the Cartesian product of two vector spaces is indeed a vector space.The above definition generalizes easily to higher-order combinations.

Example. R3 = R × R × R.

1A definition of dimension is forthcoming...


2.3Subspaces

A (nonempty) subsetS of a vector spaceX is called asubspaceofX if S, when endowed with the addition and scalar multiplicationoperations defined forX , is a vector space,i.e., αx + βy ∈ S wheneverx,y ∈ S andα, β ∈ F .

Example. The subset ofX = R2[−1, 1] consisting of symmetric functions (f(−t) = f(t)) is a subspace ofX . It is clearly closedunder addition and scalar multiplication.

What are the four types of subspaces of R3? ??

Intuition: think of a subspace like a line or plane (or hyperplane) through the origin.

Properties of subspaces• 0 ∈ S• {0} is a subspace ofX• X is a subspace ofX• A subspace not equal to the entire spaceX is called aproper subspace• If M andN are subspaces of a vector spaceX , then the intersectionM ∩ N is also a subspace ofX . Proof.see text

Think: intersection of planes (through the origin) in 3d.• Typically theunion of two subspaces isnot a subspace.

Think: union of planes (through the origin) in 3d.

Although unions usually fail, we can combine two subspaces by an appropriate sum, defined next.

Sum of subsets

Definition. If S andT are twosubsetsof a vector space, then thesum of those subsets, denotedS + T is defined by

S + T = {x = s + t : s ∈ S, t ∈ T} .

Example. What is the sum of a plane and a line (both through origin) in R3? ??

Example. ConsiderX = R2, with S = {(x, 0) : x ∈ [0, 1]} andT = {(0, y) : y ∈ [0, 1]}. ThenS + T is the unit square.

Proposition. If M andN are subspaces of a vector spaceX , then the sumM + N is a subspace ofX .

Proof.see text

Does the previous example illustrate this proposition? ??

Example. LetX = {f : f(t) = a sin(t + φ) for a, φ ∈ R} (with the usual definitions of addition and scalar multiplication2).ThenM = {f : f(t) = a sin(t) for a ∈ R} andN = {f : f(t) = a cos(t) for a ∈ R} are both (proper) subspaces ofX .

What is M + N? ??

2It is time to stop saying this. From now on we leave it implicit whenever this is clear, which it usually is.


Linear combinations

Definition. A finitesum∑n

i=1 αixi for xi ∈ X andαi ∈ F is called alinear combination.

(The associative property of vector addition allows us to write such a sum without parentheses.)

Depending on where thexi’s originated we get various properties of linear combinations.• X : If xi ∈ X , i = 1, . . . , n, then

∑ni=1 αixi ∈ X . This is shown easily by induction from the definition of a vector space.

• M : If xi ∈ M, i = 1, . . . , n, whereM is a subspace, then∑n

i=1 αixi ∈ M by induction from the definition of a subspace.Any linear combination of vectors from a subspace is also in the subspace.

• S: What if we take linear combinations from asubsetrather than asubspace?

Definition. If S is asubsetof a vector spaceX , then thesubspace generatedby S is the subspace of linear combinations drawnfrom S, defined by

[S] =

{x ∈ X : x =

n∑

i=1

αixi, for xi ∈ S, αi ∈ F , andn ∈ N

}.

• [S] is also called thespanor thelinear hull of S.• [S] is indeed a subspace ofX since a linear combinations of linear combinations is itself a linear combination.• [S] is the smallest subspace ofX containingS, i.e., if M is a subspace ofX that containsS, then[S] ⊆ M .

• If M is a subspace ofX , then[M ] = ??• ClearlyS ⊆ [S]• Note that onlyfinitesums are involved, as in all linear combinations.

Example. For X = R3, what is [S] when S consists of a line through the origin plus any point not on that line?

??

Intuition: a subspace of a general vector space generalizesthe notion of a line or plane through the origin of Euclidean 3D space.

What about lines or planes that are not through the origin?

Linear varieties

skip for now. Not needed in Ch. 2 problems. Wait until3.10.

Definition. A subsetV of a vector spaceX is called alinear variety iff V = x0 + M for somex0 ∈ X and some subspaceMof X . Another term used isaffine subspace.

Linear varieties arise in certain minimum norm problems.


2.5Linear independence

Often we need to quantify how “big” a subspace is.

Definition. A vectorx is calledlinearly dependenton a setS of vectors iffx ∈ [S], i.e., x is in thespanof S.

Otherwise, ifx /∈ [S], thenx is calledlinearly independentof S.

Definition. A setS of vectors is called alinearly independent setif each vector in the set is linearly independent of the remainingvectors in the set,i.e.,

∀x ∈ S, x /∈ [S − {x}] .

Remark.S may be uncountable, but testing whetherx ∈ [S − {x}] requires consideration only of finite sums, by the definitionoflinear combinations.

Example. This illustrates thatS can be uncountable!

LetX = {f : [0, 1] → R} and definegs(t) =

{1, t = s0, otherwise.

ThenS = {gs : s ∈ [0, 1]} is a linearly independent subset ofX .

Theorem. A finite set of vectors{x1, . . . ,xn} is linearly independent iff∑n

i=1 αixi = 0 implies thatαi = 0, ∀i.

(We are skipping proofs that are found in basic linear algebra books.)

Corollary . If a finite setS = {x1, . . . ,xn} is linearly independent andy ∈ [S], theny has auniqueexpansiony =∑n

i=1 αixi

for someα1, . . . , αn ∈ F .

Basis and Dimension

Definition. A setS is called abasisor Hamel basis[3, p 183] forX iff S is linearly independent and[S] = X .

Luenberger [4, p. 20] says “finite set” but Naylor [3, p. 183] and Maddox [2, p. 74] do not. Let us agree to use the above definition,rather than Luenberger’s.

Note thatX can (and usually will) have more than one basis!

Definition. If X has a basisS that is a finite set, then we callX finite dimensional.Otherwise, if no such finiteS exists, we callX infinite dimensional.

Definition. A space with a basis consisting ofn elements is called ann-dimensional space.This terminology is acceptable thanks to the following result.

Theorem. Any two bases for a finite-dimensional vector space contain the same number of elements.

Most, but not all, properties of (more easily understood) finite-dimensional spaces generalize to infinite-dimensional spaces.

Example. Pn = {polynomials of degree≤ n}A basis is

{1, t, t2, . . . , tn

}, which has dimensionn. Why linearly independent? ??

Another basis is theLegendre polynomials: dk/dtk(t2 − 1)k, k = 1, . . . , n.

Exercise. C[0, 1] is infinite dimensional. Hint: prove by counter example consideringPn+1.

A basis is a generalization of the usual concept of acoordinate system.

Fact. [3, p 184] If S is linearly independent set in a vector spaceX , then there exists a basisB for X such thatS ⊆ B.Thus, every vector space has a Hamel basis, (since the empty set is linearly independent).

However, “Hamel basis is not the only concept of basis that arises in analysis. There are concepts of basis that involve topologicalas well as linear structure. ... In applications involving infinite-dimensional spaces, a useful basis, if one even exists, is usually


something other than a Hamel basis. For example, a complete orthonormal set is far more useful in an infinite-dimensionalHilbertspace than a Hamel basis” [3, p. 183]. (More on this later!)

We often need sets with less rigid structure than subspaces but that nevertheless still have some structure, so we digress a bit here.

2.4Convexity

Luenberger describes convexity as “the fundamental algebraic concept of vector space.” (p. 25)

According to Oxford English dictionary, analgebra is “a calculus of symbols combining according to certain defined laws.”

Definition. A setK in a vector space is calledconvexiff for any x,y ∈ K, αx + (1 − α)y ∈ K for all α ∈ [0, 1].

Geometrically: for any two points in a convex set, the “line segment” between them is also in the set.

Properties of convex sets• Subspaces and linear varieties are convex.• {0} is “vacuously” convex.• Forα ∈ F , αK , {x = αk : k ∈ K} is convex ifK is convex (magnification or minification of a set)• K1 + K2 is convex ifK1 andK2 are convex sets in a common vector space.• If C is a collection of convex sets (in a common vector space), then

⋂K∈C K is a convex set. (Important forPOCSmethods.)

Convex hull

Definition. Theconvex coveror convex hullof a setS in a vector space is the smallest convex set containingS, denotedco(S).Equivalently, the convex hull ofS is the intersection of all convex sets containingS:

co(S) =⋂

{cK convex : S⊆K}K. (Picture in 2D of blob and its convex hull)

Problem 2.4gives a more constructive form forco(S).

Cone

Definition. A setC in a vector space is called acone with vertex at the origin if x ∈ C implies thatαx ∈ C for all α ∈ [0,∞).

Convex ConeCone

Example. The space of nonnegative continuous (real) functions is aconvex conein the vector space of continuous functions.

Relationships between cones, convex sets, and subspaces

SubspacesCones

Convex Cones

Subsets

Convex Sets{X}

Exercise. An arbitrary union of subspaces (in a common VS) is a cone.


2.6Normed linear spaces

We’ve gone about as far as we can with just the basic axioms of avector space. Fortunately, most of the vector spaces of interesthave additional structure: a norm.

Definition. A norm on a vector spaceX is a function‖·‖ : X → R that satisfies the following for everyx ∈ X .• ‖x‖ ≥ 0 (nonnegativity)• ‖x‖ = 0 iff x = 0 (positive definiteness)• ‖αx‖ = |α| ‖x‖ , ∀α ∈ F (scaling property) (homogeneity)• ‖x + y‖ ≤ ‖x‖ + ‖y‖ , ∀y ∈ X (triangle inequality )

Notice that the absolute value function|α| appears above. It is here where our “restriction” to the fieldsR andC enters3.

Definition. (X , ‖·‖) is anormed vector spaceor normed linear spaceor normed linear vector spaceor justnormed space4.

Clearly a norm generalizes the usual notion oflength.

The following lemma arises remarkably frequently in proofs.

Lemma. In a normed space,‖x‖ − ‖y‖ ≤ ‖x − y‖ for all x,y ∈ X , i.e., | ‖x‖ − ‖y‖ | ≤ ‖x − y‖.Proof.‖x‖ = ‖x − y + y‖ ≤ ‖x − y‖ + ‖y‖ .

Example. Euclideann-space:En ,

(Rn, ‖x‖ =

√∑ni=1 x2

i

)

Example. Euclideann-space with a weighted norm:(Rn, ‖x‖ =

√∑ni=1 wix2

i

)for wi > 0.

Example. A different norm forRn: (Rn, ‖x‖ = maxk |xk|)Proof.Clearly this norm is nonnegative, and is zero only if|xk| = 0 ∀k, i.e., if x = 0. And maxk |αxk| = |α|maxk |xk|.For the triangle inequality, we note thatmaxk |xk + yk| ≤ maxk [|xk| + |yk|] ≤ maxk |xk| + maxk |yk|.Those are all finite-dimensional examples.

Example. The vector space of continuous functions on the interval[a, b] with the norm‖f‖2 =√∫ b

a|f(t)|2 dt.

What if we replaced “continuous” with “square integrable?”(see footnote p. 32 regarding equivalence classes of functions that are equal a.e.)

Example. C[a, b]. The space of continuous functions on the interval[a, b] with the norm‖f‖∞ = maxa≤t≤b |f(t)|.Later we will see a reason why this “max” norm can be preferable.

Example. Space of realm × n matrices, with‖A‖ =√

tr(AAT ).

Problems involving spaces of matrices arise fairly frequently in systems analysis.

R2 and R

ForR2 there are three particularly important norms. Forx = (a, b) consider:

√a2 + b2 Euclidean orE2

|a| + |b| `1 or city blockmax{|a|, |b|} `∞

Are there others? Yes,e.g., weighted versions:√

w1a2 + w2b2, wk > 0.

What about R? One norm is‖x‖ = |x|. Are there others?If g(x) is a norm onR, then one condition is thatg(αx) = |α|g(x),∀α ∈ R.Thus, choosingx = 1 we must haveg(α) = |α|g(1), whereg(1) > 0 since1 6= 0.

Thus all norms onR have the form‖x‖ = w|x| for somew > 0.All such norms areequivalent, in a sense to be defined in HW, so there is no point in considering anything but the casew = 1.So we usually just speak ofR, rather thanE1 = (R, ‖·‖ = | · |).

3Probably it is possible to define normed spaces over other fields for which one can define a suitable|·| function that satisfies:(i) |α| = 0 ⇐⇒ α = 0, (ii) |αβ| = |α| |β|, (iii) |α + β| ≤ |α| + |β|. But would such spaces be useful?

4The term normed space should suffice because the presence of a vector space is implied in the axioms of a norm, since vector addition, scalar multiplication,and the zero vector are all part of those axioms.


2.10The `p andLp spaces

These normed spaces are ubiquitous in the engineering literature, and represent perhaps the most important examples of(infinitedimensional) normed vector spaces.

Definition. Let p ∈ [1,∞). The spacep consists of all sequences of scalarsa1, a2, . . . for which∑∞

i=1 |ai|p < ∞. The norm ofa vectorx ∈ `p is defined by

‖x‖p =

( ∞∑

i=1

|ai|p)1/p

< ∞.

Definition. The space∞ consists ofbounded sequences. The norm of a vectorx = {ai} in `∞ is defined by

‖x‖∞ = supi

|ai|.

Before referring to these spaces as normed spaces, we must confirm that each of the functionals defined above is indeed anorm.• ‖αx‖p = |α| ‖x‖p is trivial to verify• ‖x‖p > 0 unlessx = 0.• What about the triangle inequality? The holds due to theMinkowski inequality , which in turn follows from the Holder

inequality.

Theorem. (The Holder inequality)If p ∈ [1,∞) andq ∈ [1,∞) satisfy1/p + 1/q = 1, and ifx = (a1, a2, . . .) ∈ `p andy = (b1, b2, . . .) ∈ `q, then

∞∑

i=1

|aibi| ≤ ‖x‖p ‖y‖q .

Moreover, equality holds iff either of the following two conditions hold:• eitherx or y equal0, or

• bothx andy are nonzero and

(|ai|‖x‖p

)1/q

=

(|bi|‖y‖q

)1/p

, ∀i.

See errata: Luenberger’s equality condition omits the cases wherex or y equal0.

Proof.See text.

The special casep = q = 2 is particularly important, and is known as theCauchy-Schwarz inequality:

∑

i

|aibi| ≤√∑

i

|ai|2√∑

i

|bi|2 and hence

∣∣∣∣∣∑

i

aibi

∣∣∣∣∣ ≤√∑

i

|ai|2√∑

i

|bi|2.

Theorem. (The Minkowski inequality)If x,y ∈ `p, p ∈ [1,∞] then so isx + y, and‖x + y‖p ≤ ‖x‖p + ‖y‖p.For p ∈ [1,∞), equality holds iffx andy are linearly dependent.See errata: Luenberger’s condition for equality is incomplete since it omits the cases wherex or y equal0.

Proof.See text.(It uses the Holder inequality.)

At one point in the proof, we have the inequality(∑n

i=1 |ai + bi|p)1/p ≤ (∑n

i=1 |ai|p)1/p+ (∑n

i=1 |bi|p)1/p.

Taking the limit asn → ∞ on the RHS (which increases monotonically withn) yields(∑n

i=1 |ai + bi|p)1/p ≤ ‖x‖p + ‖y‖p .Then taking the limit asn → ∞ on the LHS yields‖x + y‖p ≤ ‖x‖p + ‖y‖p .This technique of first working with one side and then the other arises frequently.


The trouble with Riemann integration

TheLp space, defined below, involves Lebesgue integration. Why? Because the set of Riemann integrable functions is not as

general as we would like. For example, consider the functionf(t) =

{1, t ∈ Q

0, otherwise.This function is not Riemann integrable,

but it is Lebesgue integrable. (Its Lebesgue integral is zero since the function is nonzero only on a set of Lebesgue measure zero.)When a function is Riemann integrable, its Lebesgue integralwill equal its Riemann integral.

Considering Lebesgue integrable functions will more than general enough for any engineering problems.

The Lp space

Definition. Let p ∈ [1,∞). The spaceLp[a, b] consists of all real-valuedmeasurablefunctionsx on the interval[a, b] for which|x(t)|p is Lebesgue integrable. The norm on this space is defined as

‖x‖p =

(∫ b

a

|x(t)|p dt

)1/p

.

However, there is a subtle caveat with this space. There are functions that are nonzero on a set of measure zero for which‖x‖p = 0.So to consider this space to be a normed space, we must treat functions that are equal almost everywhere (a.e.) as being equivalent.

In other words, a vector inLp is really anequivalence classof measurable functions that are all equal almost everywhere.

This makes the definition ofL∞[a, b] a bit more subtle than∞. In particular, we cannot define‖x‖∞ to be simply the obviouschoice “supa≤t≤b |x(t)| ,” because that value will be different for different functions in the equivalence class. Instead, we define

‖x‖∞ = essential supremum of|x(t)| = infimumy(t)=x(t) a.e.

sup |y(t)| = ess sup|x(t)| .

See [4, p. 33] for an example.

If x ∈ Lp[a, b] andy ∈ Lq[a, b] with p, q > 1 and1/p + 1/q = 1, then the Holder inequality is∫ b

a|x(t)y(t)|dt ≤ ‖x‖p ‖y‖q .

Similarly, ‖x + y‖p ≤ ‖x‖p + ‖y‖p is theMinkowski inequality for Lp. The proofs are very similar.


Topological concepts

According to the Oxford English dictionary,topology is the “branch of mathematics concerned with those properties of figures andsurfaces which are independent of size and shape and are unchanged by any deformation that is continuous, neither creating newpoints nor fusing existing ones; hence, with those of abstract spaces that are invariant under homeomorphic transformations.”

Anyway, we have been systematically generalizing notions from geometry to more general settings, but one important concept wehave yet to generalize is that ofdistance.

The concept of distance is central to any discussion ofoptimization, since many optimization problems involve finding the element(within a set) that is theclosestto some given point outside that set.

Definition. If x andy are two points in a normed space(X , ‖·‖), then thedistance5 betweenx andy is defined by

d(x,y) = ‖x − y‖ .

More generally, ifS is a nonempty subset ofX , then thedistancebetween a pointx ∈ X and the setS is defined by

d(x, S) = infy∈S

‖x − y‖ .

Example. In R, d(1, (3, 4]) = 2, for the usual norm‖x‖ = |x|. (Picture)Note that there is no “closest point,” a complication that will require careful attention later.

Example. In R2, what is the distance between (0, 0) and {(a, b) : 2 ≤ a ≤ 3, 1 ≤ b ≤ 2}? (Trick question) ??

How would you define the distance between two sets S and T? ??

Properties of distanced. (We will use these later.)

Lemma 2.1 |d(x, S)− d(y, S)| ≤ ‖x − y‖ . (Picture)

Proof. d(x, S) = infz∈S ‖x − z‖ = infz∈S ‖(x − y) − (z − y)‖ ≥ infz∈S ‖z − y‖ − ‖x − y‖ = d(y, S)−‖x − y‖ . Nowrearrange. 2

Lemma 2.2 For any two subsetsU andV of a normed spaceX , d(U, V ) ≤ d(x, U) + d(x, V ), ∀x ∈ X . (Picture)

Proof.d(U, V ) = infu∈U d(u, V ) ≤ infu∈U d(x, V ) + ‖x − u‖ = d(x, V ) + d(x, U). 2

Lemma. Let (X , ‖·‖) be a normed space,M ⊆ X a subspace, andx ∈ X a point. Then∀α ∈ F : d(αx,M) = |α| d(x,M) .Proof.Trivial for α = 0.Forα 6= 0: d(αx,M) = infy∈M ‖αx − y‖ = infz∈M ‖αx − αz‖ = |α| infz∈M ‖x − z‖ = |α| d(x,M) (Picture) . 2

Preview of optimization

Many problems in optimization can be expressed as follows.Givenx in a normed space(X , ‖·‖), and a subsetS in X , find “the” vectors ∈ S that minimizes‖x − s‖.

What questions should we ask about such problems?• Is there any bests? I.e., does there exists? ∈ S s.t. ‖x − s?‖ = d(x, S)?• If so, iss? unique?• How iss? characterized? (Better yet would be an explicit formula fors?.)

One purpose of some of the material that follows is to answer these questions. (But be prepared for a challenge since some aspectsof these questions remain open problems!) We will return to these questions after introducing Hilbert spaces inCh. 3.

5This type of distance is a special case of the more general concept of a distance function that is called ametric if it satisfies:• d(x, x) = 0• d(x, y) > 0 if x 6= y

• d(x, y) = d(y, x)• d(x, y) ≤ d(x, z) + d(z, y) (triangle inequality )

Exercise.Verify that a metric is a more general concept than a norm,i.e., find a metricd(x, y) that is not of the formd(x, y) = ‖x− y‖ for any norm‖·‖.

Hint: a metric need not satisfy the scaling property.??


2.7Open sets

Looking ahead: in optimization we often use iterative algorithms that generate sequences, and we need to know when a sequenceconverges to a limit that belongs to the same set as the sequence itself. Such questions are related to whether a set is closed or not,so we need concepts of open and closed sets.

Definition. Theopen spherecentered atx of radiusε is defined byS(x, ε) , {y ∈ X : ‖x − y‖ < ε} .

Also called theopen ball, which is perhaps more descriptive since a sphere is often considered to be a surface, whereas a ballmight be more often considered to be a solid.

Example. Consider(X = C[0, 1], ‖·‖∞) andx = 0.ThenS(x, 2) =

{continuous functionsf : [0, 1] → R : maxt∈[0,1] |f(t)| < 2

}. (Picture of continuous functions between±2.)

Clearly this definition of a “sphere” generalizes the 3-space notion of a sphere!Now we need a definition for “inside,” another generalization.

Definition. Let P be a subset of a normed space(X , ‖·‖).A point p ∈ P is called aninterior point of P iff ∃ε > 0 s.t.S(p, ε) ⊆ P. (Picture in 2D with � in a generic set.)

Example. LetX ={(a, b) ∈ R2 : a = b

}(the line at 45◦ in the plane) with‖x‖ = |a| if x = (a, a).

Is x = (1, 1) an interior point of X , i.e., does there exist an open ball that is a subset of X? ??

Suppose we change to X = R2 with the usual Euclidean norm and S ={(a, b) ∈ R2 : a = b

}.

Is x = (1, 1) an interior point of S? ??

Which points within an open sphere are interior points? ??

Definition. The interior of a setP , denotedInt(P ), is the collection of all interior points ofP .

Int(P ) , {p ∈ P : p is an interior point ofP} = {p ∈ P : ∃ε > 0 s.t.S(p, ε) ⊆ P} .

Definition. A setP is calledopen iff P = Int(P ).

Remark.Int(P ) ⊆ P , so to show a setP is open we must showP ⊆ Int(P ), i.e., show thatx ∈ P =⇒ ∃ε > 0 s.t.S(x, ε) ⊂ P.

Examples of open sets.• ∅ andX are open sets• open spheres are open (exercise)• Int(Int(P )) = Int(P )• Int(P ) is open• P × Q is open inX × Y if P is open inX andQ is open inY (problem 2.16 p. 44)• (1, 2) × (0, 1) ⊂ E2 is open

Proposition. x ∈ Int(P ) ⇐⇒ d(x, P ) > 0. (Picture)

Proof.x ∈ Int(P ) =⇒ ∃ε > 0 s.t.S(x, ε) ⊆ P, sod(x, P ) = infy∈ eP ‖x − y‖ = infy/∈P ‖x − y‖ ≥ ε,

sincey /∈ P =⇒ y /∈ S(x, ε) =⇒ ‖y − x‖ ≥ ε.

We can prove theconverseby proving itscontrapositive6.Supposex /∈ Int(P ). Then∀ε > 0, S(x, ε) ∩ P 6= ∅, so∃y ∈ P s.t. ‖x − y‖ < ε. Thusd(x, P ) = inf

y∈ eP ‖x − y‖ = 0.

Alternative direct proof.δ = d(x, P ) > 0 =⇒ ‖z − x‖ ≥ δ > 0,∀z ∈ P . But y ∈ S(x, δ) =⇒ ‖y − x‖ < δ =⇒ y /∈ P =⇒y ∈ P. SoS(x, δ) ⊆ P and hencex ∈ Int(P ) . 2

Fact. Similarly: x ∈ Int(P)

⇐⇒ d(x, P ) > 0, since˜P = P.

6We have shown thatA =⇒ B and we want to show the converseB =⇒ A, the contrapositive of which is: not A=⇒ not B.


Closed sets

Definition. A point x ∈ X is called aclosure point (or acluster point or anadherent point) of a setP iff

∀ε > 0, ∃p ∈ P s.t. ‖x − p‖ < ε.

Fact. If P is nonempty, thenx is a closure point ofP iff d(x, P ) = 0.

Definition. The collection of all closure points of a setP is called theclosureof P and is denotedP .

Example. P = {1, 1/2, 1/3, . . . , 1/10} =⇒ P = P .

Example. P = [0, 1) =⇒ P = [0, 1].

Example. P = {1/n : n ∈ N} =⇒ P = P ∪ {0} .

Example. Q = R.

Properties.• P = {x ∈ X : d(x, P ) = 0} if P is nonempty. (IfP is empty, then so isP .)• P ⊆ P

• P = P

Definition. A setP is calledclosediff P = P .

Remark.SinceP ⊆ P , to show a setP is closed we must showP ⊆ P , i.e., if p is any closure point ofP , thenp ∈ P .

Examples of closed sets.• ∅ is closed (!)• X is closed (!)• {x} is closed• {y : d(x,y) ≤ 1} is closed

Proposition.• P is open=⇒ P is closed• P is closed=⇒ P is open

Proof.SupposeP is open.x ∈ P =⇒ ∃ε > 0 s.t.S(x, ε) ⊆ P =⇒ d(x, P ) ≥ ε > 0 =⇒ x /∈ P .

Thus the contrapositive isx ∈ P =⇒ x /∈ P i.e., P ⊆ P . HenceP is closed.

SupposeP is closed.x ∈ P =⇒ x /∈ P = P =⇒ ε , d(x, P ) > 0. ThusS(x, ε) ⊂ P sox ∈ Int(P)

.

ThusP ⊆ Int(P)

and we concludeP is open. (See Luenberger errata for p. 25.) 2

Remark.We could also state the proposition simply as “P is open⇐⇒ P is closed.”

Proposition.• The intersection of afinite numberof open sets is open.• The union of afinite numberof closed sets is closed.• The intersection of anarbitrary numberof closed sets is closed.• The union of anarbitrary numberof open sets is open.

Proof.Exercise.

Example. Consider the open intervalsSn = (−1/n, 1/n) ⊂ R, n = 1, 2, . . ..

Then∪∞n=1Sn = (−1, 1), which is open. And∩N

n=1Sn = SN , which is open.What is ∩∞n=1Sn? ??


Now we consider open and closedness in the context of the special types of sets we considered previously: convex sets andsubspaces.

Proposition. If C is a convex set in a normed space, thenC andInt(C) are convex.

Proof.see text

Open and closed subspaces

Fact. If M is a subspace ofX andM is open, thenM = X [3, p. 229].

Fact. If M is a subspace ofX , thenM is a subspace ofX [3, p. 229].

In general, subspaces are not necessarily closed in infinitedimensional normed spaces.

Example. The subspaceM of continuous functions inL2.Because the rect function is inL2 and is a closure point ofM but is not inM . (Picture) .

Example. The subspaceM of finite-length sequences in2.Because the infinite geometric series(1, 1/2, 1/4, 1/8, . . .) is in `2, and is a closure point ofM , but is not inM .

Caution! We have now seen one of our first examples of a situation where our intuition fromE3 does not generalize to generalnormed spaces! We have been thinking of subspaces as being like planes or hyperplanes. Yet even though planes are closed subsetsof E3, subspaces are not necessarily closed in general.

Remark. We had to use infinite-dimensional examples above because (as we will show soon), finite-dimensional subspaces areclosed.

Closed sets and distances

Lemma. In a normed space, letU andV be disjoint subsets. IfV is closed, thend(u, V ) > 0,∀u ∈ U .Proof.Pick anyu ∈ U and supposed(u, V ) = 0. Thenu ∈ V = V sinceV is closed.But u ∈ V contradicts the assumption thatU andV are disjoint. 2

Lemma 2.3 In a normed space,d(y, S) > 0 =⇒ d(y, S) > 0.

??

Bounded sets

Definition. A setS in a normed space(X , ‖·‖) is calledbounded iff ∃M < ∞ such that‖x‖ ≤ M, ∀x ∈ S.

Are closed sets bounded? ??

Bounded sets can be open, or closed, or neither.


Sequences

Definition. A sequenceis a set of vectors indexed by the natural numbersN, e.g., {xn} = {xn : n ∈ N}.Formally, a sequence is a mapping fromN to some vector spaceX .

Definition. If {xn} is a sequence, andn1 < n2 < · · · , then{xni} is called asubsequenceof {xn}.

Notation.{xn} ∈ X iff xn ∈ X , ∀n

2.8Convergence of sequences

Definition. In a normed space, we say a sequence of vectors{xn} convergesto a vectorx iff the sequence of real numbers‖xn − x‖ converges to zero, in which case we writexn → x or limn→∞ xn = x.In other words, considering the definition of convergence ofreal numbers:

xn → x ⇐⇒ ‖xn − x‖ → 0

⇐⇒ ∀ε > 0, ∃Nε < ∞ s.t.n ≥ Nε =⇒ ‖xn − x‖ < ε

⇐⇒ ∀ε > 0, ∃Nε < ∞ s.t.n ≥ Nε =⇒ xn ∈ S(x, ε).

Example. (Infinite dimensional, of course.) ConsiderL2[0, 1] with x(t) = 1 andxn(t) = 1 + t/n.

Thenxn → x because‖x − xn‖2 =√∫ 1

0|x(t) − xn(t)|2 dt =

√∫ 1

0|t/n|2 dt = 1√

3 n→ 0 asn → ∞.

Example. (Infinite dimensional, of course.) Consider`p with x = (1, 1/2, 1/3, . . .) andxn = (1, 1/2, . . . , 1/n, 0, 0, . . .).

Then‖xn − x‖p =(∑∞

k=n+1 1/kp)1/p

. That power series is convergent (and hence goes to zero asn → ∞) for p > 1, butdiverges forp = 1. So we can say “xn →x” in `2, for example, but we cannot say that in`1!

So all norms are not equivalent in general, unlike in finite-dimensional vector spaces.

Proposition. The limit of a convergent sequence is unique.

Proof.Supposexn → x andxn → y. Then∀n:

‖x − y‖ = ‖x − xn + xn − y‖ ≤ ‖x − xn‖ + ‖xn − y‖

which→ 0 asn → ∞. Thus‖x − y‖ = 0 sox − y = 0 sox = y. 2

Proposition. xn → x =⇒ ‖xn‖ → ‖x‖ .

Proof.As shown earlier,| ‖x‖ − ‖y‖ | ≤ ‖x − y‖ . Thus,|‖xn‖ − ‖x‖| ≤ ‖xn − x‖ → 0 if xn → x, so‖xn‖ → ‖x‖. 2

Proposition. If xn → x, thensupn ‖xn‖ < ∞, i.e., convergent sequences arebounded.

Proof.∃N1 s.t. ‖xn − x‖ < 1, ∀n > N1. Thus‖xn‖ = ‖xn − x + x‖ ≤ ‖xn − x‖ + ‖x‖ < 1 + ‖x‖ ,sosupn ‖xn‖ ≤ max {1 + ‖x‖ , ‖x1‖ , . . . , ‖xN1

‖} < ∞. (Picture) 2


The concept of a closed set is closely connected with limits of sequences.

Definition. If S is a subset ofX , we callx ∈ X a limit point of S iff there is a sequence of elements ofS that converges tox.

Proposition. x is a limit point ofS iff x ∈ S. Thus limit points and cluster points are equivalent!

Proof. If x is a limit point, then∃ {xn} ∈ S s.t.xn → x. Thusd(x, {xn}∞n=1) = 0, so7 d(x, S) = 0. Hencex ∈ S.Supposex ∈ S. Then∀ε > 0, ∃y ∈ S s.t. ‖x − y‖ < ε. Chooseε = 1/n and identifyxn with the correspondingy.Since‖xn − x‖ < 1/n, we seexn → x, sox is a limit point ofS. 2

Corollary . A set is closed iff it contains its limit points.

Summary• distance• open sphere• interior point & interior• open set = its interior• closure point:d(x, P ) = 0, & closure• closed set = its closure• bounded set• convergence sequences• limit point of a set (exists convergent sequence to it) = cluster point• closed sets contain their limit points

Series (A special kind of sequence.)

Definition. An infinite seriesof the form∑∞

i=1 xi is said toconvergeto x in a normed space iff the sequence of partial sumssn =

∑ni=1 xi converges tox, in which case we writex =

∑∞i=1 xi as short hand forx = limn→∞

∑ni=1 xi.

Caution. Sometimes as a lazy short hand one might write∑∞

i=1 1/i = ∞. Saying that∑∞

i=1 αi = ∞ is shorthand for saying∀M < ∞, ∃N ∈ N s.t.n > N =⇒∑n

i=1 αi > M .Since∞ /∈ R, it is truly an abuse of notation to write

∑∞n=1 1/n = ∞, because in the above definition of convergence of an infinite

series, it is implied that the limitx is an element ofX .

Example. In `2, considerx = (1, 1/2, 1/3, . . .) andxn = (0, . . . , 0, 1/n, 0, . . .) where1/n is in thenth element.Then

∑∞i=1 xi converges tox. (But it does not converge in1.)

7A ⊆ B =⇒ d(x, B) ≤ d(x, A). (Picture)


2.11Cauchy sequences

Often in convergence analysis it is easier to examine‖xn − xm‖ than‖xn − x?‖, especially if we have not yet shown that a limitx? even exists.

Definition. A sequence{xn} in a normed space is called aCauchy sequenceiff

‖xn − xm‖ → 0 asn,m → ∞.

In other words,∀ε > 0, ∃N > 0 s.t.n,m > N =⇒ ‖xn − xm‖ < ε.

Fact. In a normed space, every convergent sequence is a Cauchy sequence, since ifxn → x, then

‖xn − xm‖ ≤ ‖xn − x‖ + ‖xm − x‖ → 0.

Note the repeated use of the triangle inequality in proofs. In a normed space, it is about all we have to work with!

Theconverseof the above fact isnot true in general: Cauchy sequences need not converge in general normed spaces.This is a bit unfortunate since the converse is what we would really like to use usually!

Example. Consider the sequence of functionsfn(t) = 1 − e−nt in the normed space

(X = {f : [0, 1] → R : f continuous} , ‖f‖ = ‖f‖1 =

∫ 1

0

|f(t)|dt

), (Picture) .

‖fn − fm‖ =∫ 1

0|e−nt − e−mt|dt = |(1 − e−n)/n − (1 − e−m)/m| → 0 asn,m → ∞. So{fn} is Cauchy.

But the “apparent limit” offn is a step function, which is not continuous, and hence not an element ofX .

How can we “fix” this problem?• Broaden the vector spaceX toL1[0, 1] (i.e., drop the continuity requirement).• Replace the norm‖f‖1 with ‖f‖∞.

One can show for this example that‖fn − fm‖∞ does not approach zero as “n,m → ∞.”(For any fixedm, ‖fn − fm‖∞ → 1 asn → ∞.)

This “incompleteness” can also arise even in subspaces of`2.

Example. X = set of infinite sequence of reals with only finitely many nonzero terms:

X = {(a1, . . . , ak, 0, 0, . . .) : k ≥ 1, ai ∈ R}

with the`p norm‖x‖p . Now consider the sequence with elementsxn = (1, 1/2, 1/4, . . . , 1/2n, 0, 0, . . .).

Since‖xn − xm‖p =(∑max{n,m}

k=min{n+1,m+1}(

12k

)p)1/p

≤(

12p min{m,n}

11−1/2p

)1/p

→ 0 asn,m → ∞, {xn} is Cauchy. But there

is nox ∈ X to which{xn} converges.

In some sense, the problem is thatX has “holes” in it. Broadening the space is the natural solution.Is there an alternate norm that would make this vector space X complete? (I doubt it, can you show otherwise?)


2.11Banach spaces

Often we prefer to use normed spaces that are free of the pathologies described in the preceding examples, so we name them.

Definition. A normed space(X , ‖·‖) is calledcompleteiff every Cauchy sequence inX has a limit inX (and hence converges).

Definition. A complete normed space is called aBanach space.

Examples of Banach spaces

•(Rn, ‖·‖p

), for p ∈ [1,∞]

• C[a, b] with ‖f‖∞ = supt∈[a,b] |f(t)|• `p for p ∈ [1,∞] with usual‖x‖p

• Lp[a, b] for p ∈ [1,∞] with usual‖f‖p

Is finding a suitable Banach space usually difficult? Fortunately not, since every normed space has acompletion.

Theorem. If (X , ‖·‖X ) is a normed space, then∃(Y, ‖·‖Y

)that is a Banach space (called acompletion of X ) with

• X is a subspace ofY• X = Y• x ∈ X =⇒ ‖x‖X = ‖x‖Y .

Moreover,Y is essentially unique (i.e., all the completions ofX are isometric with one another [3, p. 121]).

Example. L1[0, 1] is the completion of(X = {f : [0, 1] → R : f continuous} , ‖f‖ = ‖f‖1 =

∫ 1

0|f(t)|dt

).

Example. Lp[a, b] is the completion ofRp[a, b] for p ∈ [1,∞].

Showing such results forLp requires measure theory.

Showing completeness ofC[a, b] uses the following fact that is a key result from Math 451.

Theorem. (R, | · |) is complete,i.e., if {αn} ⊂ R is Cauchy, then∃α ∈ R s.t.αn → α.

There exist bounded functions that are Lebesgue integrablebut not Riemann integrable,e.g. the indicator function on the rationals[3, p. 561]. And in factR1[a, b] is not complete [3, p. 564].

2.12Complete subsets

Definition. A subsetS of a normed space iscompleteiff every Cauchy sequence from the subset converges to a limit within S.

Example. Any finite set is complete. (The next theorems give more interesting cases.)

Theorem. In a normed space, any complete subset is closed.

Proof. If P ⊆ X is complete, then every Cauchy sequence inP has a limit inP .Thus all convergent sequences (which are of course Cauchy) have limits inP . ThusP is closed. 2

Theorem. In a Banach space, a subset is complete if and only if it is closed.

Proof.The “only if” direction follows from the preceding theorem.SupposeP is closed and{xn} is a Cauchy sequence inP (and hence inX ). SinceX is Banach,∃x ∈ X such thatxn → x.Hencex is a limit point ofP . SinceP is closed,x ∈ P . ThusP is complete. 2

Exercise. In a normed space, intersections of arbitrarily many complete subsets are complete.

What about unions of complete subsets? ?? ??


There is an asymmetry between the two preceding theorems. Weassumed a normed space to show complete set=⇒ closed set,but we assumed a Banach space to show closed set=⇒ complete set.

Is the stronger assumption (of a Banach space) truly necessary to show closed set =⇒ complete set? ??

Ok, but in some (incomplete) X is there a closed proper subset that is incomplete? ??

In the preceding theorem we assumed we are already working ina Banach space. What if we only have an ordinary normed space?Are there complete subsets of it? The next theorem shows thatthe answer can be yes, at least for finite-dimensional subspaces.

Recall that ifM is a subspace of a normed spaceX , thend(αx,M) = |α| d(x,M) for all x ∈ X andα ∈ F .

Theorem. Any finite-dimensional subspace of a normed space is complete (and hence closed).

Proof.By induction on the dimension of the subspace.

For a 1D subspace:M = {αe : α ∈ F}, wheree ∈ X is a fixed basis vector.Any Cauchy sequence{xn} in this subspace has elements of the formxn = αne. Since‖xn − xm‖ = |αn − αm| ‖e‖, thesequence of reals{αn} is also Cauchy and hence convergent (to some limitα) sinceE1 = (R, | · |) is complete.Thus,‖xn − αe‖ = |αn − α| ‖e‖ → 0, soxn → x = αe ∈ M . Hence, any 1D subspace of a normed space is complete.

Now assume the theorem is true for subspaces of dimensionN − 1.SupposeM is aN -dimensional subspace of a normed spaceX . We must show thatM is complete.

Let e1, . . . ,eN denote a basis forM . Fork = 1, . . . , N , defineMk = [{e1, . . . ,ek−1,ek+1, . . . ,eN}] andδk = d(ek,Mk) .

Claim: δk > 0.Supposeδk = 0. Then sinceMk is aN − 1 dimensional space it is complete by assumption and hence closed, sod(ek,Mk) = 0would implyek ∈ Mk, contradicting the linear independence of{ek}.

Suppose{xn} ∈ M is Cauchy. Eachxn has a (unique) representationxn =∑N

k=1 λnkek, so for eachk ∈ {1, . . . , N}:

‖xn − xm‖ =

∥∥∥∥∥

N∑

k=1

(λnk − λm

k )ek

∥∥∥∥∥ =

∥∥∥∥∥(λnk − λm

k )ek −N∑

j 6=k

(λmj − λn

j )ej

︸︷︷︸∈Mk

∥∥∥∥∥ ≥ d((λnk − λm

k )ek,Mk)

= |λnk − λm

k | d(ek,Mk) = |λnk − λm

k | δk.

Since‖xn − xm‖ → 0 for a Cauchy sequence, we see that|λnk − λm

k | → 0 sinceδk > 0. Thus{λnk} is Cauchy and hence (by the

completeness ofR) converges to some limitλk. Definingx =∑N

k=1 λkek ∈ M we find thatxn → x since

‖xn − x‖ =

∥∥∥∥∥

N∑

k=1

(λnk − λk)ek

∥∥∥∥∥ ≤N∑

k=1

|λnk − λk| ‖ek‖ → 0.

Thus any such Cauchy sequence inM converges to a limit inM , soM is complete. 2

Note this proof’s immediate use of the basis to represent anyx ∈ M . Proofs about finite-dimensional spaces often start this way.

Corollary .• Any finite-dimensional subspace of a normed space is closed.• Any finite-dimensional normed space is complete (and closed).

Exercise. [3, p. 218]. A (Hamel) basis for a Banach space is either finiteor uncountably infinite.


2.9Transformations

In systems theory, we analyze many systems and mathematicaloperations thattransformone signal into another.

Definition. LetX andY be two vector spaces (over a common fieldF), and letD be a subset ofX .A rule “T ” that assigns a single elementy ∈ Y to eachx ∈ D is called atransformation fromX toY with domain D.We writey = T (x) andT : X → Y. or T : D → Y.

Seep. 27or c1 review notes forone-to-oneandonto.

Definition. A transformation from a vector spaceX (over a scalar fieldF) intoF is called afunctional onX .

Example. A norm is a functional.

Linear transformations

Definition. A transformationT : X → Y (whereX andY are vector spaces over a common field) is calledlinear iff

T (αx1 + βx2) = α T (x1) +β T (x2), ∀x1,x2 ∈ X , ∀α, β ∈ F .

Definition. If T is a linear transformation fromX intoX itself, then we sayT is a linear operator.

However, the terminology distinguishing linear transformations from linear operators is not universal, and the two terms are oftenused interchangeably.

Example. LetF = R and letX = Y be the space of continuous functions on[0, 1].

Define the linear transformationT by: F = T (f) iff F (t) =∫ t

0f(τ) dτ .

Integration (with suitable limits) is a linear transformation.

Simple fact for linear transformations:• T (0) = 0. Proof.T (0) = T (00) = 0T (0) = 0. This is called the “zero in, zero out” property in linear systems theory.

Caution! (From Linear Systemsby T. Kailath.) By induction it follows thatT (∑n

i=1 αixi) =∑n

i=1 αi T (xi) for any finiten, butthe abovedoes notimply in general that linearity holds for infinite summations or integrals.Further assumptions about “smoothness” or “regularity” or“continuity” of T are needed for that.

This point is always glossed over in introductory signals and systems courses, where infinite sums (and integrals) are routinely“passed through” linear systems according to the superposition property, with no attempt to verify the validity of suchexchanges.


Continuity

To define such continuity, we restrict attention to normed spaces.Note that a transformation can be continuous for one pair of normed spaces but discontinuous for another pair,e.g.[3, p. 63,65].

Definition. A transformationT from anormed space(X , ‖·‖X ) into anormed space(Y, ‖·‖Y

)is calledcontinuousat x0 ∈ X

iff∀ε > 0, ∃δ = δ(x0, ε) > 0 s.t. ‖x − x0‖X < δ =⇒ ‖T (x)−T (x0)‖Y < ε.

Definition. If T is continuous at allx0 ∈ X , then we simply callT continuous.

Example. f : X → R defined byf(x) = d(x, S) is continuous for any setS in a normed spaceX , since|d(x, S) − d(y, S)| ≤‖x − y‖.

Definition. A transformationT from anormed space(X , ‖·‖X ) into anormed space(Y, ‖·‖Y

)is calleduniformly continuous

iff∀ε > 0, ∃δ = δ(ε) > 0 s.t. ‖x − z‖X < δ =⇒ ‖T (x)−T (z)‖Y < ε.

Hereδ depends only onε. Sometimesδ(ε) is called themodulus of continuity of T .

Example. Is ‖·‖ uniformly continuous? ??

We often want to exchange transformations and limits; the following proposition shows that continuity is the key condition.

Proposition. A transformationT from anormed space(X , ‖·‖X ) into anormed space(Y, ‖·‖Y

)is continuousatx ∈ X iff

xn → x =⇒ T (xn) → T (x) (for any such sequence{xn}), i.e., limn→∞

T (xn) = T(

limn→∞

xn

).

Proof.see text

Exercise. A linear transformation from a finite-dimensional normed space to any normed space is continuous.

Thus any “counter-example” showing a linear transformation that is not continuous will be infinite dimensional,e.g., the integratorsystem example [3, p.63].

Example. ConsiderX = Y = L2[R] and the linear operatorT corresponding to the integral transformation:y(t) =∫ t

−∞ x(τ) dτ .We show thatT is not continuous atx0 = 0. (And in fact is discontinuous everywhere!)

Consider the specific signals:

-t

6x(t)

a = 2/δ−a

b = (δ/2)3/2

-t

6y(t)

a−a

ab

Then‖x − 0‖ =√∫

|x(t)|2 dt =√

2ab2 = δ/√

2 < δ but‖T (x) − T (0)‖ = ‖y − 0‖ =√

23b2a3 =

√2/3.

An example of a continuous (linear) operator in an infinite dimensional vector space is the discrete-time convolution operator.

Example. ConsiderX = Y = `∞ and the transformationy = T (x) ⇐⇒ yn =∑∞

k=−∞ hn−kxk, whereh ∈ `1.Note that 1 corresponds to BIBO stability!Exercise. Show that‖T (x)‖∞ ≤ ‖h‖1 ‖x‖∞ . (This also ensures thatT (x) is well defined.)Thus, sinceT is linear, ifx,z ∈ `∞ then‖T (x) − T (z)‖∞ = ‖T (x − z)‖∞ ≤ ‖h‖1 ‖x − z‖∞ , soT is (uniformly) continuous,with δ(ε) = ε/ ‖h‖1. So BIBO LTI systems are uniformly continuous, and thus for such systems we can freely exchange limitsand sums. (And in fact we would need to make such exchanges to derive rigorously convolution properties like the commutativeproperty.)We will talk much more about suchbounded linear operatorsin Ch. 6.


2.13Compactness

Optimization is about maximizing a functional over some set, or more precisely, usually about finding the maximizer (within someset) of a functional. When does a functional achieve its maximum? We wish that the answer were “if the set is closed and bounded,”but unfortunately that is incorrect in general. The conceptof acompact sethelps answer this question in general.

Definition. A subsetK of a normed space(X , ‖·‖) is calledcompactor sequentially compactiff every sequence{xn} in K hassome subsequence{xni

} that converges to a limitx ∈ K.

(This is not quite the definition of compactness typically used first in a real analysis course, but it is more convenient for ourpurposes. One can first define a compact set in terms of set coverings, and then prove that a metric space is compact if and only ifit is sequentially compact,e.g., [2, p. 63].)

As usual, having defined a new concept, we now attempt to relate it to previously defined concepts.

Can a cone (other than {0}) be compact? ??

Exercise. In a normed space, arbitrary intersections of compact subsets are compact.

Exercise. What about unions? ?? ??

Lemma. If xn → x and{xni} is a subsequence of{xn}, thenlimi→∞ xni

= x.Proof.Pick anyε > 0. xn → x =⇒ ∃N s.t.n > N =⇒ ‖xn − x‖ < ε.ChooseI such thati > I =⇒ ni > N . Theni > I =⇒ ‖xni

− x‖ < ε. 2

Proposition. A compact subsetK of a normed space(X , ‖·‖) is complete, closed, (and bounded).

Proof.• Claim: K is complete, because Cauchy sequences will have limits in a compact setK.

Let {xn} be any Cauchy inK, then∀ε > 0, ∃M s.t.n,m > M =⇒ ‖xn − xm‖ < ε/2.SinceK is compact,∃ {ni} s.t.xni

→ x ∈ K, i.e., ∀ε > 0,∃I s.t. i > I =⇒ ‖xni− x‖ < ε/2.

For anyε > 0, let N = max{M,nI}.Forn > N andi > I we have‖xn − x‖ ≤ ‖xn − xni

‖ + ‖xni− x‖ < ε/2 + ε/2 = ε. Thusxn → x.

• To showK is closed, we show thatxn → x =⇒ x ∈ K for {xn} ∈ K.{xn} ∈ K =⇒ ∃xni

→ y ∈ K. But xn → x =⇒ xni→ x, sox = y ∈ K. ThusK is closed.

• Before we show boundedness, we need to prove the Weierstrasstheorem! 2

So is the reverse true? Are closed and bounded sets compact?The following theorem shows that in general the answer is “yes” in finite-dimensional normed spaces. (But not in general.)

Theorem. In a normed space(X , ‖·‖), the following are equivalent.(a) X is finite dimensional.(b) Every closed and bounded subset is compact.(c) The closed unit ballB1(0) = {x ∈ X : ‖x‖ ≤ 1} is compact [3, p. 269].

Proof.(a)=⇒ (b) is the Heine-Borel theorem (see Math 451...)(b) =⇒ (c) is obvious(c) =⇒ (a) will be a homework problem

This theorem illustrates why infinite-dimensional spaces are more challenging than finite-dimensional spaces. The fact that everyclosed and bounded subset is compact can be very useful in finite dimensional problems, and we do not have this tool in generalinfinite-dimensional cases.

By having a notion of compactness, we can make statements (such as the Weierstrass theorem below) that apply to both infinite andfinite-dimensional spaces, whereas those statements wouldnot hold if we merely assumed that the sets were closed and bounded.

Analogy: we write sup when we think max. Here, we write compact when we think “closed and bounded.”

Summary


In any normed space:• compact=⇒ complete=⇒ closed• compact=⇒ closed and bounded

In a Banach space: closed=⇒ completeIn a finite-dimensional normed space: closed and bounded=⇒ compact

Examples

As has been / will be shown, any compact set is closed, bounded, and complete, and that the reverse is true in finite-dimensionalspaces. But what about infinite-dimensional spaces?

Example.

Consider any of the Banach spaces`p, for p ∈ [1,∞] and the subsetB = {x ∈ X : ‖x‖ ≤ 1} . This is a closed set, and hence itis complete (sinceX is complete). Furthermore,B is clearly bounded.

We now proceed to exhibit a sequence{xn} in B that has no convergent subsequences.

Let xn denote the sequence in`p whosenth element is unity and all other elements are zero. Clearly{xn} ∈ B.Yet ‖xn − xm‖p = 1, so it is impossible for this{xn} to have any convergent subsequences since any such convergent subse-quence would need to be Cauchy which cannot occur when‖xn − xm‖p = 1,∀n,m ∈ N.

So the setB in `p is closed, bounded, and complete, but not compact.

One might begin to wonder then: are there any compact sets in infinite-dimensional spaces?

Example. Any finite set in an infinite-dimensional space is compact.

But that is not very interesting since the set{x1, . . . ,xn} is just a closed and bounded subset of the finite-dimensionalsubspace[x1, . . . ,xn], so of course it is compact.

Example. In any normed spaceX , suppose{xn} converges tox ∈ X . Then the set{x} ∪ ∪n {xn} is compact.

This still seems like a rather contrived and limited compactset.

Are there any interesting (e.g., nonempty)convexcompact sets in infinite-dimensional spaces?

Conjecture: the following (convex) set is (?) compact:{

x ∈ `1 :∞∑

i=1

2i|xi| ≤ 1

}. (2-1)

Note: the following set isnot compact {x ∈ `1 : ‖x‖1 =

∞∑

i=1

|xi| ≤ 1

},

as shown above, so the question is whether adding the “2i” part is enough of a change to make a compact set.

Extra credit (=30 homework points) to anyone who can show that (2-1) is or is not compact, or who can give a different exampleof a nontrivial convex, compact set in an infinite dimensional normed space.


Compactness, closedness, and distances

Lemma 2.4 In a normed space, closed subsets of compact subsets are themselves compact.

??

Lemma 2.5 In a normed space, any compact set contains the closures of all of its subsets.

??

Lemma 2.6 If U andV are compact, disjoint subsets of a normed space, thend(U, V ) > 0. (Picture)

Proof.Supposed(U, V ) = 0. Then there exists{xn} ∈ U and{yn} ∈ V such that‖xn − yn‖ → 0.SinceU is compact, there is a subsequence{xni

} that converges to somex ∈ U .Now ‖yni

− x‖ ≤ ‖yni− xni

‖ + ‖xni− x‖ → 0 asi → ∞, soyni

→ x ∈ U .But sinceV is compact, it is also closed, so the limit of{yni

} must lie inV , contradicting the disjointedness ofU andV . 2

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . .Would it suffice for U and V to be closed?

No. Consider the setsU ={(x, y) ∈ R2 : y > 0, x ≥ 1/y

}andV =

{(x, y) ∈ R2 : y < 0, x ≤ −1/y

}. (Picture)

These sets are closed and disjoint, yetd(U, V ) = 0.

Can you find a 1D example? ??. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . .Would it suffice for U and V to be closed and bounded?

(We need not look in finite-dimensional normed spaces for counter-examples since there all closed and bounded sets are compact.)

For a counter-example, we turn to the (incomplete) normed space{X = {finite-length real sequences} , ‖·‖p

}, where‖·‖p is the

usual`p norm. DefineU = ∪∞n=1xn andV = ∪∞

n=1yn, wherexn = (1, 1/2, . . . , 1/2n, 0, 0, . . .) andyn = xn + (1/n, 0, 0, . . .).ThenU andV are disjoint, closed (in this incompleteX ), and bounded (for anyp ∈ [1,∞]) yetd(U, V ) = 0.

So it seems that we do need a stronger condition than “closed and bounded” in Lemma 2.6.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . .Would “complete and bounded” suffice? Or do we need compactness?


Upper semicontinuous functions

Definition. A (real) functionalf defined on a normed space(X , ‖·‖) is calledupper semicontinuousatx ∈ X iff

∀ε > 0, ∃δ > 0 s.t.∀y ∈ X , ‖y − x‖ < δ =⇒ f(y) < f(x) + ε.

We callf lower semicontinuousif at x iff −f is upper semicontinuous atx.Fact. f is continuous iff f is both upper and lower semicontinuous.

Equivalently, one can show thatf is u.s.c. atx iff f(x) ≥ lim supy→x

f(y).

S = lim supy→x

f(y) iff∀ε > 0, ∃δ > 0 s.t. ‖y − x‖ < δ =⇒ f(y) < S + ε, and∀ε > 0, ∀δ > 0, ∃y s.t. ‖y − x‖ < δ andf(y) > S − ε.

Loosely speaking, it is like a least upper bound “in the limit.”

Example. Consider the following “almost piecewise continuous” real-valued function.

20

2 4 6

10

f(x)

x

What is lim supy→2 f(y)? We get the biggest limit approaching from the left, solim sup

y→2 f(y) = 20. Sof is u.s.c. atx = 2.Similarly, lim sup

y→4 f(y) = 20, sof is not u.s.c. atx = 4. But lim supy→6 f(y) = 20, sof is u.s.c. atx = 6.

This function is u.s.c. everywhere except?? and lower semicontinuous everywhere except??

Does this function achieve a maximum on [3,5]? ?? What about on [1,3]? ?? How about on [0,1)? ??

Theorem. (Weierstrass)An upper semicontinuous (real) functionalf on a compact subsetK of a normed space(X , ‖·‖)(i) is bounded onK, and (ii) achieves a maximum onK.

Proof.Let M = supx∈K f(x) =

{inf {g ∈ R : g ≥ f(x), ∀x ∈ K} , if f is bounded above onK∞, otherwise.

(At this point we do not know ifM is finite or not.)By definition of supremum,∃ a sequence{xn} ∈ K suchf(xn) → M . (This is true even ifM = ∞.)However, the definition of supremum alone does not ensure that xn converges.

SinceK is compact,∃ a convergent subsequencexni→ x? ∈ K, for somex?.

Since subsequences of convergent sequences have the same limit, limi→∞ f(xni) = M , considering{f(xn)} as a sequence inR.

Sincef is upper semicontinuous,f(x?) ≥ lim supi→∞ f(xni) = M .

Sincef(x) is real and hence finite,f(x?) ≥ M impliesM must be finite.... f is bounded onK.

On the other hand, by definition of supremum, sincex? ∈ K, M ≥ f(x?).So we concludef(x?) = M , meaning thatx? achieves the maximum off onK. 2

Corollary . A real-valued,continuous functionalf on a compact subsetK of a normed space(X , ‖·‖) achieves its maximumand minimum onK.

Example. On a normed space(X , ‖·‖), the functionf : X → R defined byf(x) = ‖x‖ is continuous.Proof.Recall| ‖x‖ − ‖y‖ | ≤ ‖x − y‖, sox → y =⇒ ‖x − y‖ → 0 =⇒ |f(x) − f(y)| → 0.

An inf. dim. example like splines would be nice here...

Corollary (to Weierstrass theorem)A compact subsetK of a normed space is (complete, closed, and) bounded.

Proof.Since‖·‖ is continuous, by the Weierstrass theorem∃y ∈ K s.t.M , ‖y‖ = supx∈K ‖x‖.

Hence‖x‖ ≤ M < ∞, ∀x ∈ K, i.e., K is bounded. 2


2.14Quotient Spaces

skip

2.15Denseness

One last topological concept.

Definition. A subsetD of a normed space(X , ‖·‖) is calleddenseiff any of the following equivalent conditions hold.• ∀x ∈ X and∀ε > 0, ∃y ∈ D s.t. ‖x − y‖ < ε, i.e., d(x,D) = 0.• D = X .• ∀x ∈ X , ∃ {yn} ∈ D s.t.yn → x

Example. The set of rational numbers is dense in the real line.

Separability

Definition. A normed spaceX is calledseparableiff it contains acountabledense set,i.e., D =⋃∞

n=1 {yn}.

Example. Euclidean spaceEn is separable. The collection of vectorsx = (a1, . . . , an) with rational components is countable anddense inEn.

Example. `p is separable forp ∈ [1,∞)If Dn = {(r1, . . . , rn, 0, 0, . . .) : rk ∈ Q}, thenD = ∪∞

n=1Dn is dense in p.See text for proof.

Example. Lp is separable forp ∈ [1,∞)

If Dn = {∑nk=1 rk1Ik

(t) : rk ∈ Q, Ik = (ak, bk), ak, bk ∈ Q} for n ∈ N, thenD = ∪∞n=1Dn is dense inLp.

Alternatively, if Dn = {f : [a, b] → R : f is piecewise linear withn ∈ N breakpoints atrk ∈ Q andf(rk) ∈ Q},thenD = ∪∞

n=1Dn is dense inLp.

Example. C[a, b] is separable. The set of all polynomials with rational coefficients is countable and dense inC[a, b].If Dn = {r0 + r1t + · · · + rntn : rk ∈ Q}, thenD = ∪∞

n=1Dn is dense inC[a, b].

Example. `∞ andL∞ are not separable (Problem 2.21)

Schauder basis

Definition. In a normed space,{xn} ∈ X is aSchauder basisfor X iff for eachx ∈ X , there exists a unique sequence{λn}such thatx =

∑∞n=1 λnxn [5] [2, p. 98].

Theorem. If a normed space has a Schauder basis, then it is separable [2, p. 100].

What about the converse?The famous Banach conjecture that every separable Banach space has a Schauder basis was shown to be incorrect by Elfon in1973 [2, p. 100]. This could be considered surprising since separable, complete, normed spaces should be about as “nice as theycome.”. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . .So is the concept of separability of limited use to use? No, thanks the following key result.

Fact. A Hilbert space has a countable orthonormal basis iff it is separable [3, p. 314].(Any countable orthonormal basis is a Schauder basis.)


Summary

Geometrical concepts and their generalizations.

point x ∈ Xline {αx : α ∈ R}plane {∑i αixi : αi ∈ R}cone x ∈ C =⇒ αx ∈ C for α ≥ 0length ‖x‖sphere {x ∈ X : ‖x‖ < ε}distance ‖x − y‖coordinate system basis

Hierarchy of spaces: vector space⊃ normed space⊃ Banach space

Some principal results• A subset of a Banach space is complete iff it is closed.• Any finite-dimensional subspace of a normed space is complete (and hence closed).• A real-valued, continuous functional on a compact subset ofa normed space achieves its maximum and minimum on that subset.• Any closed and bounded subset of a finite dimensional normed space is compact.

c© J. Fessler, October 4, 2004, 12:44 (student version)

Bibliography

[1] P. Enflo. A counterexample to the approximation problem in Banach spaces.Acta Math, 130:309–17, 1973.

[2] I. J. Maddox.Elements of functional analysis. Cambridge, 2 edition, 1988.

[3] A. W. Naylor and G. R. Sell.Linear operator theory in engineering and science. Springer-Verlag, New York, 2 edition, 1982.

[4] D. G. Luenberger.Optimization by vector space methods. Wiley, New York, 1969.

[5] J. Schauder. Zur theorie stetiger abbildungen in funktionenrumen.Math. Zeitsch., 26:47–65, 1927.

[6] P. P. Vaidyanathan. Generalizations of the sampling theorem: Seven decades after Nyquist.IEEE Tr. Circ. Sys. I, Fundamentaltheory and applications, 48(9):1094–109, September 2001.

[7] F. Deutsch. The convexity of Chebyshev sets in Hilbert space. In A. Yanushauskas Th. M. Rassias, H. M. Srivastava, editor,Topics in polynomials of one and several variables and theirapplications, pages 143–50. World Sci. Publishing, River Edge,NJ, 1993.

[8] M. Jiang. On Johnson’s example of a nonconvex Chebyshev set. J. Approx. Theory, 74(2):152–8, August 1993.

[9] P. L. Combettes and H. J. Trussell. Method of successive projections for finding a common point of sets in metric spaces. J.Optim. Theory Appl., 67(3):487–507, December 1990.

linear spacesweb.eecs.umich.edu/~fessler/course/600/l/l02.pdf · 2004. 10. 4. · vector spaces in...

Documents