Discrete Structures & Algorithms

Functions & Asymptotic Complexity

2

To understand how to perform computations efficiently, we need to know– How to count!– How to sum a sequence of steps.– Reason about the complexity/efficiency of an

algorithm.– Prove the correctness of our analysis.

• Logic• Methods of inference• Proof strategies• Induction• …

– (and eventually) How to create better algorithms.

3

Functions

• We can represent the number of steps needed to achieve a goal as a function of the size of the problem.

• Next: A quick refresher on functions– You should know (most of this) from other

math/engineering courses.

4

Functions

Definition: A function f : A B is a subset of AxB where a A, b B unique and <a,b>

f.(A function is a mapping of elements from one set to another.)

5

Functions

Definition: a function f : A B is a subset of AxB where a A, b B unique and <a,b> f.

A

B

A

B

A point!

6

Functions

A = {Michael, Tito, Janet, Cindy, Bobby}B = {Katherine Evans, Carol Brady, Mother

Teresa}

Let f: A B be defined as f(a) = mother(a).

Michael Tito Janet Cindy Bobby

Katherine

Carol

Mother Teresa

7

Image and preimage

For any set S A, image(S) = {b : a S, f(a) = b}

So, image({Michael, Tito}) = {Katherine Evans} image(A) = B - {Mother Teresa}

image(S) = f(S)

Michael Tito

Janet Cindy Bobby

Katherine Evans

Carol Brady

Mother Teresa

8

Image and preimage

For any S B, preimage(S) = {a: b S, f(a) = b}

So, preimage({Carol Brady}) = {Cindy, Bobby} preimage(B) = A

preimage(S) = f-

1(S)

Michael Tito Janet Cindy Bobby

Katherine Evans

Carol Brady

Mother Teresa

9

Image and preimage

What is image(preimage(S))?a) Sb) { }c) subset of Sd) superset of S e) who knows?

S

Michael Tito Janet Cindy Bobby

Katherine Evans

Carol Brady

Mother Teresa

10

Functions and their properties

• You should be familiar with these properties (past classes)

or

• Functions derive many of their properties from sets (they are defined on sets)

• As you already know much about sets -> should be able to derive these properties

11

Functions and their properties

f(A B) f(A) f(B)?

f(A B) = {y : x (A B), f(x) = y}

Choose an arbitrary b f(A B), and show that it must also be an element of f(A) f(B).

So, x (A B) such that f(x) = y.

If x A (it is), then f(x) = y f(A).

If x B (it is), then f(x) = y f(B).

y f(A), and y f(B), so b f(A) f(B).

12

Injections

A function f: A B is one-to-one (injective, an injection) if a,b,c, (f(a) = b f(c) = b) a = c

Not one-to-one

Every b B has at most 1 preimage.

Michael Tito Janet Cindy Bobby

Katherine Evans

Carol Brady

Mother Teresa

13

Surjections

A function f: A B is surjective if b B, a A f(a) = b

Not surjective

Every b B has at least 1 preimage.

Michael Tito Janet Cindy Bobby

Katherine Evans

Carol Brady

Mother Teresa

14

Bijections

A function f: A B is bijective if it is both one-to-one (injective) and surjective.

Isaak Bri

Lynette

Aidan Evan

Cinda Dee Deb Katrina Dawn

Every b B has exactly 1 preimage.

An important implication of this

characteristic:The preimage (f-1)

is a function!

Isaak Bri

Lynette

Aidan Evan

Cinda Dee Deb Katrina Dawn

15

Examples

Suppose f: R+ R+, f(x) = x2.

Is f one-to-one?

Is f surjective?

Is f bijective?

yesyes

yes

16

Examples

Suppose f: R R+, f(x) = x2.

Is f one-to-one?

Is f surjective?

Is f bijective?

noyes

no

17

Examples

Suppose f: R R, f(x) = x2.

Is f one-to-one?

Is f surjective?

Is f bijective?

no

no

no

18

Composition of functions

Let g:XY, and f:YZ be functions. Then the composition of f and g is:

(f o g)(x) = f(g(x))

g f

Asymptotic complexity

20

Asymptotic complexity

• Goal: Estimate properties (e.g., running time) of an algorithm as a function of input size n for large n.

• Expressed using only the highest-order term in the expression for the exact running time.– Instead of exact running time, say (n2).

• Describes behavior of function in the limit.

21

Asymptotic notation

• , O, , o, • Defined for functions over the natural numbers.

– Example: f(n) (n2).– Describes how f(n) grows in comparison to n2.

(n2) define a set of functions; in practice used to compare two function sizes.

• The notations describes the rate-of-growth for a set of functions.

22

-notation (big-Theta)

(g(n)) = {f(n) : positive constants c1, c2, n0, such that n n0,

we have c1g(n) f(n) c2g(n) }

For function g(n), we define (g(n)), big-Theta of n, as the set:

g(n) is an asymptotically tight bound for f(n).

Intuitively: Set of all functions that have the same rate of growth as g(n).

23

-notationFor function g(n), we define (g(n)), big-Theta of n, as the set:

g(n) is an asymptotically tight bound for f(n).

Technically, f(n) (g(n)).Older usage, f(n) = (g(n)).I’ll accept either…

(g(n)) = {f(n) : positive constants c1, c2, n0, such that n n0,

we have c1g(n) f(n) c2g(n) }

24

-notationFor function g(n), we define (g(n)), big-Theta of n, as the set:

If f(n) (g(n))

Then ? (0, ∞, or a constant )

(g(n)) = {f(n) : positive constants c1, c2, n0, such that n n0,

we have c1g(n) f(n) c2g(n) }

)(

)(lim

ng

nfn

25

Example

For f(n) = 10n2 - 3n (n2)• What constants for n0, c1, and c2 will work?• Make c1 a little smaller than the leading

coefficient, and c2 a little bigger.• To compare orders of growth, look at the

leading term.

Exercise: Prove that n2/2-3n (n2)

(g(n)) = {f(n) : positive constants c1, c2, and n0,

such that n n0, c1g(n) f(n) c2g(n)}

26

Example

• Does n5 (n4)?• Does 300n3 (n4)?• How about 22n (2n)?

• Does lg(n!) (n lg n)?– Proof using Stirling’s approximation (in the text)

for lg(n!).

(g(n)) = {f(n) : positive constants c1, c2, and n0, such that n n0, c1g(n) f(n)

c2g(n)}

27

O-notation

O(g(n)) = {f(n):

positive constants c and n0, such that n n0,

we have 0 f(n) cg(n) }

For function g(n), we define O(g(n)), big-O of n, as the set:

O(g(n)) is an asymptotic upper bound for f(n).

Intuitively: Set of all functions f whose rate of growth is the same as or lower than that of g(n).

f(n) = (g(n)) f(n) = O(g(n)).

(g(n)) O(g(n)).

?)(

)(lim

ng

nfn

28

Examples

• Any linear function an + b is in O(n2). How?

• Show that 3n3 O(n4) for appropriate c and n0.

O(g(n)) = {f(n) : positive constants c and n0, such that n n0, we have 0 f(n) cg(n) }

29

-notation (omega)

(g(n)) is an asymptotic lower bound for f(n).

Intuitively: Set of all functions whose rate of growth is the same as or higher than that of g(n).

f(n) = (g(n)) f(n) = (g(n)).

(g(n)) (g(n)).

(g(n)) = {f(n) : positive constants c and n0, such that n n0,

we have 0 cg(n) f(n)}

For function g(n), we define (g(n)), big-Omega of n, as the set:

?)(

)(lim

ng

nfn

30

Example

n (lg n). Choose c and n0.

(g(n)) = {f(n) : positive constants c and n0, such that n n0, we have 0 cg(n) f(n)}

31

Relations between , O,

32

Relations between , , O

(g(n)) = O(g(n)) (g(n))

• In practice, asymptotically tight bounds are obtained from asymptotic upper and lower bounds.

Theorem: For any two functions g(n) and f(n),

f(n) (g(n)) iff

f(n) O(g(n)) and f(n) (g(n)).

33

Example• Insertion sort takes (n2) in the worst case, so sorting

(as a problem) is O(n2). Why?

• Any sort algorithm must look at each item, so sorting is (n).

• In fact, using (e.g.) merge sort, sorting is (n lg n) in the worst case.

– No comparison sort to do better in the worst case. [We may not see a proof of this result in this course.]

34

Asymptotic notation in equations

• Can use asymptotic notation in equations to replace expressions containing lower-order terms.

• For example,4n3 + 3n2 + 2n + 1 = 4n3 + 3n2 + (n) = 4n3 + (n2) = (n3). How do we interpret

this?

• In equations, (f(n)) always stands for an anonymous function g(n) (f(n))– In the example above, (n2) stands for

3n2 + 2n + 1.

35

o-notation

f(n) becomes insignificant relative to g(n) as n approaches infinity:

lim [f(n) / g(n)] = 0.

n

g(n) is an upper bound for f(n) that is not asymptotically tight.

Observe the difference in this definition from previous ones. Why?

o(g(n)) = {f(n): c > 0, n0 > 0 such that n n0, we have 0 f(n) < cg(n)}.

For a given function g(n), the set little-o:

36

(g(n)) = {f(n): c > 0, n0 > 0 such that n n0, we have 0 cg(n) < f(n)}.

–notation (little-omega)

f(n) becomes arbitrarily large relative to g(n) as n approaches infinity:

lim [f(n) / g(n)] = .

n

g(n) is a lower bound for f(n) that is not asymptotically tight.

For a given function g(n), the set little-omega:

37

Limits• lim [f(n) / g(n)] = 0 f(n) (g(n)) n

• lim [f(n) / g(n)] < f(n) (g(n)) n

• 0 < lim [f(n) / g(n)] < f(n) (g(n)) n

• 0 < lim [f(n) / g(n)] f(n) (g(n)) n

• lim [f(n) / g(n)] = f(n) (g(n)) n

• lim [f(n) / g(n)] undefined can not say n

38

Relationships

• Subset relations between order-of-growth sets.

RR( f )O( f )

( f ) ( f )o( f )• f

39

Properties

• Transitivity– f(n) = (g(n)) & g(n) = (h(n)) f(n) = (h(n))– f(n) = O(g(n)) & g(n) = O(h(n)) f(n) = O(h(n))– f(n) = (g(n)) & g(n) = (h(n)) f(n) = (h(n))– f(n) = o (g(n)) & g(n) = o (h(n)) f(n) = o (h(n))– f(n) = (g(n)) & g(n) = (h(n)) f(n) = (h(n))

• Reflexivity– f(n) = (f(n))– f(n) = O(f(n))– f(n) = (f(n))

40

Properties• Symmetry

f(n) = (g(n)) iff g(n) = (f(n))

• Complementarity f(n) = O(g(n)) iff g(n) = (f(n)) f(n) = o(g(n)) iff g(n) = ((f(n))

Common functions

43

Exponentials

• Useful (and elementary) Identities:

• Exponentials and polynomials

)(

0lim

nb

n

b

n

aon

an

nmnm

mnnm

aaa

aa

aa

)(

11

44

Logarithms

x = logba is the exponent for a = bx.

Natural log: ln a = logea

Binary log: lg a = log2a

lg2a = (lg a)2

lg lg a = lg (lg a)

ac

ab

bb

c

cb

bn

b

ccc

a

bb

b

ca

ba

aa

b

aa

ana

baab

ba

loglog

log

log

1log

log)/1(log

log

loglog

loglog

loglog)(log

45

Logarithms and exponentials – Bases

• If the base of a logarithm is changed from one constant to another, the value is altered by a constant factor.– Example: log2 n = log10 n * log210

– Base of logarithm is not an issue in asymptotic notation.

• Exponentials with different bases differ by a exponential factor (not a constant factor).– Example: 2n = (2/3)n *3n.

46

ExerciseExpress functions in A in asymptotic notation using functions in B.

A B

5n2 + 100n 3n2 + 2

A (n2), n2 (B) A (B)

log3(n2) log2(n3)

logba = logca / logcb; A = 2lgn / lg3, B = 3lgn, A/B =2/(3lg3)

nlg4 3lg n

alog b = blog a; B =3lg n=nlg 3; A/B =nlg(4/3) as n

lg2n n1/2

lim ( lga n / nb ) = 0 (here a = 2 and b = 1/2) A (B) n

A (B)

A (B)

A (B)

A (B)

47

Wrap-up

• What are the different asymptotic bounds on functions?

• How are the asymptotic bounds related?• Asymptotic bounds and algorithmic efficiency

48

Procedure mergesort (L=a1, a2, ….an)

if n>1 then m:= n/2

L1 = a1, a2, …. am

L2 = am+1, …. an L = merge (mergesort(L1), mergesort (L2))

Recursive proofCorrectness Complexity