241-303 discrete maths: runtime/4 1 discrete maths objective – –to describe the big-oh notation...

241-303 Discrete Maths: RunTime/4 1

Discrete Maths

• Objective– to describe the Big-Oh notation for estimating the

running time of programs

241-303, Semester 1 2014-2015

4. Running Time of Programs


Overview

1. Running Time

2. Big-Oh and Approximate Running Time

3. Big-Oh for Programs

4. Analyzing Function Calls

5. Analyzing Recursive Functions

6. Towers of Hanoi

7. Further Information


1. Running Time

• What is the running time of this program?

void main(){ int i, n; scanf("%d", &n); for(i=0; i<n; i++) printf("%d"\n", i);}

continued


• There is no single answer!– the running time depends on the size of the n va

lue

• Instead of a time answer in seconds, we want a time answer which is related to the size of the input.

continued


• For example:– programTime(n) = constant * n– this means that as n gets bigger, so does the pro

gram time– running time is linearly related to the input

size of n

runningtime

constant * n


Running Time Theory

• A program/algorithm has a running time T(n)– n is some measure of the input size

• T(n) is the largest amount of time the program takes on any input of size n

• Time units are left unspecified.

continued


• A typical result is: – T(n) = c*n, where c is some constant

• but often we just ignore c

– this means the program has linear running time

• T(n) values for different programs can be used to compare their relative running times– selection sort: T(n) = n2

– merge sort: T(n) = n log n– so, merge sort is “better” for larger n sizes


1.1. Different Kinds of Running Time

• Usually T(n) is the worst-case running time– the maximum running time on any input of size n

• Tavg(n) is the average running time of the program over all inputs of size n– more realistic– very hard to calculate– not considered by us


1.2. T(n) Example

• Loop fragment for finding the index of the smallest value in A[] array of size n:

(2) small = 0;(3) for(j = 1; j < n; j++)(4) if (A[j] < A[small])(5) small = j;

• Count each assignment and test as 1 “time unit”.


Calculation• The for loop executes n-1 times

– each loop carries out (in the worse case) 4 ops• test of j < n, if test, small assign, j increment

– total loop time = 4(n-1)– plus 3 ops at start and end

• small assign (line 2), init of j (line 3), final j < n test

• Total time T(n) = 4(n-1) + 3 = 4n -1

– running time is linear with the size of the array


1.3. Comparing Different T()’s

• If input size < 50, program B is faster.• But for large n’s, which are more common in real

code, program B gets worse and worse.

5000

10000

15000

20000

T(n)value

20 40 60 80 100

input size n

Ta(n) = 100n

Tb(n) = 2n2


1.4. Common Growth Formulae & Names

• Formula (n = input size) Name

n linearn2 quadraticn3 cubicnm polynomial, e.g. n10

mn ( m >= 2)exponential, e.g. 5n

n! factorial1 constantlog n logarithmicn log nlog log n


1.5. Execution Times

• 3 9 50 100 1000 106

n 3 9 50 100 1ms 1secn2 9 81 2.5ms 10ms 1sec 12 daysn3 27 729 125ms 1sec 16.7 min 31,710yr2n 8 512 36yr 4*1016 yr 3*10287 yr3*10301016yrlog n 2 3 6 7 10 20

n (no. of instructions)

grow

th f

orm

ula

T()

if n is 50, you willwait 36 years for an answer!

Assume 1 instruction takes 1 microsec (10-6 secs) to execute.How long will n instructions take?


Notes• Logarithmic running times are best.

• Polynomial running times are acceptable, if the power isn’t too big– e.g. n2 is ok, n100 is terrible

• Exponential times mean sloooooooow code.– some size problems may take longer to finish than th

e lifetime of the universe!


1.6. Why use T(n)?

• T() can guide our choice of which algorithm to implement, or program to use– e.g. selection sort or merge sort?

• T() helps us look for better algorithms in our own code, without expensive implementation, testing, and measurement.


Arguments against T(n)

• Algorithms often perform much better on average than the worst case used in T()– quicksort is n log n on a “random” array, but n2 i

n the worse case

– but for most algorithms, the worst case is a good predictor of its running time

– average case analyses can be done, but they are harder mathematically

continued


• Some people say:– “Who cares about running time? In a few years, machine

s will be so fast that even bad algorithms will be fast.”

– History shows this argument to be wrong. As machines get faster, problem sizes get bigger.

– Most interesting problems (e.g. computer vision, natural language processing) always require more resources• fast algorithms will always be needed

continued


• Some people say:– "Benchmarking (running programs on a standar

d set of test cases) is easier."

– This is sometimes true, but the benchmarks only give numbers for that particular machine, OS, compiler, computer language.


2. Big-Oh and Approximate Running Time

• Big-Oh mathematical notation simplifies the process of estimating the running time of programs– it uses T(n), but ignores constant factors which

depend on compiler/machine behaviour

continued


• The Big-Oh value specifies running time independent of:– machine architecture

• e.g. don’t consider the running speed of individual machine operations

– machine load (usage)• e.g. time delays due to other users

– compiler design effects • e.g. gcc versus Borland C


Example• In the code fragment example on slide 9, we assu

med that assigment and testing takes 1 “time unit”. This means:

T(n) = 4n -1

• The Big-Oh value, O(), uses the T(n) value but ignores constants (which will actually vary from machine to machine). This means:

T(n) is O(n)we say "T(n) is order n"


More Examples

• T(n) value Big Oh value: O()– 10n2+ 50n+100 O(n2)– (n+1)2 O(n2)– n10 O(2n)– 5n3 + 1 O(n3)

• These simplifications have a mathematical reason, which is explained in section 2.2.

hard tounderstand


2.1. Is Big-Oh Useful?

• O() ignores constant factors, which means it is a more reliable measure across platforms/compilers.

• It can be compared with Big-Oh values for other algorithms.– i.e. linear is better than polynomial and expone

ntial, but worse than logarithmic


2.2. Definition of Big-Oh

• The connection between T() and O() is:– when T(n) is O( f(n) ), it means that f(n) is the m

ost important thing in T() when n is large

• More formally, for some integer n0 and constant c > 0– T(n) is O( f(n) ),

if for all integers n >= n0, T(n) <= c*f(n)

continued


• n0 and c are called witnesses to the relationship:

T(n) is O( f(n) )


Example 1

• T(n) = 10n2 + 50n + 100– which allows that T(n) is O(n2)

• Why?– Witnesses: n0 = 1, c = 160

– then T(n) <= c*f(n), n >= 1so 10n2 + 50n + 100 <= 160 n2

since10n2 + 50n + 100 <=10n2 + 50n2 + 100n2 <= 160 n2

Informally, the n2 part isthe most important thing in

the T() function


Example 2

• T(n) = (n+1)2

– which allows that T(n) is O(n2)


– then T(n) <= c*f(n), n >= 1so (n+1)2 <= 4n2

sincen2 + 2n + 1 <=n2 + 2n2 + n2 <= 4n2


Example 3

• T(n) = n10

– which allows that T(n) is O(2n)


– then T(n) <= c*f(n), n >= 64so n10 <= 2n

since10*log2 n <= n (by taking log2s)

which is true when n >= 64(10*log2 64 == 10*6; 60 <= 64)


2.4. Some Observations about O()

• When choosing an O() approximation to T(), remember that:– constant factors do not matter

• e.g. T(n) = (n+1)2 is O(n2)

– low-order terms do not matter• e.g. T(n) = 10n2 + 50n + 100 is O(n2)

– there are many possible witnesses


2.5. Simplifying O() Expressions

• Inside an O() expressions, always drop constant factors and low-order terms.

• For example:– T(n) = 3n5+ 10n4 + n– T(n) is O(3n5)– but, T(n) is O(n5) is simpler and tighter


3. Big-Oh for Programs

• First decide on a size measure for the data in the program. This will become the n.

• Data Type Possible Size Measure

integer its valuestring its lengtharray its length


3.1. Building a Big-Oh Result

• The Big-Oh value for a program is built up inductively by:– 1) Calculate the Big-Oh’s for all the simple stat

ements in the program• e.g. assignment, arithmetic

– 2) Then use those value to obtain the Big-Oh’s for the complex statements• e.g. blocks, for loops, if-statements


Simple Statements (in C)

• We assume that simple statements always take a constant amount of time to execute– written as O(1)

• Kinds of simple statements:– assignment, break, continue, return, all library f

unctions (e.g. putchar(),scanf()), arithmetic, boolean tests, array indexing


Complex Statements

• The Big-Oh value for a complex statement is a combination of the Big-Oh values of its component simple statements.

• Kinds of complex statements:– blocks { ... }– conditionals: if-then-else, switch– loops: for, while, do-while

continued


• The easiest way to see how complex statement timings are based on simple statements (and other complex statements) is by drawing a structure tree for the program.

3.2. Structure Trees


Example: binary conversion

void main(){ int i;

(1) scanf(“%d”, &i);(2) while (i > 0) {(3) putchar(‘0’ + i%2);(4) i = i/2;

}(5) putchar(‘\n’);

}


Structure Tree for Example

block1-5

1 5while2-4

block3-4

3 4

the time for thisis the time for

(3) + (4)


3.3. Details for Complex Statements

• Blocks: Running time bound =

summation of the bounds of its parts.

• The summation rule means that only the largest Big-Oh value is considered.

"summation" means 'add'


Block Calculation Graphically

O( f1(n) )

O( f2(n) )

O( fk(n) )

O( f1(n) + f2(n) + ... + fk(n))

In other words:

O( largest fi(n) )

summation rule


Block Summation Rule Example

• First block's time T1(n) = O(n2)• Second block's time T2(n) = O(n)

• Total running time = O(n2 + n)= O(n2)the largest part


• Conditionals: Running time bound = the cost of the if-test + larger of the bounds for the

if- and else- parts

• When the if-test is a simple statement (a boolean test), it is O(1).

Conditionals e.g. if statements, switches


Conditional Graphically

Test

ElsePart

IfPart

O(1)

O( max( f1(n), f2(n)) +1 )which is the same asO( max( f1(n), f2(n)) )

O( f1(n) ) O( f2(n) )


If Example

• Code fragment:if (x < y) // O(1) foo(x); // O(n)else bar(y); // O(n2)

• Total running time = O( max(n,n2) + 1)= O(n2 + 1)= O(n2)


• Loops: Running time bound is usually =the max. number of times round the loop *the time to execute the loop body once

• But we must include O(1) for the increment and test each time around the loop.

• Must also include the initialization and final test costs (both O(1)).

Loops


While Graphically

Test

Body

O(1)

O( f(n) )

At mostg(n) times

aroundO( g(n)*f(n) )

Altogether this is:O( g(n)*(f(n)+1) + 1 )

which can be simplified to:O( g(n)*f(n) )


While Loop Example• Code fragment:

x = 0;while (x < n) { // O(1) for test foo(x, n); // O(n2) x++; // O(1)}

• Total running time of loop:= O( n*( 1 + n2 + 1) + 1 )

= O(n3 + 2n + 1) = O(n3)


Do-While Graphically

Test

Body

O(1)

O( f(n) )

At mostg(n) times

around

O( g(n)*f(n) )

Altogether this is:O( g(n)*(f(n)+1) + 1 )

which can be simplified to:O( g(n)*f(n) )


For-loop Graphically

Test

Body

Increment

Initialize

O(1)

O( f(n) )

At mostg(n) times

around

O( g(n)*(f(n)+1+1) + 1)which can be simplified to:

O( g(n)*f(n) )

O(1)

O(1)


For Loop Example

• Code Fragment:for (i=0; i < n; i++) foo(i, n); // O(n2)

• It helps to rewrite this as a while loop:

i=0; // O(1)while (i < n) { // O(1) for test foo(i, n); // O(n2) i++; // O(1)}

continued


• Running time for the for loop:= O( 1 + n*( 1 + n2 + 1) + 1 )

= O( 2 + n3 + 2n )

= O(n3)


3.4.1. Example: nested loops

(1) for(i=0; i < n; i++)_ (2) for (j = 0; j < n; j++)(3) A[i][j] = 0;

• line (3) is a simple op - takes O(1)• line (2) is a loop carried out n times

– takes O(n *1) = O(n)

• line (1) is a loop carried out n times– takes O(n * n) = O(n2)


3.4.2. Example: if statement

(1) if (A[0][0] == 0) {(2) for(i=0; i < n; i++)_ (3) for (j = 0; j < n; j++)(4) A[i][j] = 0;

}(5) else {(6) for (i=0; i < n; i++)(7) A[i][i] = 1;

}

continued


• The if-test takes O(1);the if block takes O(n2);the else block takes O(n).

• Total running time:= O(1) + O( max(n2, n) )

= O(1) + O(n2)

= O(n2) // using the summation rule


3.4.3. Time for a Binary Conversion

void main(){ int i;

(1) scanf(“%d”, &i);(2) while (i > 0) {(3) putchar(‘0’ + i%2);(4) i = i/2;

}(5) putchar(‘\n’);

}

continued


• Lines 1, 2, 3, 4, 5: each O(1)• Block of 3-4 is O(1) + O(1) = O(1)

• While of 2-4 loops at most (log2 i)+1 times

– total running time = O(1 * log2 i+1) = O(log2 i)

• Block of 1-5:= O(1) + O(log2 i) + O(1)

= O(log2 i)

why?


Why (log2 i)+1 ?• Assume i = 2k

• Start 1st iteration, i = 2k

Start 2nd iteration, i = 2k-1

Start 3rd iteration, i = 2k-2

Start kth iteration, i = 2k-(k-1) = 21 = 2Start k+1th iteration, i = 2k-k = 20 = 1– the while will terminate after this iteration

• Since 2k = i, so k = log2 i

• So k+1, the no. of iterations, = (log2 i)+1


Using a Structure Tree

block1-5

1 5while2-4

block3-4

3 4

O(1)

O(1)O(1)

O(1)

O(1)

O(log2 i)

O(log2 i)


3.4.4. Time for a Selection Sort

void selectionSort(int A[], int n) { int i, j, small, temp;(1) for (i=0; i < n-1; i++) {(2) small = i;(3) for( j= i+1; j < n; j++)(4) if (A[j] < A[small]) (5) small = j;(6) temp = A[small];(7) A[small] = A[i];(8) A[i] = temp; } }


Selection Sort Structure Treefor1-8

6 7

block2-8

for3-52

5

if4-5

8


• Lines 2, 5, 6, 7, 8: each is O(1)• If of 4-5 is O(max(1,0)+1) = O(1)• For of 3-5 is O( (n-(i+1))*1) = O(n-i-1)

= O(n), simplified• Block of 2-8

= O(1) + O(n) + O(1) + O(1) + O(1) = O(n)

• For of 1-8 is:= O( (n-1) * n) = O(n2 - n)

= O(n2), simplified

if part else part


4. Analyzing Function calls

• In this section, we assume that the functions

are not recursive– we add recursion in section (5)

• Size measures for all the functions must be similar, so they can be combined to give the program’s Big-Oh value.


Example Program

#include <stdio.h>

int bar(int x, int n); int foo(int x, int n):

void main() { int a, n;(1) scanf(“%d”, &n);(2) a = foo(0, n);(3) printf(“%d\n”, bar(a,n)); }

continued


int bar(int x, int n)int bar(int x, int n) { int i; { int i;(4) for(i = 1; i <= n; i++)(4) for(i = 1; i <= n; i++)(5) x += i;(5) x += i;(6) return x;(6) return x; } }

int foo(int x, int n) int foo(int x, int n) { int i; { int i;(7) for(i = 1; i <= n; i++)(7) for(i = 1; i <= n; i++)(8) x += bar(i, n);(8) x += bar(i, n);(9) return x;(9) return x; } }


Calling Graph

main

foo

bar


Calculating Times with a Calling Graph

• 1. Calculate times for Group 0 functions– those that call no other user functions

• 2. Calculate times for Group 1 functions– those that call Group 0 functions only

• 3. Calculate times for Group 2 functions– those that call Group 0 and Group 1 functions only

• 4. Continue until the time for main() is obtained.


Example Program Analysis

• Group 0: bar() is O(n)

• Group 1: foo() is O( n * n) = O(n2)

• Group 2: main() is= O(1) + O(n2) + O(1) + O(n)

= O(n2)

bar() in body


5. Analyzing Recursive Functions

• Recursive functions call themselves with a “smaller size” argument, and terminate by calling a base case.

int factorial(int n){ if (n <= 1) return 1; else return n*factorial(n-1);}


Running Time for a Recursive Function

• 1. Develop basis and inductive statements for the running time.

• 2. Solve the corresponding recurrence relation.– this usually requires the Big-Oh notation to be rew

ritten as constants and multiples of n– e.g. O(1) becomes a, O(n) becomes b*n,

O(n2) becomes c*n2, etc.

continued


• 3. Translate the solved relation back into Big-Oh notation– rewrite the remaining constants back into Bi

g-Oh form– e.g. a becomes O(1), b*n becomes O(n)


5.1. Factorial Running Time• Step 1.

– Basis: T(1) = O(1)– Induction: T(n) = O(1) + T(n-1), for n > 1

• Step 2.– Simplify the relation by replacing the O() notation w

ith constants.– Basis: T(1) = a– Induction: T(n) = b + T(n-1), for n > 1


• The simplest way to solve T(n) is to calculate it for some values of n, and then guess the general expression.

T(1) = aT(2) = b + T(1) = b + aT(3) = b + T(2) = 2b + aT(4) = b + T(3) = 3b + a

• “Obviously”, the general form is:T(n) = ((n-1)*b) + a

= bn + (a-b)continued


• Step 3. Translate back:T(n) = bn + (a-b)

• Replace constants by Big-Oh notation:T(n) = O(n) + O(1)

= O(n)

• The running time for recursive factorial is O(n). That is fast.


5.2. Recursive Selection Sort

void rSSort(int A[], int n){ int imax, i; if (n == 1) return; else { imax = 0; /* A[0] is biggest */ for (i=1; i < n; i++) if (A[i] > A[imax]) imax = i; swap(A, n-1, imax); rSSort(A, n-1); }}


• Step 1.– Basis: T(1) = O(1)– Induction: T(n) = O(n-1) + T(n-1), for n > 1

• Step 2.– Basis: T(1) = a– Induction: T(n) = b(n-1) + T(n-1), for n > 1

Running Time

continued

multiple of n-1

theloop call to

rSSort()

Assume swap() is O(1), so ignore

n == the size of the array


• Solve the relation:– T(1) = a

T(2) = b + T(1) = b + aT(3) = 2b + T(2) = 2b + b + aT(4) = 3b + T(3) = 3b + 2b + b + a

• General Form:– T(n) = (n-1)b + ... + b + a

= a + b(n-1)n/2

1

0

)(n

i

ibanT

continued


• Step 3. Translate back:T(n) = a + b(n-1)n/2

• Replace constants by Big-Oh notation:T(n) = O(1) + O(n2) + O(n)

= O(n2)

• The running time for recursive selection sort is O(n2). That is slow for large arrays.


5.3. Binary Searchint binSrch(char A[], int i,int j, char key){ int k; if (i > j) /* key not found */ return -1; k = (i+j)/2; if (key == A[k]) /* key found */ return k; if (key < A[k]) j = k-1; /* search left half */ else i = k+1; /* search right half */ return binSrch(A, i, j, key);}


Execution Example

/* find 'h' in A[] */binSrch(A, 0, 7, ‘h’);

0 1 2 3 4 5 6 7

A a d f g h w x y

you use binary searchto search for a numberin a phone book


Running Time

• Step 1.– Basis: T(1) = O(1)– Induction: T(n) = O(1) + T(< n/2 >), for n > 0

• Step 2.– Basis: T(1) = a– Induction: T(n) = b + T(< n/2 >), for n > 0

n == the range of the arraybeing looked at


Solve the Relation

• This time the relation is harder to solve since the precise value of < n/2 > depends on n.

• This affects how many calls it takes to get to T(1).

• Assume the simple case:– n starts as a power of 2: 2k

continued


• Evaluate T(n), where n is 2k:T(2k) = b + T(2k-1)

= b + b + T(2k-2)= b + b + b + T(2k-3)= b + b + b + .... + b + T(1)

• General Form:

T(n) = bk +a

continued

k of thesee.g. T(32) = b + b + b + b + b + 1


• We assumed that 2k = n, so k = log2 n

• This means that:T(n) = b*log2 n + a

• Step 3. Replace constants by Big-Oh notation:T(n) = O(log2 n) + O(1)

= O(log2 n)

• Running time for binary search is O(log2 n).


6. Towers of HanoiPole A

Pole BPole C

disks

How long to move 50 disksfrom A to C? 36 years


• Problem: transfer all the disks from pole A to pole B.– 1. In each step, only 1 disk can be moved from o

ne pole to another.

– 2. A disk may never be placed on top of a smaller disk.

– 3. Pole C may be used.


Solution for 3 Disks

• In the following, X --> Y means “move the top disk from Pole X to Pole Y”.

• Problem: move 3 disks from A to B• Solution:

A --> B, A --> C, B --> C, A --> B,C --> A, C --> B, A --> B


Pseudo-code

void move(int n, Pole x, Pole y, Pole z)/* move n disks from x to y, use z */{ if (n == 1) print “x --> y”; else { move( n-1, x, z, y); // x to z, use y print “x --> y”; move( n-1, z, y, x); // z to y, use x }}


Recursive Part Graphically

• move( n-1, x, z, y);

print “x --> y”;

move( n-1, z, y, x);

x y zmove(n, x, y, z)

x y z

x y z

x y z


Execution for 3 Disksmove 3 from A to B

using C

move 2 from A to Cusing B

move 2 from C to Busing A

move 1 from A to Busing C

move 1 from B to Cusing A

A --> B

A --> C

A --> B B --> C

start stop

. . . .

continued

nextslide


move 2 from C to Busing A

move 1 from A to Busing C

move 1 from C to Ausing B

C --> B

C --> A A --> B

. . . .

. . . .


Size for the Running Time

• We will calculate the running time by using the number of disk moves as the size measure.

• So T(n) = number of disk moves to solve the n-disk problem.


Running Time

• Step 1.– Basis: T(1) = O(1)– Induction: T(n) = 2*T(n-1) + O(1), for n > 1

• Step 2.– Basis: T(1) = a– Induction: T(n) = 2*T(n-1) + b, for n > 1

continued


• Solve the relation:T(n) = 2T(n-1) + b

= 2( 2T(n-2) + b ) + b= 4T(n-2) + 2b + b= 4( 2T(n-3) + b ) + 2b + b= 8T(n-3) + 4b + 2b + b

:= 2n-1T(1) + (2n-2 + 2n-3 + ... + 2 + 1)b= 2n-1a + (2n-2 + 2n-3 + ... + 2 + 1)b~= (2n-1 + 2n-2 + 2n-3 + ... + 2 + 1)b= (2n - 1)b

continued


• Step 3. Translate back:T(n) = (2n - 1)b

• Replace constants by Big-Oh notation:T(n) = O(2n)

• The running time for “Towers of Hanoi” is O(2n)– exponential growth!

continued


• This exponential growth means that a practical limit is quickly reached to the number of disks that can be moved.

• Look at slide 13 to see approximately how long it would take to move 1000 disks if a single disk move takes 1 microsec.


More Information

• “Towers of Hanoi” written in every language (C, C++, Java, JavaScript, Fortran, Ada, ML, etc):http://www.pangea.ca/kolar/javascript/Hanoi/HTonWebE.html

• A nice Java applet:http://www.cut-the-knot.com/recurrence/hanoi.html

continued


• Some history about the problem, animations, a Windows game: http://www.lhs.berkeley.edu/Java/Tower/

• AlgorithmicsDavid HarelAddison-Wesley, 1987– various places in the text, first on p.32


7. Further Information

• DM: section 5.3, chapter 5

241-303 discrete maths: runtime/4 1 discrete maths objective – –to describe the big-oh notation...

Documents