11

Algorithm Analysis

CptS 223 – Advanced Data Structures

Larry HolderSchool of Electrical Engineering and Computer Science

Washington State University

2

Purpose

Why bother analyzing code; isn’t getting it to work enough?

Estimate time and memory in the average case and worst caseIdentify bottlenecks, i.e., where to reduce timeSpeed up critical algorithms

3

Algorithm

AlgorithmWell-defined computational procedure for transforming inputs to outputs

ProblemSpecifies the desired input-output relationship

Correct algorithmProduces the correct output for every possible input in finite timeSolves the problem

4

Algorithm AnalysisPredict resource utilization of an algorithm

Running timeMemory

Dependent on architectureSerialParallelQuantum

5

What to AnalyzeMain focus is on running time

Memory/time tradeoffMemory is cheap ($20 per GB)

Simple serial computing modelSingle processor, infinite memory

Cray XMT Supercomputer•Up to 64 TB (65,536 GB) shared memory•Up to 8000 processors•128 independent threads per processor•$150M

6

What to AnalyzeRunning time T(N)

N is typically the size of the inputSorting?Multiplying two integers?Multiplying two matrices?Traversing a graph?

T(N) measures number of primitive operations performed

E.g., addition, multiplication, comparison, assignment

7

Example

Costint sum (int n) 0

int partialSum; 0

partialSum = 0; 1for (int i = 1; i <= n; i++) 1+(N+1)+N

partialSum += i * i * i; N*(1+1+2)return partialSum; 1

T(N) = 6N+4

8

What to Analyze

Worst-case running time Tworst(N)Average-case running time Tavg(N)Tavg(N) <= Tworst(N)Typically analyze worst-case behavior

Average case hard to computeWorst-case gives guaranteed upper bound

9

Rate of Growth

Exact expressions for T(N) meaningless and hard to compareRate of growth

Asymptotic behavior of T(N) as N gets bigUsually expressed as fastest growing term in T(N), dropping constant coefficientsE.g., T(N) = 3N2 + N + 1 Θ(N2)

10

Rate of Growth

T(N) = O(f(N)) if there are positive constants c and n0 such that T(N) ≤ cf(N) when N ≥ n0

Asymptotic upper bound“Big-Oh” notation

T(N) = Ω(g(N)) if there are positive constants c and n0 such that T(N) ≥ cg(N) when N ≥ n0

Asymptotic lower bound“Big-Omega” notation

Rate of Growth

11

n0

f(N)g(N)

N

T(N)

g(N) = O(f(N))f(N) = Ω(g(N))

12

Rate of Growth

T(N) = Θ(h(N)) if and only if T(N) = O(h(N)) and T(N) = Ω(h(N))

Asymptotic tight bound

T(N) = o(p(N)) if for all constants c there exists an n0 such that T(N) < cp(N) when N>n0

I.e., T(N) = o(p(N)) if T(N) = O(p(N)) and T(N) ≠Θ(p(N))“Little-oh” notation

13

Rate of Growth

N2 = O(N2) = O(N3) = O(2N)N2 = Ω(1) = Ω(N) = Ω(N2)N2 = Θ(N2)N2 = o(N3)2N2 + 1 = Θ(?)N2 + N = Θ(?)

14

Rate of Growth

Rule 1: If T1(N) = O(f(N)) and T2(N) = O(g(N)), then

T1(N) + T2(N) = O(f(N) + g(N))T1(N) * T2(N) = O(f(N) * g(N))

Rule 2: If T(N) is a polynomial of degree k, then T(N) = Θ(Nk)Rule 3: logk N = O(N) for any constant k

15

Rate of Growth

16

Example

Maximum subsequence sum problemGiven N integers A1, A2, …, AN, find the maximum value (≥ 0) of:

Don’t need the actual sequence (i,j)E.g., <1, -4, 4, 2, -3, 5, 8, -2> 16

∑ =

j

ik kA

17

MaxSubSum: Solution 1

Compute each possible subsequence independently

MaxSubSum1 (A)maxSum = 0for i = 1 to N

for j = i to Nsum = 0for k = i to j

sum = sum + A[k]if (sum > maxSum)then maxSum = sum

return maxSum

18

19

Solution 1: Analysis

Three nested for loops, each iterating at most N timesOperations inside for loops take constant timeThus, T(N) = ?But, for loops don’t always iterate N times

20

Solution 1: Analysis

More precisely

∑∑∑−

=

−

= =

Θ=1

0

1

)1()(N

i

N

ij

j

ikNT

21

MaxSubSum: Solution 2

Note thatSo, no reason to re-compute sum each time MaxSubSum2 (A)

maxSum = 0for i = 1 to N

sum = 0for j = i to N

sum = sum + A[j]if (sum > maxSum)then maxSum = sum

return maxSum

∑∑ −

==+=

1j

ik kjj

ik k AAA

22

23

Solution 2: Analysis

)()(

)1()(

2

1

0

1

NNT

NTN

i

N

ij

Θ=

Θ=∑∑−

=

−

=

24

MaxSubSum: Solution 3

Recursive, divide and conquerDivide sequence in half

A1..center and A(center+1)..N

Recursively compute MaxSubSum of left halfRecursively compute MaxSubSum of right halfCompute MaxSubSum of sequence constrained to use Acenter and A(center+1)

E.g., <1, -4, 4, 2, -3, 5, 8, -2>

25

MaxSubSum: Solution 3MaxSubSum3 (A, i, j)

maxSum = 0if (i = j)then if A[i] > 0

then maxSum = A[i]else k = floor((i+j)/2)

maxSumLeft = MaxSubSum3(A,i,k)maxSumRight = MaxSubSum3(A,k+1,j)// compute maxSumThruCentermaxSum = Maximum (maxSumLeft,

maxSumRight,maxSumThruCenter)

return maxSum

26

27

28

29

Solution 3: Analysis

T(1) = Θ(1)T(N) = 2T(N/2) + Θ(N)T(N) = Θ(?)

30

MaxSubSum: Solution 4

ObservationAny negative subsequence cannot be a prefix to the maximum sequenceOr, a positive, contiguous subsequence is always worth adding

T(N) = ?

MaxSubSum4 (A)maxSum = 0sum = 0for j = 1 to N

sum = sum + A[j]if (sum > maxSum)then maxSum = sumelse if (sum < 0)

then sum = 0return maxSum

31

32

MaxSubSum Running Times

Time in seconds.Does not include time to read array.38 min

26 days

33

MaxSubSum Running Times

34

MaxSubSum Running Times

35

Logarithmic Behavior

T(N) = O(log2 N)Usually occurs when

Problem can be halved in constant timeSolutions to sub-problems combined in constant time

ExamplesBinary searchEuclid’s algorithmExponentiation

36

Binary Search

Given an integer X and integers A0,A1,…,AN-1, which are presorted and already in memory, find i such that Ai=X, or return i = -1 if X is not in the input.T(N) = O(log2 N)T(N) = Θ(log2 N) ?

37

38

Euclid’s Algorithm

Compute the greatest common divisor gcd(M,N) between the integers M and N

I.e., largest integer that divides bothUsed in encryption

Euclid’s Algorithm

39

Example: gcd(3360,225)•m = 3360, n = 225•m = 225, n = 210•m = 210, n = 15•m = 15, n = 0

Euclid’s Algorithm: Analysis

Note: After two iterations, remainder is at most half its original value

Thm. 2.1: If M > N, then M mod N < M/2

T(N) = 2 log2 N = O(log2 N)log2225 = 7.8, T(225) = 16 (?)

Better worst case: T(N) = 1.44 log2 NT(225) = 11

Average case:T(225) = 6

40

47.1/)ln2ln12()( 2 += πNNT

Exponentiation

Compute XN

Obvious algorithm:Observation

XN = XN/2 * XN/2 (for even N)XN = X(N-1)/2 * X(N-1)/2 * X (for odd N)

Minimize multiplications T(N)T(N) = 2 log2 N = O(log2 N)

41

pow(x,n)result = 1for i = 1 to n

result = result * xreturn result

Exponentiation

42

T(N) = Θ(1), N ≤ 1T(N) = T(N/2) + Θ(1), N > 1

T(N) = O(log2 N)T(N) = Θ(log2 N) ?

Summary

Algorithm analysisBound running time as input gets bigRate of growth: O() and Θ()Compare algorithmsRecursion and logarithmic behavior

43