css342: algorithm analysis1 professor: munehiro fukuda
TRANSCRIPT
CSS342: Algorithm Analysis 2
Today’s Topics
1. Mathematical analysis of algorithm complexity
2. Examples of algorithm complexity
3. Algorithm improvement from O(N3) to O(N)
4. Efficiency of search algorithms
CSS342: Algorithm Analysis 3
Measuring Algorithm Efficiency• Cost of a computer program
– Human cost:
Developing good problem-solving skills and programming style
– Execution cost:
Algorithm efficiency Then, what in efficiency should we focus?
Clever trick in coding? Execution time? How should we compare programs?
Based on a particular computer? What data should be given?
Mathematical Analysis
CSS342: Algorithm Analysis 4
Counting an Algorithm’s operationsfor ( i = 1; i <= n; i++ )
for ( j = 1; j <= i; j++ )
for ( k = 1; k <= 5; k++ )
task( );
Task: t operations ( or may take t seconds)
Loop on k: 5 * t
Loop on j: 5 * t (i = 1), 5 * t * 2 (i = 2), … 5 * t * n (i = n)
n
Loop on i: ∑(5 * t * i) = 5 * t * n * (n + 1) / 2i=1
Mathematical Analysis
CSS342: Algorithm Analysis 5
Algorithm Growth Rates• Nested loop i/j/k requires 5n(n+1)/2 time units
to process a problem of size n.– Units may be seconds, minutes, and even dependent
on what machines we use!
• Nested loop i/j/k requires time proportional to 5n(n+1)/2– Focus on how quickly the algorithm’s time
requirement grows as a function of the problem size.
Growth rate
Mathematical Analysis
CSS342: Algorithm Analysis 6
Order-of-Magnitude Analysis
Let Algorithm A require time proportional to f(n).
• A is of order at most g(n), O(g(n)) if f(n) <= k * g(n), for a positive constant k and n > n0
• A is of order at least g(n), Ω(g(n)) iff(n) >= k * g(n), for a positive constant k and n > n0
• A is of order g(n), Θ(g(n)) iff(n) = O(g(n)) and f(n) = Ω(g(n))
Mathematical Analysis
CSS342: Algorithm Analysis 7
Example of O(g(n)), Ω(g(n)) ,andΘ(g(n))
Suppose Algorithm A requires time proportional to f(n) = n2 – 3n + 10
• f(n) < k* n2 , if k = 3 and n >= 2, and thusf(n) = O(n2 )
• f(n) > k * n2 , if k = 0.5 n>= 0, and thusf(n) = Ω(n2)
• Thereforef(n) = Θ(n2)
27104.53
12822
380.51
01000
3n2N2 -3N+100.5n2n
Mathematical Analysis
CSS342: Algorithm Analysis 8
Example of Program Analysisj = n
while ( j >= 1 ) {
for ( i = 1; i <= j; i++ )
x = x + 1; // How many times is x=x+1 executed?
j = j / 2;
}
T(n): # x=x+1 executed
In the 1st while-loop, j = n. for-loop executes x=x+1 n times. T(n)=Ω(n)
In the 2nd while-loop, j <= n/2, n/2 times
In the 3rd while-loop, j <= n/4, n/4 times
In the kth while-loop, j <= n/2k-1, n/2k-1times
Mathematical Analysis
CSS342: Algorithm Analysis 9
Example of Program Analysis Cont’d
Geometric Sum:
a + ar1 + ar2 + …. + arn = a(rn+1 – 1)/(r – 1), where r != 1
(Proof is given in the week 4, “Induction”)
Thus,
T(n) <= n + n/2 + n/4 + … + n/2k-1
= n + n*(1/2) + n*(1/2)2 + … + n * (1/2)(k-1)
= n((1/2)k-1)/(1/2 – 1) = n(1-1/2k)/(1-1/2)
= 2n(1-1/2k) <= 2n
T(n) = O(n)
Therefore, T(n) = Θ(n)
Mathematical Analysis
CSS342: Algorithm Analysis 10
Focusing on Big O Notation
• Upper-bound information is the main concern • In most cases, O(g(n))=Ω(g(n)) =Θ(g(n))
• O(1) < O(log2n) < O(n) < O(n*log2n) < O(n2) < O(n3) < O(2n)
• Focus on the dominant factor:– Low-order terms can be ignored. O(n3+4n) = O(n3)
– Multiplicative constant in the high-order term can be ignored. O(3n3+4) = O(n3)
• O(f(n)) + O(g(n)) = O(f(n) + g(n))• If f(n) = O(g(n)) and g(n) = O(h(n)) , then f(n)=O(h(n))
Mathematical Analysis
CSS342: Algorithm Analysis 11
Intuitive interpretation of growth-rate functions
Increase too rapidly to be practicalExponentialO(2n)
Ex. Three nested loopsCubicO(n3)
Ex. Two nested loopsQuadraticO(n2)
Increase more rapidly than a liner algorithm. Ex. Merge sort
O(nlog2n)
Increase directly with sizeLinearO(n)
Increase slowly as size increases. Ex. Binary search
LogarithmicO(log2n)
Independent of problem sizeConstantO(1)
Mathematical Analysis
CSS342: Algorithm Analysis 12
Examples of Algorithm ComplexityMinimum Element in an Array
• Given an array of N items, find the smallest item.• What order (in big O) will this problem be bound to?
3 9 1 8 5 2 0 7 4 6 -1 10 -2
N1
Examples
CSS342: Algorithm Analysis 13
Examples of Algorithm ComplexityClosest Points in the Plane
• Given N points in a plane (that is, an x-y coordinate system), find the pair of points that are closest together.
• What order (in big O) will this problem be bound to?
2,1
x
y1,41,6
4,74,5
5,45,6
7,18,4
6,76,3
sqrt( (x2 – x1)2 + (y2 – y1)2 )
Examples
CSS342: Algorithm Analysis 14
Examples of Algorithm ComplexityColinear Points in the Plane
• Given N points in a plane (that is, an x-y coordinate system), determine if any tree form a straight line.
• What order (in big O) will this problem be bound to?
2,1
x
y1,41,6
4,74,5
5,45,6
7,18,4
6,76,3
Find y = ax + b for two points.Check if the 3rd point satisfies this equation
Examples
CSS342: Algorithm Analysis 15
The Maximum Contiguous Subsequence Sum Problem
• Given a sequence of integers I1..IN, find and identify the sub sequence corresponding to the maximum value of ∑k=i
jAk.
Algorithm Improvement
-2 11 -4 13 -5 2 1 -3 4 -2 -1 6 -2
N
20
1
CSS342: Algorithm Analysis 16
O(N3) AlgorithmAlgorithm Improvement
-2 11 -4 13 -5 2 1 -3 4 -2 -1 6 -2
N1
i j
∑k=ijAk
The most inner loop
The intermediate loop
j++The most outer loop i++
int maxSum = 0;for ( int i = 0; i < n; i++ ) for ( int j = i; j < n; j++ ) { int thisSum = 0; for ( int k = i; k <= j; k++ ) thisSum += a[k];
if ( thisSum > maxSum ) { maxSum = thisSum; seqStart = i; seqEnd = j; } }
We don’t need this loop!Simply increment j and add a new integer to the sum.
CSS342: Algorithm Analysis 17
O(N2) AlgorithmAlgorithm Improvement
-2 11 -4 13 -5 2 1 -3 4 -2 -1 6 -2
N1
i j
∑j=ijAj
The intermediate loop
j++The most outer loop i++
int maxSum = 0;for ( int i = 0; i < n; i++ ) { int thisSum = 0; for ( int j = i; j < n; j++ ) { thisSum += a[j];
if ( thisSum > maxSum ) { maxSum = thisSum; seqStart = i; seqEnd = j; } }}
CSS342: Algorithm Analysis 18
O(N) AlgorithmAlgorithm Improvement
N1∑ < 0
Ignore this partFrom here, restart the summation.
∑j=inAj
Generally, the summation increases but if it gets below 0
int thisSum = 0;int maxSum = 0;for ( int i = 0, j = 0; j < n; j++ ) { thisSum += a[j];
if ( thisSum > maxSum ) { maxSum = thisSum; seqStart = i; seqEnd = j; } else if ( thisSun < 0 ) { i = j + 1; thisSum = 0; }}
CSS342: Algorithm Analysis 19
Worst, Best, and Average-case Analysis
• Worst-case analysis:– Algorithm A requires no more than k * f(n) time units.
– It might happen rarely, if at all, in practice.
• Best-case analysis:– Like worst-case analysis, it might happen rarely.
• Average-case analysis:– A requires no more than k * f(n) time units for all but a finite
number of values of n
– Difficulties: determining probability and distribution of size n and input data
Efficiency of Search Algorithms
CSS342: Algorithm Analysis 20
Efficiency of Sequential Search
29 10 14 1337 4652 75 5 43 6921 188
Best caseO(1)
Hit!
Worst caseHit!
O(n)
Average caseHit!
O(n/2) = O(n)(Assume desired items are uniformly distributed.)
Efficiency of Search Algorithms
CSS342: Algorithm Analysis 21
Efficiency of Binary Search
2910 1413 37 46 52 755 43 69211 88 9991
Efficiency of Search Algorithms
template <class Comparable>int binarySearch( const vector<Comparable> &a, const Comparable &x ) { int low = 0; int high = a.size( ) – 1; int mid;
while ( low <= high ) { mid = ( low + high ) / 2;
if ( a[mid] < x ) low = mid + 1; else if ( a[mid] > x ) high = mid – 1; else return mid; } return NOT_FOUND; // NOT_FOUND = -1}
low =0 high=15mid = (15 + 0)/2 = 7
10x
anew high=7–1=6
CSS342: Algorithm Analysis 22
Efficiency of Binary Search
N = 2 K = 1N = 4 K = 2N = 8 K = 3N = 16 K = 4N = 2k K = K2K-1 < N < 2K
K – 1 < log2N < K
K < 1 + log2N < K + 1 K = O(log2N)
2910 1413 37 46 52 755 43 69211 88
1st step
9991
2nd step3rd step
4th stepHit!
Efficiency of Search Algorithms
CSS342: Algorithm Analysis 23
Efficiency of Interpolation Search
• A way you start your search for “Hank Aaron” from the first 20 pages in a 500-page white page:next = low + ┌ (x – a[low]) / (a[high] – a[low]) * (high – low – 1)┐Example:
Assume low = 0, high = 999, a[low] = 1000, a[high]=1000000.If you search for the value 12000:next = 0 + ┌(12000 – 1000)/(1000000 – 1000) * (999 – 0 – 1) ┐ = 10
• Two assumptions:– Each access must be very expensive like a disk access.– Data must be uniformly distributed
• Complexity– Average: O(log log N)– Worst: O(N)
Efficiency of Search Algorithms
CSS342: Algorithm Analysis 24
Some Note for Algorithm Analysis
• Checking an algorithm analysis1. Empirically observe running time as increasing N.2. Divide the time by O(f(N)).3. The dividend should converge to a positive constant.
• Limitations of Big-Oh Analysis– Does not work for a small N– Worst-case analysis may be uncommon– Average-case analysis is more difficult than worse
case– Empirically observation depends on memory
hierarchy and multiprogramming level, etc..
Limitations
CSS342: Algorithm Analysis 25
Intractable, Unsolvable, and NP problems
• Good algorithms:– Their worst-case time is proportional to a polynomial.
• Intractable problems:– Their worst-case time cannot be bounded to a polynomial.
• Unsolvable algorithms:– They have no algorithms at all. Ex. Halting problem
• NP(Non-Polynomial) problems:– They are thought to be intractable, but not yet proved.
– Ex1. Traveling salesperson problem
– Ex2. Hamiltonian cycle
Limitations