Evaluating & Improving Algorithms

Analyzing US Change

While M>0 c <- value of largest coin with value <= M give c coin to customer M <- M-c• Three statements per loop • Loop executes k times, where k is…• Therefore, running time is 3*k

Generalized US Change

for i <- 0 to d-1 Calculate and give the max number of

this coin (c[i])• How many times does this loop run? (k)• How many operations are in calculate &

give max? (x)• Resulting time is k*x

General Rules for Calculating Time

• Steps in sequence are added• A function is replaced by its # steps (or

count each call as 1 step)• Total # of steps in a loop gets multiplied

by the # of times the loop executes

Summary so far…

• Original US change– O(N) where N is # coins given

• Generalized change algorithm– O(N) where N is # different coins in system– But this doesn’t work for all cases!

Brute Force* Change

N <- 1While (not done) construct a combination of N coins if it adds up to M return it (done=true) else N <- N+1

*Brute Force = try all combinations

How many loops?

• One loop for every combination with too few coins 1! + 2! + …

• One loop for every combination with the right number of coins before the correct one– Best case for this part is 0– Worst case for this part is k! where k is the

number of coins in the solution

Summary and Analysis

• Original US change– O(N) where N is # coins given (linear)

• Generalized change algorithm– O(N) where N is # different coins – O(1) for a fixed coin system (constant)

• Brute force change algorithm– O (N * N!) where N is # coins given– (Worse than 2N)!

What We’ve Learned

• Brute force algorithms (try all possibilities) are guaranteed to work, but…

• Brute force algorithms (usually) take too long

• We need ways to speed up the work

Detour: Recursion

• Specify a solution to a problem in terms of solutions to one or more smaller problems

• Mathematically elegant

Recursive Fibonacci

• Mathematical definition– F(0) = 0– F(1) = 1– F(N) = F(N-1) + F(N-2)

• Implemented directly:– F(4) = F(3)+F(2) = F(2)+F(1)+F(2) =

F(1)+F(0)+F(1)+F(2) = 1+1+1+F(2) = 3+F(1)+F(0) = 3+1+1 = 5

A Tree of Recursive CallsF(4)

F(3) F(2)

F(2) F(0)

1F(1) F(1)

F(1) F(0)1 1

1 1

Dynamic Programming

• Save intermediate results from which the final result can be computed

• Order the computations so you have all saved results before you need them

• Example: our Fibonacci Algorithm

Dynamic Programming For Fibonacci

• Generate answers from smallest to largest• Use previous answers in calculationsfibonacci (n) O(n) where n is input f[0] <- 1 f[1] <- 1 for i <- 2 to n-1 n times f[i] <- f[i-1]+f[i-2]1 step return f[n-1]

Recursion Isn’t Always Slower

• Recursive US Change (M)if (M>0)

c <- value of largest coin with value <= M give c coin to customer Recursive US Change(M-c)• Only one recursive call, at end• This is “tail recursion”, just as fast as the

original loop

Greedy Algorithms

• Order your algorithm to solve the “biggest chunk” first– Example: always give the biggest coin

possible in the change problem– Not every greedy algorithm is correct (e.g.

generalized change)

Branch and Bound

• Consider a brute force problem solution as a series of choices

• When a single choice is avoided (pruned), all future choices that depend on it are also avoided

• AI calls this “heuristic search”

Randomized

• Make each choice randomly until the problem is solved

• Not guaranteed to solve the problem• Technically not even algorithms, if not

guaranteed to halt (e.g. by time limit)• Sometimes we can prove that a “good

enough” solution is “likely enough” and these are simple to implement

Find the Phone (Brute Force)

Find the phone: if the phone is at your location return true (phone is found) else for each direction (left, right, forw, back) take a step if (find the phone) return true return false (phone not found here)

Find the Phone (Branch & Bound)

Find the phone: if the phone is at your location return true (phone is found) else listen to the phone & rule out opposite direction for each remaining direction take a step if (find the phone) return true return false (phone not found here)

Find the Phone (Greedy)Find the phone: if the phone is at your location return true (phone is found) else listen & pick best direction (left, right, forw, back) take a step if (find the phone) return true return false (phone not found here)

• What if there’s an irreplaceable Ming vase between you and the phone?

Find the Phone (Randomized)Find the phone: if the phone is at your location return true (phone is found) else repeat forever pick a random direction (left, right, forw, back) take a step if (find the phone) return true

Divide and Conquer

• If the problem is small enough, solve it directly

• Else– Break the problem down into 2 or more

subproblems– Solve each one separately (recursive step)– Combine the solutions

• Works best when subproblems are equal size

Divide & Conquer Example

• Quick Sort– Pick a value from the list (k)– Divide the input into “<k” list and “>k” list– Sort each list separately– Build the result:

• Sorted “<k” list followed by k followed by sorted “>=k” list

Machine Learning

• Given: many instances of input and output• Discover: a function that relates them• Generalize: to previously unseen inputs

• Generally need large amounts of representative data

• ML algorithms are fairly time consuming

Summary of Techniques

• Brute Force– Try every possibility– Rarely practical for reasonable size

problems• Dynamic programming

– Solve subproblems in the right order and save solutions to build the final answer

Summary (cont’d)

• Greedy Algorithms– Take the “biggest step” first each time. Not always

correct.• Branch and Bound

– Rule out impossible choices and their successors, but try everything else

• Divide and Conquer– Combine results from smaller subproblems

Summary (cont’d)

• Machine learning– Use large amounts of data to

“automatically” generate an appropriate solution algorithm

• Randomized algorithms– Try solutions at random until one is found

(or “good enough”)– Not always completely random (e.g.

genetic algorithms)