3 - 1 - introduction to greedy algorithms (13 min)

8/20/2019 3 - 1 - Introduction to Greedy Algorithms (13 Min)

http://slidepdf.com/reader/full/3-1-introduction-to-greedy-algorithms-13-min 1/8

Today, we're going to embark on thediscussion of a new algorithm designparadigm.Namely, that of designing and analyzinggreedy algorithms.So to put this study of greedy algorithmsin a little bit of context, let's just

zoom out.Let's both review some of the algorithmdesign paradigms that we've already seen,as well as look forward to some thatwe're going to learn later on, in thiscourse.So it's sort of a sad fact of life thatin algorithm design, there's no onesilver bullet.There's no magic potion that's the curefor all your computational problems.So instead, the best we can do, and the

focus of these courses, is to discussgeneral techniques that apply to lots ofdifferent problems that arise and lots ofdifferent domains.So that's what I mean by algorithm designparadigms.High level problem solving strategiesthat cut across multiple applications.So let's look at some examples.Back in part one, we started with thedivide and conquer algorithm designparadigm.A canonical example of that paradigm

being the merge sort algorithm.So remember in divide and conquer whatyou do is you take your problem you breakit into smaller sub problems, you solvethe sub problems recursively and then youcombine the results into a solution forthe original problem.Like how in merge sort you recursivelysort two sub arrays and then merge theresults to get a sorted version of theoriginal input array.Another paradigm that we touched on in

part one,although we didn't discuss it anywherenear as thoroughly, is that of randomizedalgorithms.So the idea that you could have code flipcoins,that is, make random choices, inside thecode itself.Often, this leads to simpler, morepractical, or more elegant algorithms.A canonical application here is the quicksort algorithm using a random pivotelement.

But we also saw applications for example,to the design of hash functions.So the next measure paradigm we're going



to discuss is that of greedy algorithms.So these are algorithms that iterativelymake myopic decisions.In fact, we've already seen an example ofa greedy algorithm in part one namelyDijkstra's shortest path algorithm.And then, the final paradigm we're going

to discuss in this class is that ofdynamic programming,a very powerful paradigm which solves, inparticular, two of the motivatingquestions we saw earlier namely, sequencealignment, and distributed shortestpaths.So what is a greedy algorithm anyways?Well, to be honest, I'm not going tooffer you a formal definition.In fact, much blood and ink has beenspilled over which algorithm is precisely

greedy algorithms.But, I'll give you a sort of informaldescription.A sort of rule of thumb for what greedyalgorithms usually look like.Generally speaking, what a greedyalgorithm does, is make a sequence ofdecisions with each decision being mademyopically.That is, it seems like a good idea at thetime and then you hope that everythingworks out at the end.The best way to get a feel for greedy

algorithms is to see examples and theupcoming lectures will give you a numberof them.But I want to point out we've actuallyalready seen an example of a greedyalgorithm in part one of this course,namely Dijkstra's shortest pathalgorithm.So in what sense is Dijkstra's algorithma greedy algorithm?Well if you recall the psuedo code forDijkstra's algorithm, you'll recallthere's one main wild loop and thealgorithm process's exactly one newdestination vertex in each iteration ofthis wild loop, so there's exactly N - 1iterations overall, where N is the numberof vertices.So the algorithm only gets one shot tocompute the shortest path to a givendestination. It never goes back andrevisits the decision, in that sense thedecisions are myoptic, irrevocable andthat's the sense in which Dijkstra'salgorithm is greedy.

So let me pause for a moment to discussthe greedy algorithm design paradigmgenerally.



Probably this discussion will seem alittle abstract so I recommend yourevisit this discussion on the slideafter we've seen a few examples so atthat point I think it will really hithome.So let me proceed by comparing it and

contrasting it to the paradigm we'vealready studied in depth.That of divide and conquer algorithms.So you'll recall that in a divide andconquer algorithm what you do is, youbreak the problem into sub-problems.So, maybe you take an input array and yousplit it into two sub-arrays.Then you solve the smaller sub-problemsrecursively, and then you combine theresults of the sub-problems into asolution to the original input.

So the greedy paradigm is quite differentin several respects.First, both a strength and a weakness ofthe greedy algorithm design paradigm isjust how easy it is to apply.So it's often quite easy to come up withplausible greedy algorithms for aproblem, even multiple differenceplausible greedy algorithms.I think that a point of contrast withdivide and conquer algorithms.Often it's tricky to come up with aplausible divide and conquer algorithm,

and usually you have this eureka momentwhere you finally figure out how todecompose the problem in the right way.And once you have the eureka moment,you're good to go.So secondly, I'm happy to report thatanalyzing running time of greedyalgorithms will generally be much easierthan it was with divide and conqueralgorithms.For divide and conquer algorithms it wasreally unclear whether they were fast orslow, because we had to understand therunning time over multiple levels ofrecursion.On the one hand problems were size wasgetting smaller, but on the other hand,the number of some problems wasproliferating.So we had to work hard, we developedthese powerful tools like the mastermethod, and some other techniques, forfiguring out just how fast an algorithmlike Merge Sort runs, or just how fast analgorithm like Strassen's fast matrix

multiplication algorithm runs.In contrast with greedy algorithms, itwill often be a one liner.



Often it will be clear that the work isdominated by say, a sorting sub routineand of course we all know that sortingtakes n log and time if you use asensible algorithm for it.Now the catch,and this is the third point of

comparison,is we're generally going to have to workmuch harder to understand correctnessissues of greedy algorithms.For divide-and-conquer algorithms wedidn't talk much about correctness.It was generally a pretty straightforwardinduction proof.You can review the lectures on Quicksortif you want an example of one of thosecanonical inductive correctness proofs.But the game totally changes with greedy

algorithms.In fact, given a greedy algorithm weoften won't even have very good intuitionfor whether or not they are correct.Let alone how to prove they're correct.So even with a correct algorithm, it'soften hard to figure out, why it'scorrect.And in fact, if you remember only onething from all of this greedy algorithmdiscussion many years from now,I hope one key thing you remember isthey're often not correct.

Often, especially if it's one youproposed yourself which you're verybiased, in favor of.You will think the algorithm, the greedyalgorithm must be correct because it's sonatural.But many of them are not, so keep that inmind.So to give you some immediate practicewith the ubiquitous incorrectness ofnatural algorithm.Let's review a point that we alreadycovered in part one of this classconcerning Dijkstra's algorithm.Now, in part one we made a big deal ofwhat a justly famous algorithm Dijkstra'sshortest path algorithm is, it runsbrazenly fast and it computes all theshortest paths.What else do you want?Well remember there was an assumptionwhen we proved that the Dijkstra'salgorithm is correct.We assumed that every edge of the givennetwork has a non negative length.

We did not allow negative edge lengths.And as we discussed in part one, youknow,



for many applications, you only careabout non negative edge lengths.But there are applications where you dowant negative edge lengths.So let's review on this quiz whyDijkstra's is actually incorrect, despitebeing so natural.

it's incorrect when edges can havenegative lengths.So I've drawn in green, a very simpleshortest path network with three edgesand I've annotated the edges with theirlinks.You'll notice one of those edges doeshave a negative length, the edge from Vto W with length minus two.So the question is consider the sourcevertex S and the destination vertex W.And the question is, what is the shortest

path distance computed by Dijkstra'salgorithm and you may have to go andreview just a pseudo code in part one oron the web.to answer that part of the question andthen what is in fact the actual shortestpath distance from S to W where as usualthe length of a path is just the sum ofthe lengths of the edges in the path.All right, so the correct answer is D.So let's start with the second part ofthe question, what is the actual lengthof a shortest path from S to W when

there's only two paths at all in thegraph?The one straight from S to W that haslength 2, and the one that goes by theintermediate point V that has length 3 +-21, = 1 which is shorter.So, SVW is the shortest path that haslength 1. Why is Dijkstra incorrect? Wellif you go back to the pseudo code ofDijkstra, you'll see that in the veryfirst iteration it will greedily find theclosest vertex to S in that case this isW, W is closer then V. It will greedilycompute the shortest path distance to Wknowing the information it has right nowand all it knows is there's this one hotpath from S to W, so it will irrevocablycommute to the shortest path distancefrom S to W as 2.Never reconsidering that decision later.So Dijkstra will terminate with theincorrect output that the shortest pathlink from S to W is 2.This doesn't contradict anything weproved in part one, because we

established correctness of Dijkstra onlyunder the assumption that all edge linksare non-negative, an assumption which is



violated in this particular example.But again, the takeaway point here isthat, you know, it's easy to write down agreedy algorithm, especially if you cameup with it yourself.You probably believe deep in your heartthat it's got to be correct all the time,

but more often than not, probably yourgreedy heuristic is nothing more than aheuristic.And there will be instances in which itdoes the wrong thing.So keep that in mind in greedy algorithmdesign.So now that my conscience is clear,having warned you about the perils ofgreedy algorithm design, let's turn toproofs of correctness.That is if you have a greedy algorithm

that is correct.And we will see some notable examples inthe coming lectures.How would you actually establish thateffect?Or if you have a greedy algorithm, andyou don't know whether or not it iscorrect,how would you approach trying tounderstand which one it is, whether it'scorrect or not?So let me level with you.Proving greedy algorithm is correct.

Frankly, is sort of, more art thanscience.So, unlike the divide and conquerparadigm, where everything was somewhatformulaic.We had these black box ways of evaluatingrecurrences.We had this sort of, template for provingalgorithms correct.Really, proving correctness of greedyalgorithms takes a lot of creativity.And it has a bit of an ad hoc flavor.That said, as usual, to the extent thatthey are recurring themes.That is what I will spend our timetogether emphasizing.So let me tell you just again about veryhigh level.How you might go about this.You, again, might want to revisit thiscontext aft-, content after you've seensome examples where I think it'll make alot more sense.So method one is our old friend orperhaps nemesis depending on your

disposition, namely proofs by induction.Now for a greedy algorithms remember whatthey do, they sequentially make a bunch



of irrevocable decisions, so here theinduction is going to be on decisionsmade by the algorithm.And if you go back to our proof ofcorrectness of Dijkstra's algorithm, thatin fact is exactly how we provedDijkstra's algorithm correct.

It was by induction of the number ofiterations, in each iteration of the mainwild loop.Computed the shortest path to one newdestination.And we always proof that assuming all ofour previous computations were correct,that's the inductive hypothesis.Then so is the computation in the currentiteration.And so then by induction, everything thealgorithm ever does is correct.

So that's a greedyproof by induction that a greedyalgorithm can be correct.And we might see some more examples ofthose in, for other algorithms in thelectures to come.Some of the text books call this methodof proof greedy stays ahead,meaning you always proof greedy's doingthe right thing iteration by iteration.So a second approach to approving thecorrectness of greedy algorithms whichworks in a lot of cases is what's called

an exchange argument.So you haven't yet seen any examples ofexchange arguments in this class so Ican't really tell you what they are butthat's what we're going to proceed next.I'm going to argue by an exchangeargument that a couple of differencefamous greedy algorithms are in factcorrected.It has a couple of different flavors oneflavor is to approach it bycontradiction.You assume for contradiction that agreedy algorithm is incorrect and thenyou show that you can take an optimalsolution and exchange two elements ofthat optimal solution and get somethingeven better which of course contradictsthe assumption that you started with anoptimal solution.a different flavor would be to graduallyexchange an optimal solution into the oneoutput by a greedy algorithm withoutmaking the solution any worse.That would show that the output of the

greedy algorithm is in fact optimal.And formally that's done by an inductionon the number of exchanges required to



transfer an optimum solution into yours.And finally, I've already said it once,but let me say it again, there's not awhole lot of formula behind provinggreedy algorithms correct, you often haveto be quite creative, you might have tostitch together aspects of method one and

method two, you might have to dosomething completely different.Really, any rigorous proof is fair game.

3 - 1 - introduction to greedy algorithms (13 min)

Documents