greedy algorithms

CS3381 Des & Anal of Alg (2001-2002 SemA)City Univ of HK / Dept of CS / Helena Wong 5. Greedy Algorithms - 1http://www.cs.cityu.edu.hk/~helena

Greedy Algorithms


Greedy Algorithms

Coming upCasual Introduction: Two Knapsack Problems

An Activity-Selection Problem

Greedy Algorithm Design

Huffman Codes(Chap 16.1-16.3)


2 Knapsack Problems

A thief robbing a store finds n items.ith item: worth vi dollars

wi pounds

W, wi, vi are integers.

He can carry at most W pounds.

1. 0-1 Knapsack Problem:

Which items should I

take?


2 Knapsack Problems

A thief robbing a store finds n items.ith item: worth vi dollars

wi pounds

W, wi, vi are integers.

He can carry at most W pounds.He can take fractions of items.

2. Fractional Knapsack Problem:

?


2 Knapsack ProblemsDynamic Programming Solution

If jth item is removed from his load,

Both problems exhibit the optimal-substructure property:

Consider the most valuable load that weighs at most W pounds.

the remaining load must be the most valuable load weighting at most W-wj that he can take from the n-1 original items excluding j.

=> Can be solved by dynamic programming


2 Knapsack Problems Dynamic Programming Solution

Example: 0-1 Knapsack Problem

Suppose there are n=100 ingots:30 Gold ingots: each $10000, 8 pounds (most expensive)20 Silver ingots: each $2000, 3 pound per piece50 Copper ingots: each $500, 5 pound per piece

Then, the most valuable load for to fill W pounds = The most valuable way among the followings:

(1) take 1 gold ingot + the most valuable way to fill W-8 pounds from 29 gold ingots, 20 silver ingots and 50 copper ingots

(2) take 1 silver ingot + the most valuable way to fill W-3 pounds from 30 gold ingots, 19 silver ingots and 50 copper ingots

(3) take 1 copper ingot + the most valuable way to fill W-5 pounds from 30 gold ingots, 20 silver ingots and 49 copper ingots


2 Knapsack Problems Dynamic Programming Solution

Example: Fractional Knapsack Problem

Suppose there are totally n = 100 pounds of metal dust:30 pounds Gold dust: each pound $10000 (most expensive)20 pounds Silver dust: each pound $200050 pounds Copper dust: each pound $500

Then, the most valuable way to fill a capacity of W pounds = The most valuable way among the followings:

(1) take 1 pound of gold + the most valuable way to fill W-1 pounds from 29 pounds of gold, 20 pounds of silver, 50 pounds of copper

(2) take 1 pound of silver + the most valuable way to fill W-1 pounds from 30 pounds of gold, 19 pounds of silver, 50 pounds of copper

(3) take 1 pound copper + the most valuable way to fill W-1 pounds from 30 pounds of gold, 20 pounds of silver, 49 pounds of copper


2 Knapsack ProblemsBy Greedy Strategy

Both problems are similar. But Fractional Knapsack Problem can be solved in a greedy strategy.

Step 1. Compute the value per pound for each itemEg. gold dust: $10000 per pound (most expensive) Silver dust: $2000 per poundCopper dust: $500 per pound

Step 2. Take as much as possible of the most expensive (ie. Gold dust)

Step 3. If the supply of that item is exhausted (ie. no more gold) and he can still carry more, he takes as much as possible of the item that is next most expensive and so forth until he can’t carry any more.


Knapsack Problems By Greedy Strategy

We can solve the Fractional Knapsack Problem by a greedy algorithm:Always makes the choice that looks best at the moment.ie. A locally optimal Choice

To see why we can’t solve 0-1 Knapsack Problem by greedy

strategy, read Chp 16.2.


Greedy Algorithms2 techniques for solving optimization problems:1. Dynamic Programming2. Greedy Algorithms (“Greedy Strategy”)

Greedy Approach can solve these problems:

For the optimization problems:

Dynamic Programming can solve these problems:

For some optimization problems, Dynamic Programming is “overkill”Greedy Strategy is simpler and more efficient.


Activity-Selection ProblemFor a set of proposed activities that wish to use a lecture hall, select a maximum-size subset of “compatible activities”.

Set of activities: S={a1,a2,…an}

Duration of activity ai: [start_timei, finish_timei)

Activities sorted in increasing order of finish time:i 1 2 3 4 5 6 7 8 9 10 11start_timei 1 3 0 5 3 5 6 8 8 2 12

finish_timei 4 5 6 7 8 9 10 11 12 13 14


Activity-Selection Problemi 1 2 3 4 5 6 7 8 9 10 11start_timei 1 3 0 5 3 5 6 8 8 2 12

finish_timei 4 5 6 7 8 9 10 11 12 13 14

Compatible activities:{a3, a9, a11},{a1,a4,a8,a11},{a2,a4,a9,a11}

01234567891011121314

time a1 a2 a3 a4 a5 a6 a7 a8 a9 a10 a11


Activity-Selection ProblemDynamic Programming Solution (Step 1)

Step 1. Characterize the structure of an optimal solution.

S: i 1 2 3 4 5 6 7 8 9 10 11(=n)start_timei 1 3 0 5 3 5 6 8 8 2 12finish_timei 4 5 6 7 8 9 10 11 12 13 14

eg

Definition: Sij={akS: finish_timeistart_timek<finish_timek start_timej}

01234567891011121314

time a1 a2 a3 a4 a5 a6 a7 a8 a9 a10 a11Let Si,j be the set of activities that

start after ai finishes and

finish before aj starts.

eg. S2,11=

01234567891011121314



time a1 a2 a3

okok

a4 a5

okokokok

a6

okokokok

a7

okokok

a8

okokokok

a9 a10 a11



S: i 1 2 3 4 5 6 7 8 9 10 11(=n)start_timei 1 3 0 5 3 5 6 8 8 2 12finish_timei 4 5 6 7 8 9 10 11 12 13 14

Add fictitious activities: a0 and an+1:

S: i 0 1 2 3 4 5 6 7 8 9 10 11 12start_timei 1 3 0 5 3 5 6 8 8 2 12 finish_timei 0 4 5 6 7 8 9 10 11 12 13 14

ie. S0,n+1

={a1,a2,a3,a4,a5,a6,a7,a8,a9,a10,a11} = S

Note: If i>=j then Si,j=Ø

0123456789

1011121314

time a1 a2 a3 a4 a5 a6 a7 a8 a9 a10 a11a0 a12


Substructure:


Suppose a solution to Si,j includes activity ak,

then,2 subproblems are generated: Si,k, Sk,j

The problem:For a set of proposed activities that wish to use a lecture hall, select a maximum-size subset of “compatible activities

Select a maximum-size subset of compatible activities from S0,n+1.

=

0123456789

1011121314


1011121314


The maximum-size subset Ai,j

of compatible activities is:

Ai,j=Ai,k U {ak} U Ak,jSuppose a solution to S0,n+1 contains a7, then, 2 subproblems are generated: S0,7 and S7,n+1



Step 2. Recursively define an optimal solution

Let c[i,j] = number of activities in a maximum-size subset of compatible activities in Si,j.

If i>=j, then Si,j=Ø, ie. c[i,j]=0.0 if Si,j=ØMaxi<k<j {c[i,k] + c[k,j] + 1} if Si,jØc(i,j) =

Step 3. Compute the value of an optimal solution in a bottom-up fashion

Step 4. Construct an optimal solution from computed information.


Activity-Selection ProblemGreedy Strategy Solution

Consider any nonempty subproblem Si,j, and let am be the activity in Si,j with the earliest finish time.

01234567891011121314

time a1 a2 a3

okok

a4 a5

okokokok

a6

okokokok

a7

okokok

a8

okokokok

a9 a10 a11

eg. S2,11={a4,a6,a7,a8,a9}

Among {a4,a6,a7,a8,a9}, a4 will finish earliest

1. A4 is used in the solution

2. After choosing A4, there are 2 subproblems: S2,4 and S4,11. But S2,4 is empty. Only S4,11

remains as a subproblem.

Then,1. Am is used in some maximum-

size subset of compatible activities of Si,j.

2. The subproblem Si,m is empty, so that choosing am leaves the subproblem Sm,j as the only one that may be nonempty.

0 if Si,j=Ø

Maxi<k<j {c[i,k]+c[k,j]+1} if Si,jØc(i,j) =



That is,To solve S0,12, we select a1 that will finish earliest, and solve for S1,12.

To solve S1,12, we select a4 that will finish earliest, and solve for S4,12.

To solve S4,12, we select a8 that will finish earliest, and solve for S8,12.

…Greedy Choices (Locally optimal choice)To leave as much opportunity as possible for the remaining activities to be scheduled.

Solve the problem in a top-down fashion

Hence, to solve the Si,j:

1. Choose the activity am with the earliest finish time.

2. Solution of Si,j = {am} U Solution of subproblem Sm,j

0123456789

1011121314



Recursive-Activity-Selector(i,j)1 m = i+1

// Find first activity in Si,j

2 while m < j and start_timem < finish_timei 3 do m = m + 14 if m < j5 then return {am} U Recursive-Activity-Selector(m,j)6 else return Ø


Order of calls:Recursive-Activity-Selector(0,12)Recursive-Activity-Selector(1,12)Recursive-Activity-Selector(4,12)Recursive-Activity-Selector(8,12)Recursive-Activity-Selector(11,12)

0123456789

1011121314


m=2

Okay

m=3

Okaym=4

break the loop

Ø{11}

{8,11}{44,8,11}

{1,44,8,11}


Iterative-Activity-Selector()1 Answer = {a1}2 last_selected=13 for m = 2 to n4 if start_timem>=finish_timelast_selected

5 then Answer = Answer U {am}6 last_selected = m7 return Answer


0123456789

1011121314




For both Recursive-Activity-Selector and

Iterative-Activity-Selector,

Running times are (n)

Reason: each am are examined once.


Greedy Algorithm DesignSteps of Greedy Algorithm Design:

1. Formulate the optimization problem in the form: we make a choice and we are left with one subproblem to solve.

2. Show that the greedy choice can lead to an optimal solution, so that the greedy choice is always safe.

3. Demonstrate that an optimal solution to original problem = greedy choice + an optimal solution to the subproblem

Optimal Substructure

Property

Greedy-Choice

Property

A good clue that that a greedy

strategy will solve the problem.


Greedy Algorithm DesignComparison:

Dynamic Programming Greedy Algorithms

At each step, the choice is determined based on solutions of subproblems.

At each step, we quickly make a choice that currently looks best. --A local optimal (greedy) choice.

Bottom-up approach Top-down approach

Sub-problems are solved first. Greedy choice can be made first before solving further sub-problems.

Can be slower, more complex Usually faster, simpler


Huffman CodesHuffman Codes • For compressing data (sequence of characters)• Widely used • Very efficient (saving 20-90%)• Use a table to keep frequencies of occurrence

of characters.• Output binary string.

“Today’s weather is nice”

“001 0110 0 0 100 1000 1110”


Huffman CodesFrequency Fixed-length Variable-length

codeword codeword ‘a’ 45000 000 0‘b’ 13000 001 101‘c’ 12000 010 100‘d’ 16000 011 111‘e’ 9000 100 1101‘f’ 5000 101 1100

Example:

A file of 100,000 characters. Containing only ‘a’ to ‘e’

300,000 bits1*45000 + 3*13000 + 3*12000 +

3*16000 + 4*9000 + 4*5000 = 224,000 bits

1*45000 + 3*13000 + 3*12000 + 3*16000 + 4*9000 + 4*5000

= 224,000 bits

eg. “abc” = “000001010” eg. “abc” = “0101100”


01a:45 b:13 c:12 d:16 e:9 f:5

0

0

0

0

0

1

1

1 1

Huffman CodesThe coding schemes can be represented by trees:

Frequency Fixed-length(in thousands) codeword

‘a’ 45 000‘b’ 13 001‘c’ 12 010‘d’ 16 011‘e’ 9 100‘f’ 5 101100

86 14

1401

58 28

a:45 b:13 c:12 d:16 e:9 f:5

0

0

0

0

0

1

1

1 1

Frequency Variable-length

(in thousands) codeword

‘a’ 45 0‘b’ 13 101‘c’ 12 100‘d’ 16 111‘e’ 9 1101‘f’ 5 1100100

55

0125 30

0

0

0

1

1

1

a:45

14

e:9 f:50 1

d:16b:13 c:1201

a:45 b:13 c:12 d:16 e:9 f:5

0

0

0

0

0

1

1

1 114

0158 28

a:45 b:13 c:12 d:16 e:9 f:5

0

0

0

0

0

1

1

1 1

86 14

1401

58 28

a:45 b:13 c:12 d:16 e:9 f:5

0

0

0

0

0

1

1

1 1

Not a fullbinary tree

A full binary treeevery nonleaf node

has 2 children

A file of 100,000 characters.


Huffman Codes Frequency Codeword

‘a’ 45000 0‘b’ 13000 101‘c’ 12000 100‘d’ 16000 111‘e’ 9000 1101‘f’ 5000 1100

100

55

0125 30

0

0

0

1

1

1

a:45

14

e:9 f:50 1

d:16b:13 c:12

To find an optimal code for a file:

1. The coding must be unambiguous.Consider codes in which no codeword is also a prefix of other codeword. => Prefix CodesPrefix Codes are unambiguous.Once the codewords are decided, it is easy to compress (encode) and decompress (decode).

2. File size must be smallest.=> Can be represented by a full binary tree.=> Usually less frequent characters are at bottom

Let C be the alphabet (eg. C={‘a’,’b’,’c’,’d’,’e’,’f’})For each character c, no. of bits to encode all c’s occurrences = freqc*depthc

File size B(T) = cCfreqc*depthcEg. “abc” is coded as “0101100”


Huffman CodesHuffman code (1952) was invented to solve it.

A Greedy Approach.

Q: A min-priority queue f:5 e:9 c:12 b:13 d:16 a:45

100

55

25 30

a:45

14

e:9 f:5

d:16b:13 c:12

c:12 b:13 d:16 a:4514

f:5 e:9

d:16

a:45

14

25

c:12 b:13

30

f:5 e:9

a:45

d:1614

25

c:12 b:13

30

55

f:5 e:9

d:16 a:4514 25

c:12 b:13f:5 e:9

How do we find the optimal prefix code?


Huffman Codes

HUFFMAN(C)1 Build Q from C2 For i = 1 to |C|-13 Allocate a new node z4 z.left = x = EXTRACT_MIN(Q)5 z.right = y = EXTRACT_MIN(Q)6 z.freq = x.freq + y.freq7 Insert z into Q in correct position.8 Return EXTRACT_MIN(Q)

Q: A min-priority queue f:5 e:9 c:12 b:13 d:16 a:45

c:12 b:13 d:16 a:4514

f:5 e:9

d:16 a:4514 25

c:12 b:13f:5 e:9

….

If Q is implemented as a binary min-heap, “Build Q from C” is O(n)“EXTRACT_MIN(Q)” is O(lg n)“Insert z into Q” is O(lg n)

Huffman(C) is O(n lg n)

How is it “greedy”?


Greedy Algorithms

SummarySummaryCasual Introduction: Two Knapsack Problems

An Activity-Selection Problem

Greedy Algorithm DesignSteps of Greedy Algorithm DesignOptimal Substructure PropertyGreedy-Choice PropertyComparison with Dynamic Programming

Huffman Codes

greedy algorithms

Documents