transform and conquer

Design and Analysis of Algorithms - Chapter 6 1

Transform and ConquerTransform and Conquer

Solve problem by transforming into:Solve problem by transforming into: a more convenient instance of the same problem a more convenient instance of the same problem

((instance simplificationinstance simplification))• Presorting, Gaussian elimination, matrix inversion, Presorting, Gaussian elimination, matrix inversion,

determinant computationdeterminant computation a different representation of the same instance a different representation of the same instance

((representation changerepresentation change))• balanced search trees, heaps and heapsort, polynomial balanced search trees, heaps and heapsort, polynomial

evaluation by Horner’s rule, Fast Fourier Transformevaluation by Horner’s rule, Fast Fourier Transform a different problem altogether (a different problem altogether (problem reductionproblem reduction))

• reductions to graph problems, linear programming reductions to graph problems, linear programming


Instance simplification - Instance simplification - PresortingPresorting

Solve instance of problem by transforming into Solve instance of problem by transforming into another simpler/easier instance of the same problemanother simpler/easier instance of the same problem

Presorting:Presorting:Many problems involving lists are easier when list is Many problems involving lists are easier when list is

sorted.sorted. element uniquenesselement uniqueness computing the modecomputing the mode finding repeated elementsfinding repeated elements searching searching computing the median (selection problem)computing the median (selection problem)


Selection ProblemSelection Problem

Find the Find the kk-th-th smallest element in A[1],…A[smallest element in A[1],…A[nn]. ]. • minimum: k minimum: k = 1= 1• maximum: k maximum: k = = nn• median: k median: k = = nn/2/2

Presorting-based algorithm Presorting-based algorithm • sort listsort list• return A[return A[kk]]

Partition-based algorithm (decrease & conquer):Partition-based algorithm (decrease & conquer):• pivot/split at A[pivot/split at A[ss] using partitioning algorithm] using partitioning algorithm• if if ss==k k return A[ return A[ss] ] • else if else if s<k s<k repeat with sublist A[repeat with sublist A[ss+1],…A[+1],…A[nn]. ]. • else if s>k else if s>k repeat with sublist A[1],…A[repeat with sublist A[1],…A[ss-1]. -1].


Notes on Selection ProblemNotes on Selection Problem

Presorting-based algorithm: Presorting-based algorithm: ΩΩ((nnlglgnn) + ) + ΘΘ(1) = (1) = ΩΩ((nnlglgnn) ) Partition-based algorithm (decrease & conquer):Partition-based algorithm (decrease & conquer):

• worst case: T(worst case: T(nn) =T() =T(nn-1) + (-1) + (nn+1) -> +1) -> ΘΘ((nn22) )

• best case: best case: ΘΘ((nn) )

• average case: T(average case: T(nn) =T() =T(nn/2) + (/2) + (nn+1) -> +1) -> ΘΘ((nn) )

• Bonus:Bonus: also identifies the also identifies the kk smallest elements smallest elements

Special cases max, min: better, simpler linear Special cases max, min: better, simpler linear algorithm (brute force)algorithm (brute force)

Conclusion:Conclusion: Presorting does Presorting does notnot help in this case. help in this case.


Finding repeated elementsFinding repeated elements

Presorting-based algorithm: Presorting-based algorithm: • use mergesort (optimal): use mergesort (optimal): ΘΘ((nnlglgnn) ) • scan array to find repeated scan array to find repeated adjacentadjacent elements: elements:

ΘΘ((nn) ) • in total it makes: in total it makes: ΘΘ((nnlglgnn) )

Brute force algorithm: Brute force algorithm: ΘΘ((nn22) ) Conclusion:Conclusion: Presorting yields Presorting yields significantsignificant

improvementimprovement


Checking element uniquenessChecking element uniqueness

Brute force algorithm: Brute force algorithm: ΘΘ((nn22) ) AlgorithmAlgorithm PresortedElementUniqueness PresortedElementUniqueness

Sort the array ASort the array A

for i for i 0 to n-2 do 0 to n-2 do

if A[i]=A[i+1] return falseif A[i]=A[i+1] return false

else return trueelse return true Conclusion:Conclusion: Presorting again improves Presorting again improves Similar improvement for modeSimilar improvement for mode


Checking modeChecking mode

ModeMode is the most often met element is the most often met element Brute force: scan the list and compute the frequencies. Brute force: scan the list and compute the frequencies.

Then find the largest frequencyThen find the largest frequency AlgorithmAlgorithm PresortedMode PresortedMode

Sort the array ASort the array Ai i 1; modefrequency 1; modefrequency 0; 0;while i while i n-1 do n-1 do

runlengthrunlength1; runvalue1; runvalueA[i];A[i];while i+runlength≤n-1 and A[i+runlength]=runvalue while i+runlength≤n-1 and A[i+runlength]=runvalue

runlength runlength runlength+1 runlength+1if runlength> modefrequencyif runlength> modefrequency

modefrequencymodefrequencyrunlength, modevaluerunlength, modevaluerunvaluerunvalueiii+runlengthi+runlength

return modevaluereturn modevalue Conclusion:Conclusion: Presorting again improves Presorting again improves


Gaussian EliminationGaussian Elimination

Given a system of two linear equations with two Given a system of two linear equations with two unknownsunknowns

aa1111x + ax + a1212y = by = b11

aa2121x + ax + a2222y = by = b22

It has a unique solution unless the coefficients are It has a unique solution unless the coefficients are proportional proportional

Express one variable as function of the other and Express one variable as function of the other and substitute to solve one equation.substitute to solve one equation.

What if the system has What if the system has nn equations and equations and nn unknowns ? unknowns ?


Gaussian Elimination (2)Gaussian Elimination (2)

Transform Ax=b to A’x=b’, where A’ upper triangularTransform Ax=b to A’x=b’, where A’ upper triangular Then, solution is possible with backward substitutionThen, solution is possible with backward substitution Elementary operationsElementary operations

• Exchange equationsExchange equations

• Replace an equation with a nonzero multipleReplace an equation with a nonzero multiple

• Replace an equation with a sum or difference of this equation and Replace an equation with a sum or difference of this equation and some multiple of another equationsome multiple of another equation

ExampleExample

2x2x11 - x - x22 + x + x33 = 1 = 1

4x4x11 + x + x22 - x - x33 = 5 = 5

xx11 + x + x22 + x + x33 = 0 = 0


Gaussian Elimination (3)Gaussian Elimination (3)

Algorithm GaussEliminationAlgorithm GaussElimination

for i for i 1 to n do A[i,n+1] 1 to n do A[i,n+1]b[i]b[i]

for ifor i 1 to n-1 do 1 to n-1 do

for jfor ji+1 to n doi+1 to n do

for kfor ki to n+1 doi to n+1 do

A[j,k]A[j,k]A[j,k]-A[i,k]*A[j,i]/A[i,i]A[j,k]-A[i,k]*A[j,i]/A[i,i]

Potential problems if A[i,i] is zero or very smallPotential problems if A[i,i] is zero or very small


Gaussian Elimination (partial Gaussian Elimination (partial pivoting)pivoting)

Algorithm GaussElimination2Algorithm GaussElimination2for i for i 1 to n do A[i,n+1] 1 to n do A[i,n+1]b[i]b[i]for ifor i 1 to n-1 do 1 to n-1 do

pivotrowpivotrowiifor jfor ji+1 to n doi+1 to n do

if |A[j,i]|>|A[pivot,i]| pivotrowif |A[j,i]|>|A[pivot,i]| pivotrowjjfor kfor ki to n+1 doi to n+1 do

swap(A[i,k],A[pivotrow,k])swap(A[i,k],A[pivotrow,k])for jfor ji+1 to n doi+1 to n do

temptempA[j,i]/A[i,i]A[j,i]/A[i,i]for kfor kI to n+1 doI to n+1 do

A[j,k]A[j,k]A[j,k]-A[I,k]*tempA[j,k]-A[I,k]*temp EfficiencyEfficiency


LU decompositionLU decomposition

Byproduct of Gaussian EliminationByproduct of Gaussian Elimination Example A=LUExample A=LU

LUx=b. Denote y=Ux LUx=b. Denote y=Ux Ly=b Ly=b Solve Ly=b, then solve Ux=ySolve Ly=b, then solve Ux=y Solve with as many times with different b’s.Solve with as many times with different b’s. No extra spaceNo extra space

11 00 00

22 11 00

1/21/2 1/21/2 11

22 -1-1 11

00 33 -3-3

00 00 22


Computing a matrix inverseComputing a matrix inverse

AAAA-1-1=I=I A A singularsingular matrix does not have an inverse matrix does not have an inverse A matrix is singular if and only if one of the A matrix is singular if and only if one of the

rows is a linear combination of the other rows.rows is a linear combination of the other rows. Apply Gaussian elimination. If it yields an Apply Gaussian elimination. If it yields an

upper-triangular with no zeros on the upper-triangular with no zeros on the diagonal, then the matrix is not singulardiagonal, then the matrix is not singular

AxAxjj=e=ejj


Computing the determinantComputing the determinant

A well-known recursive formulaA well-known recursive formula What if n is large ? Efficiency ?What if n is large ? Efficiency ? Apply Gaussian elimination.Apply Gaussian elimination. The determinant of an upper-triangular The determinant of an upper-triangular

matrix is the product of elements on its matrix is the product of elements on its diagonal.diagonal.

Efficiency ?Efficiency ?

Cramer’s ruleCramer’s rule


Taxonomy of Searching Taxonomy of Searching AlgorithmsAlgorithms

Elementary searching algorithmsElementary searching algorithms• sequential searchsequential search• binary searchbinary search• binary tree searchbinary tree search

Balanced tree searchingBalanced tree searching• AVL treesAVL trees• red-black treesred-black trees• multiway balanced trees (2-3 trees, 2-3-4 trees, B trees)multiway balanced trees (2-3 trees, 2-3-4 trees, B trees)

HashingHashing• separate chainingseparate chaining• open addressingopen addressing


Balanced trees: AVL treesBalanced trees: AVL trees

For every node, difference in height between left For every node, difference in height between left and right subtree is at most 1and right subtree is at most 1

AVL property is maintained through AVL property is maintained through rotationsrotations, , each time the tree becomes unbalancedeach time the tree becomes unbalanced

lg lg nn ≤≤ hh ≤ 1.4404 lg (≤ 1.4404 lg (n n + 2) - 1.3277 + 2) - 1.3277 average:average: 1.01 lg 1.01 lg n + n + 0.1 for large 0.1 for large nn

Disadvantage: needs extra storage for maintaining Disadvantage: needs extra storage for maintaining node balancenode balance

A similar idea: red-black trees (height of subtrees A similar idea: red-black trees (height of subtrees is allowed to differ by up to a factor of 2)is allowed to differ by up to a factor of 2)


AVL tree rotationsAVL tree rotations

Small examples:Small examples:• 1, 2, 31, 2, 3• 3, 2, 1 3, 2, 1 • 1, 3, 21, 3, 2• 3, 1, 23, 1, 2

Larger example: 4, 5, 7, 2, 1, 3, 6Larger example: 4, 5, 7, 2, 1, 3, 6 See figures 6.4, 6.5 for general cases of rotations; See figures 6.4, 6.5 for general cases of rotations;


General case: single R-rotationGeneral case: single R-rotation


Double LR-rotationDouble LR-rotation


Balance factorBalance factor Algorithm maintains Algorithm maintains balance factorbalance factor for each node. For for each node. For

example:example:


HeapsortHeapsort

Definition:Definition:A A heapheap is a binary tree with the following conditions: is a binary tree with the following conditions: it is essentially complete:it is essentially complete:

The key at each node is The key at each node is ≥ keys at its children≥ keys at its children


Definition implies:Definition implies:

Given Given n,n, there exists a unique binary tree with there exists a unique binary tree with nn nodes that is essentially complete, with nodes that is essentially complete, with hh= lg = lg nn

The root has the largest keyThe root has the largest key

The subtree rooted at any node of a heap is also a The subtree rooted at any node of a heap is also a heapheap


Heapsort Algorithm:Heapsort Algorithm:

1.1. Build heapBuild heap

2.2. Remove root –exchange with last (rightmost) leafRemove root –exchange with last (rightmost) leaf

3.3. Fix up heap (excluding last leaf)Fix up heap (excluding last leaf)

Repeat 2, 3 until heap contains just one node.Repeat 2, 3 until heap contains just one node.


Heap constructionHeap construction

Insert elements in the order given breadth-first in Insert elements in the order given breadth-first in a binary treea binary tree

Starting with the last (rightmost) parental node, Starting with the last (rightmost) parental node, fix the heap rooted at it, if it does not satisfy the fix the heap rooted at it, if it does not satisfy the heap condition:heap condition:1.1. exchange it with its largest childexchange it with its largest child

2.2. fix the subtree rooted at it (now in the child’s position)fix the subtree rooted at it (now in the child’s position)

Example: 2 3 6 7 5 9Example: 2 3 6 7 5 9


Root deletionRoot deletion

The root of a heap can be deleted and the heap fixed The root of a heap can be deleted and the heap fixed up as follows:up as follows:

exchange the root with the last leaf exchange the root with the last leaf compare the new root (formerly the leaf) with each compare the new root (formerly the leaf) with each

of its children and, if one of them is larger than the of its children and, if one of them is larger than the root, exchange it with the larger of the two. root, exchange it with the larger of the two.

continue the comparison/exchange with the continue the comparison/exchange with the children of the new root until it reaches a level of children of the new root until it reaches a level of the tree where it is larger than both its children the tree where it is larger than both its children


RepresentationRepresentation

Use an array to store breadth-first traversal of heap tree:Use an array to store breadth-first traversal of heap tree: Example:Example:

Left child of node Left child of node jj is at 2 is at 2jj Right child of node Right child of node jj is at 2 is at 2jj+1+1 Parent of node Parent of node jj is at is at j j /2/2 Parental nodes are represented in the first Parental nodes are represented in the first nn /2 locations /2 locations

9

1

5 3

4 2

1 2 3 4 5 6

9 5 3 1 4 2


Bottom-up heap construction Bottom-up heap construction algorithmalgorithm


Analysis of HeapsortAnalysis of Heapsort

See algorithm HeapBottomUp in section 6.4See algorithm HeapBottomUp in section 6.4 Fix heap with “problem” at height Fix heap with “problem” at height j: j: 2 2jj

comparisonscomparisons For subtree rooted at level For subtree rooted at level ii it does 2( it does 2(hh--ii) )

comparisonscomparisons Total for heap construction phase:Total for heap construction phase:

Σ 2(h-i) 2i = 2 ( n – lg (n + 1)) = Θ(n)i=0

h-1

# nodes at level i


Analysis of Heapsort Analysis of Heapsort (continued)(continued)

Recall algorithm:Recall algorithm:1.1. Build heapBuild heap

2.2. Remove root –exchange with last (rightmost) leafRemove root –exchange with last (rightmost) leaf

3.3. Fix up heap (excluding last leaf)Fix up heap (excluding last leaf)

Repeat 2, 3 until heap contains just one node.Repeat 2, 3 until heap contains just one node.

Θ(n)

Θ(log n)

n – 1 times

Total:Total: Θ(n) + Θ( n log n) = Θ(n log n)

• Note:Note: this is the worst case. Average case also Θ(n log n).


Priority queuesPriority queues

A A priority queue priority queue is the ADT of an ordered set with is the ADT of an ordered set with the operations:the operations:• find element with highest priority find element with highest priority

• delete element with highest priority delete element with highest priority

• insert element with assigned priority insert element with assigned priority

Heaps are very good for implementing priority Heaps are very good for implementing priority queuesqueues


Insertion of a new elementInsertion of a new element

Insert element at last position in heap. Insert element at last position in heap. Compare with its parent and if it violates heap Compare with its parent and if it violates heap

condition exchange themcondition exchange them Continue comparing the new element with nodes Continue comparing the new element with nodes

up the tree until the heap condition is satisfiedup the tree until the heap condition is satisfied

Example:Example:

Efficiency:Efficiency:


Bottom-up vs. Top-down heap Bottom-up vs. Top-down heap constructionconstruction

Top down:Top down: Heaps can be constructed by Heaps can be constructed by successively inserting elements into an (initially) successively inserting elements into an (initially) empty heapempty heap

Bottom-up:Bottom-up: Put everything in and then fix it Put everything in and then fix it

Which one is better?Which one is better?


Horner’s ruleHorner’s rule

Horner published in early 19Horner published in early 19thth century century According to Knuth, the method was used by NewtonAccording to Knuth, the method was used by Newton Evaluate a polynomial at a point xEvaluate a polynomial at a point x

p(x) = ap(x) = annxxnn + a + an-1n-1xxn-1n-1 + … + a + … + a11x + ax + a00

p(x) = ( … (ap(x) = ( … (annx + ax + an-1n-1) x + … )x + a) x + … )x + a00

Example: evaluate p(x)=2xExample: evaluate p(x)=2x44-x-x33+3x+3x22+x-5 at x=3+x-5 at x=3

p(x) = x (x (x (2x-1) + 3) + 1) - 5p(x) = x (x (x (2x-1) + 3) + 1) - 5 Visualization by a tableVisualization by a table


Horner’s rule [2]Horner’s rule [2]

Algorithm Horner(P[0..n],x)Algorithm Horner(P[0..n],x)

// Evaluate polynomial at a given point// Evaluate polynomial at a given point

// Input: an array P[0..n] of coefficients and a number x// Input: an array P[0..n] of coefficients and a number x

//Output: the value of polynomial at point x//Output: the value of polynomial at point x

p p P[n] P[n]

for i for i n-1 down to 0 do n-1 down to 0 do

p p x*p + P[i] x*p + P[i]

return preturn p Efficiency ?Efficiency ? Byproduct: coefficients of the quotient of the division of p(x) Byproduct: coefficients of the quotient of the division of p(x)

by (x-xby (x-x00))


Binary exponentiationBinary exponentiation

Horner is not efficient to compute p(x)=xHorner is not efficient to compute p(x)=xnn at x=a at x=a Degenerate to brute forceDegenerate to brute force Let the binary representation n=bLet the binary representation n=bllbbl-1l-1… b… bii … b … b11bb00

p(x) = bp(x) = bllxxll + b + bl-1l-1xxl-1l-1 + … + b + … + b11x + bx + b00 and x=2 and x=2 Algorithm LeftRightBinaryExponentiationAlgorithm LeftRightBinaryExponentiation

product product a afor i for i l-1 down to 0 do l-1 down to 0 do

product product product * product product * product

if bif bii1 then product 1 then product product*a product*a

return productreturn product Example: compute aExample: compute a1313, n=13=1101, n=13=1101 EfficiencyEfficiency


Binary exponentiation (2)Binary exponentiation (2)

Compute aCompute ann Consider n = bConsider n = bll22ll + b + bl-1l-122l-1l-1 + … + b + … + b112 + b2 + b00

and multiply independent powers termsand multiply independent powers terms Algorithm RightLeftBinaryExponentiationAlgorithm RightLeftBinaryExponentiation

term term a aif bif b00=1 then product =1 then product a aelse product else product 1 1for ifor i 1 to l do 1 to l do

term term term * term term * termif bif bi i = 1 then product = 1 then product product * term product * term

return productreturn product Example: compute aExample: compute a1313, n=13=1101, n=13=1101 EfficiencyEfficiency


Least common multipleLeast common multiple

lcm(24,60)=120, lcm(11,5)=55lcm(24,60)=120, lcm(11,5)=55 Example: 24 = 2 x 2 x 2 x 3Example: 24 = 2 x 2 x 2 x 3

60 = 2 x 2 x 3 x 560 = 2 x 2 x 3 x 5

lcm(24,60) = (2x2x3) x 2 x 5lcm(24,60) = (2x2x3) x 2 x 5 Efficiency (a list of primes is required)Efficiency (a list of primes is required)

lcm(m,n) = mn / gcd(m,n)lcm(m,n) = mn / gcd(m,n)


Counting paths in a graphCounting paths in a graph

The number of different paths of length k>0 from The number of different paths of length k>0 from node i to node j equals the (i,j) element of the Anode i to node j equals the (i,j) element of the Akk, , where A the adjacency matrix where A the adjacency matrix

Example Example EfficiencyEfficiency


Reduction to graph problemsReduction to graph problems

Applies for a variety of games and puzzlesApplies for a variety of games and puzzles Build the state-space graphBuild the state-space graph

Example: peasant, wolf, goat, cabbageExample: peasant, wolf, goat, cabbage

Traverse the graph by applying what?Traverse the graph by applying what?

transform and conquer

Documents