Introduction to Algorithms

Lecture 3 – Divide and Conquer

The Methodology• Divide-and-Conquer is a useful paradigm.• Main idea– Divide the problem into k > 2 sub-problems.– Solve the sub-problems.– Combine the solutions of the sub-problems to

arrive at a solution to the original problem.

The Methodology• Divide-and-Conquer is a useful paradigm.• Main idea– Divide the problem into k > 2 sub-problems.– Solve the sub-problems, usually recursively.– Combine the solutions of the sub-problems to

arrive at a solution to the original problem.

The Methodology

• When using recursion to solve the subproblems, the analysis for running time is also natural.

• Let the size of the original problem be n and T(n) be the time

• D(n) : Time taken to divide the problem. Let there be k subproblems of sizes n1, n2, …,nk.

• To solve the ith subproblem, the time taken would be T(ni).

• C(n): Time taken to combine the solutions.

)()()()(1

nCnDnTnTk

ii

Examples• Merge Sort– Division is easy. D(n) = O(1). We get two

subproblems of size n/2 each.– Each subproblems take T(n/2) time.– Combine step takes O(n) time.– Recurrence relation is T(n) = 2T(n/2) + O(n)– Solution using Master’s theorem is T(n) = O(nlog n).

Examples• Quick sort (assuming a ``good’’ pivot)– Divide : Uses the partition algorithm. Take time

O(n).– Solving the subproblems is now 2T(n/2)– Combine: Nothing to do here.., so O(1).– Recurrence relation: T(n) = 2T(n/2) + O(n).– Solution using recurrence relation again is T(n) =

O(nlog n)

Examples• The selection algorithm from previous lecture.• The subproblem sizes are not equal.• No combination step, but an O(n) time to

divide.

A Complete Example• Let us consider the multiplication of two

square matrices A and B.• Suppose we know just to multiply scalar

values.• And we know the scheme of matrix

multiplication.

Matrix Multiplication• Can use divide and conquer as follows.• Divide the matrices into 4 parts.

X =

A B C

A00

A10

A01

A11

B00

B10

B01

B11

C00

C10

C01

C11

C00 = A00 x B00 + A10 x B01 C00 = A00 x B00 + A10 x B01

C00 = A00 x B00 + A10 x B01 C00 = A00 x B00 + A10 x B01

Matrix Multiplication• The recurrence relation can be obtained as

follows.• Divide time is O(1).• There are four subproblems. And each

subproblem is of size n/4.• Combine time is O(n2).• The recurrence relation is T(n) = 8T(n/2) + O(n2)• The solution is T(n) = O(n3).

A Better Algorithm• Surprisingly, you can do better than O(n3).• Recently, there is an O(n2.23) algorithm.• In fact, the lower bound is still open.• We will get somewhere close to that.

A Slight Detour - Motivation• Consider two complex numbers a+ib, and c+id

that we need to multiply.• The result is (ac-bd) + i(ad+bc).– Requires four multiplications and two additions.

• Suppose that multiplications are expensive compared to additions.

• Can we save on one multiplication?– Can use more additions or subtractions.

Motivation• Consider the products P1 = ac, P2 = bd, and P3

= (a+b)(c+d).• Now, the result of multiplication is (P1-P2) +

i(P3-P2-P1).• We have 3 multiplications and 5 additions/

subtractions.• A similar idea can be used for matrix

multiplication too.

Motivations• Question: Verify quickly with two complex

numbers of your choice.

Motivation• Is there a practical reason to use the new

approach even if multiplication is as expensive as an addition/subtraction?

A Better Algorithm• Notice that matrix addition is easier than

matrix multiplication.• So, we can trade-off some matrix

multiplications for matrix additions, and subtractions.

A Better Algorithm• Let A11, A12, A21, and A22 be the four

submatrices of A, each of size n/2xn/2.• Let B11, B12, B21, and B22 be the four

submatrices of B, each of size n/2xn/2.• We need,

C11 = A11 x B11 + A12 x B21 C12 = A11 x B12 + A12 x B22 C21 = A21 x B11 + A22 x B21 C22 = A21 x B12 + A22 x B22

A Better Algorithm• Strassen computes each of the eight AixBj

using linera combinations of the following seven matrix products.

• P1 = A11(B12 − B22)• P2 = (A11 + A12)B22• P3 = (A21 + A22)B11• P4 = A22(B21 − B11)• P5 = (A11 + A22)(B11 + B22)• P6 = (A12 − A22)(B21 + B22)• P7 = (A11 − A21)(B11 + B12)

Strassen’s Algorithm• Question: Verify that • C11 =P5 + P4 − P2 + P6• C12 = P1 + P2• C21 =P3 + P4• C22 =P1 + P5 − P3 − P7

A Better Algorithm• Asymptotically, Strassen’s algorithm works

better than the O(n3) standard algorithm.• In reality, notice that one has to perform 14

matrix additions and seven matrix products.• So, in practice, the method may be slow due

to the large constants involved in the O notation.

A Better Algorithm• Strassen saves one multiplication by introducing

more additions.• Thus, the recurrence relation would be T(n) = 7T(n/2) + O(n2).• Question: Solve for T(n) using Master’s theorem.

A Better Algorithm• Strassen saves one multiplication by introducing

more additions.• Thus, the recurrence relation would be T(n) = 7T(n/2) + O(n2).• Using Master’s theorem, the solution now is

T(n) = O(nlog2 7).

• Notice that log2 7 < 3. Hence, the algorithm beats the n3 time for the earlier algorithm.

Yet Another Example• We will consider sorting again. • But this time as a network/hardware for

sorting.• There are some nice principles along the way.

Comparators• A comparator is a hardware element that

takes two inputs and produces two outputs.

• We assume that a comparator operates in O(1) time.

• Can connect comparators to form a network.

Comparatorx

y

Min(x,y)

Max(x,y)

A Network

Sorting Network• A comparator network that takes n inputs and

produces a sorted rearrangement of the inputs.

Sorting Network• Size of the network : The number of

comparators used.

• Depth of a network is defined as follows.• Let the depth of a input wire be 0.• The depth of a comparator with inputs as

depth dx and dy will be max(dx, dy) + 1.• The depth of a network is the maximum depth

of a comparator in the network.

How to Build a Sorting Network• Bitonic Sequence: A sequence that monotonically

increases and then decreases monotonically.– More formally, has two indices i and j such that A(i: j-

1) is monotonically increasing, and A(j:i-1) is monotonically decreasing. • Corresponds to a shifting of A so as to get the intuitive

definition.

• Example: 11, 16, 20, 23, 34, 17, 9, 5, 1• We first construct a sorting network that can sort

input sequences that are bitonic.• We then show how to use such a network to sort

arbitrary sequences.

Sorting Network – Bitonic Sequences• Suppose that a sequence X is bitonic.• Let L(X) = {min{xi, xn/2+i}, 0 ≤ i ≤ n/2-1}.

• Let R(X) = {max{xi, xn/2+i}, 0 ≤ i ≤ n/2-1}. • Then both L(X) and R(X) are bitonic, and every

element of L(X) has a value at most the value of any element of R(X).

How to Use the Above Theorem• What the above theorem says is that a bitonic

sequence can be “divided” into two bitonic sequences of equal length, with– One sequence containing elements that are all

smaller than the other.• Can be applied recursively to divide sequences

further into smaller subsequences.• Eventually, sorts the input bitonic sequence.

In Pictures

• We have that any of min(a1,a5) to min(a4,a8) is SMALLER than any of the other four.

Put Them Together..

Bitonic Sorter• Has a depth of O(log n)• Has O(nlog n) comparators.

• But does not work for arbitrary sequences.• Also, have used a property that is yet to be

proved.

Sorting Network – Bitonic Sequences• Suppose that a sequence X is bitonic.• Let L(X) = {min{xi, xn/2+i}, 0 ≤ i ≤ n/2-1}.

• Let R(X) = {max{xi, xn/2+i}, 0 ≤ i ≤ n/2-1}. • Then both L(X) and R(X) are bitonic, and every

element of L(X) has a value at most the value of any element of R(X).

• Can be shown by the unique crossover property.

Unique Crossover Property

Unique Crossover Property• Let X be a bitonic sequence. The crossover property

states that there exists an index i such that:1. for any a in {x0, x1, …, xi-1} and for any b in {xn/2+i,

xn/2+i-1 , …, xn/2}, it holds that a ≤ b.

2. for any a in {xi, xi+1, …, xn/2-1} and for any b in {xn/2+i, xn/2+i+1, …, xn-1} , it holds that a > b .

3. or any a in {x0, x1, …, xi-1} and for any b in {xi, xi+1, …, xn/2-1} , it holds that a ≤ b , and

4. for any a in {xn/2, xn/2+1, …, xn/2+i-1} and for any b in {xn/2+i, xn/2+i+1, …, xn-1}, it holds that a ≥ b.

Unique Crossover Property• The proof is much simpler than the statement.• Let i be the smallest index such that xi > xn/2+i

with 0 ≤ i ≤ n/2-1. • It then holds that for any 0 ≤ j ≤ i-1, x0 ≤ x1 ≤ ...

≤ xi-1 ≤ xn/2+i-1 ≤ xn/2+i-2 ≤... ≤ xn/2. • Question: Why?

Unique Crossover Property• The proof is much simpler than the statement.• Let i be the smallest index such that xi > xn/2+i

with 0 ≤ i ≤ n/2-1. • It then holds that for any 0 ≤ j ≤ i-1, x0 ≤ x1 ≤ ...

≤ xi-1 ≤ xn/2+i-1 ≤ xn/2+i-2 ≤... ≤ xn/2. • Question: Why? • So property (1) holds. • Question: Verify the other three properties.

From Bitonic to Arbitrary• Recall that every 1, or 2 element sequence is

also bitonic.• So, if we can create bitonic sequences of

larger length from these shorter bitonic sequences, we can use the earlier network.

• Question: If X and Y are two sorted sequences, how to create a bitonic sequence with elements of X and Y.

From Bitonic to Arbitrary• Recall that every 1, or 2 element sequence is

also bitonic.• So, if we can create bitonic sequences of

larger length from these shorter bitonic sequences, we can use the earlier network.

• Question: If X and Y are two sorted sequences, how to create a bitonic sequence with elements of X and Y.

• Answer: Concatenate X with Reverse(Y).

The Overall Sorting Network• Built in a bottom-up fashion as follows.• Consider sorting n/2 bitonic sequences of length 2

each. – Each uses the network designed earlier.– Each produces a bitonic sequence of length 4.

• Now, sort pairs of bitonic sequences each of length 4.

• Continue building up to two bitonic sequence of length n/2 each.

• Combine the two sequences to a single bitonic sequence of n elements.

• Finally, use the bitonic sort network.

Another Example – The Closest Pair

• Consider a set of n points in k-dimensions. • Let the distance between any two points be

their Euclidean distance.• The closest pair problem is to find a pair of

points whose pairwise distance is the smallest.

The Closest Pair Problem• For any k, one can always compute the

n(n-1)/2 pairwise distances, and take the smallest.

• This takes time O(kn2).– The k refers to the time taken to compute the

distance in a k-dimensional space.

• We will show today using divide and conquer that better solutions can be designed.

In One Dimension

• If k = 1, all points are on a line.• One solution is to sort the points, and compute

adjacent distances.• This takes O(nlog n) time.• Turns out that this is also the best one can achieve.• Unfortunately, the solution does not extend to

more than one dimension.

In One Dimension• Let us design a generic algorithm using the

divide and conquer strategy that also extends to any dimension (with minimal changes).

• The divide step intuitively seems to be to find the closest pair in the first n/2 points, and the next n/2 points.

• Since the subproblems can be solved recursively, we just have to focus on the combine step.

In One Dimension

In One Dimension• Let dl be the shortest distance in the first

subproblem.• Let dr be the shortest distance in the second

subproblem.

• The closest pair is either the pair at distance dl from the first subproblem, or at distance dr from the second subproblem,

In One Dimension• Let dl be the shortest distance in the first

subproblem.• Let dr be the shortest distance in the second

subproblem.

• The closest pair is either the pair at distance d l from the first subproblem, or at distance dr from the second subproblem,

• Or a pair that crosses the subproblems at a distance less than min{dl, dr}.

• How to quickly find if such a pair exists?

In One Dimension• In the one dimensional case, this is rather easy.• Find the largest point from the first subproblem

and the smallest point from the second subproblem. Call these as x and y.

• If y – x < min{dl, dr}, then x and y are the closest pair.

• Otherwise, the closest pair is the one with mutual distance min{dl, dr}.

In One Dimension• The algorithm is as follows:

Algorithm ClosestPair(A)Begin 1. Find the median point m. 2. Find the closest pair in the points that are less than m, recursively. 3. Find the closest pair in the points that are more than m, recursively. 4. Compute dl, dr from Solutions to problems in Steps 2 and 3 respectively. 5. Find the pair, and the distance d, with one point less than m and the other more than m. 6. Return the closest pair from Steps 2, 3, and 5.End.

In One Dimension• To analyze, notice that each subproblem takes

time T(n/2).

• The combine step takes O(n) time• Why?

• The recurrence relation is T(n) = 2T(n/2) + O(n).

• The solution is Q(nlog n), according to Masters theorem.

Extend to Two Dimensions• Problem 1: How to divide so that each

subproblem has n/2 points.• Problem 2: How to combine the solutions to

the subproblems.

Two Dimensions

• Can pick the median m of the x-coordinates and draw a vertical line y = m.

• Points to the left and the right of m define the subproblems.

Two Dimensions

• Can now find the closest points in each subproblem and denote the distances by dl and dr as earlier.

Two Dimensions• How many pairs do we have to consider

during the combine step.

Two Dimensions

• How many pairs do we have to consider during the combine step.

• Ans: n2/4

Two Dimensions• So, the recurrence now becomes T(n) =

2T(n/2) + O(n2).• With T(n) = Q(n2).• No savings accrued with the divide and

conquer approach.• Unless there is a clever combine step.

Two Dimensions• Notice that while there may be as many as

n2/4 pairs that are yet uncompared, there is some structure in them.

• Also, dl and dr throw additional light.• Need to consider pairs that are at a distance

less than d* = min{dl, dr}.• Intuition: As we move along increasing y

coordinates of points on one side, the distance from a fixed point P increases.

Two Dimensions• Fix a point P on one side of the line y = m.• How many points can pair up with P to have a

distance (with P) at most d*, and their pairwise distance at least d*.

• Why is the second condition true?

Two Dimensions

• Can approximate (Upper bound) the answer as follows.

• Consider a rectangle of height 2d* and width d* with at most six points on the boundary.– Actually six is quite an over count.

Two Dimensions• Call these as the friends of P.• For each P to the left of the line y = m, there

are six friends.• Can find these six distances in O(n) time.• With some care, we can actually find these

friends for all P in O(n) time.• So, the recurrence now is T(n) = 2T(n/2) +

O(n), with a solutio of Q(nlog n).

Two Dimensions• The clever step explained:• Project all points onto the line y = m.• Sort the points to the left of line and the right

of the line by their y-coordinate.• For each P on the left, find all the points

whose projection to y = m is at a distance at most d* from P.

• To get these sorted lists, can pre-sort once.

Higher Dimensions• The same solution extends to higher

dimensions.• Notice that the combine step is operating in

one dimension smaller that the input dimension.

• However, the runtime will be O(nlogd-1 n).• Can be improved to O(nlog n) with more

ideas.

Another Example – Skyline Points

• A point P dominates a point Q if the x- and the y-coordinates of P are larger than that of Q.

• Points that are NOT dominated by any other points are said to be maximal points, or skyline points

Skyline Points • Given a set of n points in a two dimensional

space, find all the skyline (maximal) points.• Use divide and conquer.

Skyline Points• Divide the points into two equal sized sets.

Skyline Points• Solve each part recursively.

Skyline Points• Combine the solutions (skylines)

Skyline Points• Slight asymptotic improvements possible.• Solve the right subproblem first, • Filter the points in the left subproblem that

are dominated by skyline points in the right subproblem.

• Solve only the remaining part of the left subproblem.