ch. 8 & 9 – linear sorting and order statistics

Ch. 8 & 9 – Linear Sortingand

Order StatisticsWhat do you trade for speed?

Ch.8 – Linear Sorting

Sorting by Comparisons

Up to this point, all the sorting algorithms we examined depended on comparing elements of a given set, in order to sort the set. All the algorithms we came up with, were either O(n lg n) or (n lg n) or O(n2).

One can ask: can we sort a set S, consisting of elements from a totally ordered universe, in time O(|S|)?

The answer, as we might expect, is “yes, but…”

First of all, the negative result: sorting by comparisons is (worst case) (n lg n).

04/21/23 291.404


..

04/21/23 391.404


The main ideas are: 1.Every time you try to determine the relative positions of two element you must make a comparison (decision).2.The input (n elements) can come in any order.3.There are n! ways in which n different elements can be arranged.4.A “sort” is equivalent to finding (by a sequence of comparisons between two elements) the permutation of the input that leaves the input set ordered.5.Each such permutation corresponds to a leaf of the “binary decision tree” generated by the comparisons.6.The binary decision tree has n! leaves.

04/21/23 491.404


Theorem 8.1: Any comparison sort algorithm requires (n lg n) comparisons in the worst case.Proof: by the previous discussion, the binary decision tree has at least n! leaves, and height h. Since such a binary tree cannot have more than 2h leaves, we have n! ≤ 2h. Taking the logarithm (base 2) of both sides:

h ≥ lg (n!) = (n lg n),This means that there is at least ONE path of length h connecting the root to a leaf.

Corollary 8.2: HEAPSORT and MERGESORT are asymptotically optimal comparison sorts.

04/21/23 591.404


Sorting NOT by comparisons

How do we do it? • We must make some further assumptions. • For example, we need to assume more than “the set to

be sorted is a set of integers”. • More specifically, we assume the integers fall in some

range, say [1..k], where k = O(n).• This is the idea behind Counting Sort. How do we use

it?

04/21/23 691.404


..

04/21/23 791.404


..

04/21/23 891.404


Counting Sort: Time Complexity

How much time?• The for loop of l. 2-3 takes time (k).• The for loop of l. 4-5 takes time (n).• The for loop of l. 7-8 takes time (k).• The for loop of l. 10-12 takes time (n).• The overall time is (k + n).• The assumption on k gives that the overall time is (n).

04/21/23 991.404


Radix Sort – or Hollerith’s Sort

http://www.columbia.edu/acis/history/hollerith.html

What else can we use?

Assume all your integers (we are sorting sets of integers) have d or fewer digits. Pad the ones with fewer than d digits with leading 0s, for uniformity.

Assume the digits are on cards with 80 or so vertical columns, each column with room for 10 (or more) distinct holes (one for each of 0..9).

Use columns 1..d to store each integer.

Take a deck of such cards (with integers) and sort it.

04/21/23 1091.404

http://www.columbia.edu/acis/history/hollerith.html


Radix Sort

04/21/23 1191.404


Radix Sort

Lemma 8.3. Given n d-digit numbers in which each digit can take up to k possible values, RADIXSORT correctly sorts these numbers in (d(n + k)) time if the stable sort it uses takes (n + k) time.

Proof: the correctness follows by induction on the column being sorted. Sort the first column (using a stable sort). Assume the set is sorted on the first i columns (starting from the back); prove it remains sorted when we use a stable sort on the i+1st column (see Ex. 8.3-3).

COUNTINGSORT on each digit will give the result (details?).

04/21/23 1291.404


Bucket Sort

Assume all the values have equal (independent) probability of appearing as elements of [0, 1).

• Divide the interval [0, 1) into n equal sized subintervals (buckets) and distribute the n input numbers into the buckets.

• Sort the numbers in each bucket (your choice of sort).• Go through the buckets in order, listing the contents.

04/21/23 1391.404


Bucket Sort

04/21/23 1491.404


Bucket Sort

We observe that both of the for loops in l. 3-4 and 5-6 take time O(n) to execute. We need to analyze the cost of the n calls to INSERTIONSORT on l. 8.

04/21/23 1591.404

Ch.8 – Linear SortingBucket Sort

Let ni be the random variable denoting the number of elements in bucket B[i]. Since INSERTIONSORT runs in quadratic time, the running time for BUCKETSORT is

The expected time is given by

We will show that E[ni2] = 2 – 1/n, for i = 0, 1, ..., n - 1.

It should be clear that the expected number of items in each bucket is the same: by the uniform distribution.

04/21/23 1691.404

€

T n( ) = Θ n( ) + O ni2

( )i=0

n −1

∑ .

€

E T n( )[ ] = E Θ n( ) + O ni2

( )i=0

n −1

∑ ⎡

⎣ ⎢

⎤

⎦ ⎥= Θ n( ) + E O ni

2( )[ ]

i=0

n −1

∑ = Θ n( ) + O E ni2

[ ]( )i=0

n −1

∑ .


Bucket Sort

Define the indicator random variables

Xij = I{A[j] falls in bucket i}, for i = 0, …, n - 1, j = 1, …, n.

Thus

To compute E[ni2], we expand, square and regroup:

04/21/23 1791.404

€

ni = X ij .j =1

n

∑

€

E ni2

[ ] = E X iji=1

n

∑ ⎛

⎝ ⎜

⎞

⎠ ⎟

2 ⎡

⎣ ⎢ ⎢

⎤

⎦ ⎥ ⎥= E X ij X ik

k =1

n

∑j =1

n

∑ ⎡

⎣ ⎢ ⎢

⎤

⎦ ⎥ ⎥= E X ij

2

j =1

n

∑ + X ij X ik1≤k ≤nk ≠ j

∑1≤ j ≤n

∑ ⎡

⎣

⎢ ⎢ ⎢

⎤

⎦

⎥ ⎥ ⎥

= E X ij2

[ ]j =1

n

∑ + E X ij X ik[ ]1≤k ≤nk ≠ j

∑1≤ j ≤n

∑


We now compute the two sums.

The indicator random variable Xij takes the value 1 with probability 1/n and 0 with probability 1 – 1/n. The same is true of Xij

2:

E[Xij2] = 12*(1/n) + 02*(1 – 1/n) = 1/n.

We observe that, with k ≠ j, Xij and Xik are independent. This leads to:

E[XijXik] = E[Xij]*E[Xik] = (1/n)*(1/n) = 1/n2.

04/21/23 1891.404


Substitute in the last equation two slides ago:

It follows that

E[T(n)] = (n) + n*O(2 – 1/n) = (n).

Note: the same result holds as long as the distribution implies that the sum of the bucket sizes is linear in the total number of elements.

04/21/23 1991.404

€

E ni2

[ ] =1

nj =1

n

∑ +1

n21≤k ≤nk ≠ j

∑1≤ j ≤n

∑ = n⋅1

n+ n n −1( )

1

n2 =1+n −1

n= 2 −

1

n

Ch.9 – Order StatisticsQuestion: what is the time complexity for the extraction of the ith element in a totally ordered set of n elements?Answer: we first have to find an algorithm…

We can find a minimum or a maximum in (deterministic) time (n) - how?

To find the ith element, we can partition the set into two parts (as in QUICKSORT), if the pivot is the ith element, we are done; if not, we can decide which side of the pivot will contain the ith element and partition that subset further.

04/21/23 2091.404

Ch.9 – Order StatisticsThe algorithm: we use randomization to offset the probability of receiving the worst possible input sequence.

04/21/23 2191.404

Ch.9 – Order StatisticsNote that the recursive call is only on one side of the partition. That is what will allow us to conclude an expected time complexity that is linear in n. Obviously, the worst case is (n2).• Let T(n) be the random variable that denotes the running time of the algorithm on an input array A[p..r] of n elements. We obtain an upper bound on E[T(n)].•RANDOMIZED-PARTITION is equally likely to return any element as the pivot.•Therefore, for each k [1, …, n], the subarray A[p..q] has k elements (all ≤ the pivot) with probability 1/n.

04/21/23 2291.404

Ch.9 – Order Statistics3. For k [1, …, n], define the indicator random variable

Xk = I{the subarray A[p..q] has exactly k elements}.

4. If the elements of A are distinct, E[Xk] = 1/n.

5. When we call RANDOMIZE-SELECT and choose A[q] as the pivot element, we do not know a priori, if we will terminate immediately, recurse on the left subarray A[p, …, q-1] or recurse on the right subarray A[q+1, …, r].

6. The decision depends on where the ith smallest element falls w.r.t. A[q]. Since we are looking for an upper bound, we will assume the desired element is always in the larger of the two partition subarrays – and we’ll assume T(n) to be monotone increasing.

04/21/23 2391.404

Ch.9 – Order Statistics7. For a given call of Randomized-Select, the indicator

random variable Xk has the value 1 for exactly one value of k, and has the value 0 for all oher values of k.

8. When Xk = 1, the two subarrays available for the recursion have sizes k - 1 and n - k.

9. The recurrence is

7. Taking Expected values (Xk & T(max(k-1, n-k)) indep.):

04/21/23 2491.404

€

T n( ) ≤ Xk ⋅ T max k −1,n − k( )( ) + O n( )( )k =1

n

∑ = Xk ⋅T max k −1,n − k( )( ) + O n( ).k =1

n

∑

€

E T n( )( ) ≤ E Xk ⋅ T max k −1,n − k( )( ) + O n( )( )k =1

n

∑ ⎡

⎣ ⎢

⎤

⎦ ⎥= E Xk ⋅T max k −1,n − k( )( )[ ] + O n( )

k =1

n

∑

= E Xk[ ]⋅ E T max k −1,n − k( )( )[ ] + O n( )k =1

n

∑ =1

n⋅ E T max k −1,n − k( )( )[ ] + O n( )

k =1

n

∑

Ch.9 – Order Statistics

04/21/23 2591.404

11. We now need to simplify the last expression: observe that

12. If n is even, each term from T(ceil(n/2)) up to T(n-1) appears exactly twice in the summation;

13. If n is odd, each term from T(ceil(n/2)) up to T(n-1) appears exactly twice in the summation along with one appearance of T(floor(n/2)). This leads to the estimate:

The rest of the proof uses this estimate and induction on the substitution E[T(n)] ≤ c n, for some c > 0 and large enough n.

€

max k −1,n − k( ) =k −1 if k > n

2⎡ ⎤

n − k if k ≤ n2⎡ ⎤

⎧ ⎨ ⎪

⎩ ⎪

€

E T n( )[ ] ≤2

nE T k( )[ ]

k = n 2⎣ ⎦

n −1

∑ + O n( ).

Ch.9 – Order Statistics

04/21/23 2691.404

Note: with a more complex algorithm, one can compute order statistics in deterministic time (n). See the text.

ch. 8 & 9 – linear sorting and order statistics

Documents