06 analysis of algorithms: sorting in linear time

Analysis of AlgorithmsSorting in linear time

Andres Mendez-Vazquez

September 23, 2015

1 / 34

Outline1 Introduction2 Counting Sort

Counting Sort: DefinitionsComplexity

3 Least Significant Digit Radix SortRadix Sort: DefinitionsRepresentationImplementation Using QueuesComplexity

4 Bucket SortIntroductionAlgorithmExampleComplexity Analysis

5 Exercises

2 / 34

About SortingUntil now we had the following constraintGiven two numbers x and y, we compare to know if

x < yx > yx = y

The Trichotomy Law (1)

What if we do something different?How to avoid to compare?

Maybe, we can do the following1 You can try to know the position of the numbers in someway !!!2 You can try to use the relative position of the building blocks of the

numbers using certain numeric system !!!3 You can use the idea of histograms !!!

3 / 34

x < yx > yx = y

3 / 34

x < yx > yx = y

3 / 34

x < yx > yx = y

3 / 34

x < yx > yx = y

3 / 34

x < yx > yx = y

3 / 34

5 Exercises

4 / 34

Counting Sort: Definitions

Counting sort featuresCounting sort is good for sorting integers in a narrow range.It assumes the input numbers (keys) are in the range 0, ..., k.It uses an auxiliary array C [0, ..., k] to hold the number of items lessthan i for 0 ≤ i ≤ k.Complexity O(n + k), if k = O(n), then the running time isΘ(n).Counting sort is stable; it keeps records in their original order.

5 / 34

Counting Sort: DefinitionsCounting sort algorithm - Assume indexing starting at 1Counting-Sort(A, B, k)

1 let C [0..k] be a new array2 for i = 0 to k3 C [i] = 04 for j = 1 to A.length5 C [A [j]] = C [A [j]] + 16 //C [i] contains the number of elements equal to i7 for i = 1 to k8 C [i] = C [i] + C [i − 1]9 // C [i] contains the number of elements≤ i10 for j = A.length to 111 B [C [A [j]]] = A [j]12 C [A [j]] = C [A [j]]− 1

6 / 34

5 Exercises

7 / 34

Final Complexity

We have thatComplexity O(n + k), if k = O(n), then the running time isΘ(n).

8 / 34

Remarks

You can use it forIt is efficient if the range of input data is not significantly greater than thenumber of objects to be sorted.

It is STABLEA sorting algorithm is stable if whenever there are two records R and Swith the same key and with R appearing before S in the original list, Rwill appear before S in the sorted list.

This whyIt is often used as a sub-routine to another sorting algorithm like radix sort.

9 / 34

Remarks

9 / 34

Remarks

9 / 34

5 Exercises

10 / 34

Radix Sort: DefinitionsRadix sort interesting facts!

Radix sort is how IBM made its money, using punch card readers forcensus tabulation in early 1900’s.A radix sorting algorithm was originally used to sort punched cards inseveral passes.

How?It sorts each digit (or field/column) separately.Example:

3 4 5 11 2 2 47 8 9 11 2 2 5

1 2 2 41 2 2 53 4 5 17 8 9 1

It starts with the least-significant digitRadix sort must use a stable sort.

11 / 34

3 4 5 11 2 2 47 8 9 11 2 2 5

1 2 2 41 2 2 53 4 5 17 8 9 1

11 / 34

3 4 5 11 2 2 47 8 9 11 2 2 5

1 2 2 41 2 2 53 4 5 17 8 9 1

11 / 34

3 4 5 11 2 2 47 8 9 11 2 2 5

1 2 2 41 2 2 53 4 5 17 8 9 1

11 / 34

3 4 5 11 2 2 47 8 9 11 2 2 5

1 2 2 41 2 2 53 4 5 17 8 9 1

11 / 34

5 Exercises

12 / 34

It is based in the following idea

FirstEvery number can be represented in each base. For example:

1024 = 1× 103 + 0× 102 + 2× 101 + 4× 100 (2)

Thus in general, given a radix b

x = xdbd + xd−1bd−1 + ... + x0b0 (3)

It can be proved inductivelyWe can sort by using Least-Significative to Most-significative Order andwe keep an stable sort using this order

13 / 34

1024 = 1× 103 + 0× 102 + 2× 101 + 4× 100 (2)

x = xdbd + xd−1bd−1 + ... + x0b0 (3)

13 / 34

1024 = 1× 103 + 0× 102 + 2× 101 + 4× 100 (2)

x = xdbd + xd−1bd−1 + ... + x0b0 (3)

13 / 34

However

Remark 1A most significant digit (MSD) radix sort can be used to sort keys inlexicographic order.

Remark 2Unlike a least significant digit (LSD) radix sort, a most significant digitradix sort does not necessarily preserve the original order of duplicate keys.

14 / 34

However

Remark 1A most significant digit (MSD) radix sort can be used to sort keys inlexicographic order.

Remark 2Unlike a least significant digit (LSD) radix sort, a most significant digitradix sort does not necessarily preserve the original order of duplicate keys.

14 / 34

Example

Example3 2 94 5 76 5 78 3 94 3 67 2 03 5 5

7 2 03 5 54 3 64 5 76 5 73 2 98 3 9

7 2 03 2 94 3 68 3 93 5 54 5 76 5 7

3 2 93 5 54 3 64 5 76 5 77 2 08 3 9

15 / 34

Example

Example3 2 94 5 76 5 78 3 94 3 67 2 03 5 5

7 2 03 5 54 3 64 5 76 5 73 2 98 3 9

7 2 03 2 94 3 68 3 93 5 54 5 76 5 7

3 2 93 5 54 3 64 5 76 5 77 2 08 3 9

15 / 34

Example

Example3 2 94 5 76 5 78 3 94 3 67 2 03 5 5

7 2 03 5 54 3 64 5 76 5 73 2 98 3 9

7 2 03 2 94 3 68 3 93 5 54 5 76 5 7

3 2 93 5 54 3 64 5 76 5 77 2 08 3 9

15 / 34

Example

Example3 2 94 5 76 5 78 3 94 3 67 2 03 5 5

7 2 03 5 54 3 64 5 76 5 73 2 98 3 9

7 2 03 2 94 3 68 3 93 5 54 5 76 5 7

3 2 93 5 54 3 64 5 76 5 77 2 08 3 9

15 / 34

Radix Sort: Algorithm Using the Least Significative Digit

AlgorithmRadix-Sort(A, d)

1 for i = 1 to d2 use a stable sort to sort array A on digit i

16 / 34

5 Exercises

17 / 34

Iterative version using queuesHow? First, assume Radix = 10The integers are enqueued into an array of ten separate queues based ontheir digits from right to left.

Example: 170, 045, 075, 090, 002, 024, 802, 0660 : 170, 1901 : none2 : 002, 8023 : none4 : 0245 : 045, 0756 : 0667 : none8 : none9 : none

18 / 34

Iterative version using queuesHow? First, assume Radix = 10The integers are enqueued into an array of ten separate queues based ontheir digits from right to left.

Example: 170, 045, 075, 090, 002, 024, 802, 0660 : 170, 1901 : none2 : 002, 8023 : none4 : 0245 : 045, 0756 : 0667 : none8 : none9 : none

18 / 34

Iterative version using queues

Then, the queues are dequeued back into an array of integers, inincreasing order

170, 090, 002, 802, 024, 045, 075, 066

ThenYou repeat again!!!

19 / 34

Iterative version using queues

Then, the queues are dequeued back into an array of integers, inincreasing order

170, 090, 002, 802, 024, 045, 075, 066

ThenYou repeat again!!!

19 / 34

5 Exercises

20 / 34

Radix Sort: Proving the Complexity

Lemma 1Given n d-digit numbers in which each digit can take on up to k possiblevalues, RADIX-SORT correctly sorts these numbers in Θ(d(n + k)) time.

ProofQuite simple!!!

21 / 34

Radix Sort: Proving the Complexity

Lemma 1Given n d-digit numbers in which each digit can take on up to k possiblevalues, RADIX-SORT correctly sorts these numbers in Θ(d(n + k)) time.

ProofQuite simple!!!

21 / 34

Thanks Carlos Alcala Oracle Class 2014 for the ExampleImagine the followingThat you have a sequence of n IP Addresses. For example:

n IP addresses

192.168.45.120192.128.15.120100.192.168.45

...92.16.4.120

Thus, Why not use the chunks in the IP to sort the IP addresses(Think Columns)

n IP addresses

192.168.45.120192.128.15.120100.192.168.45

...92.16.4.120

22 / 34

Thanks Carlos Alcala Oracle Class 2014 for the ExampleImagine the followingThat you have a sequence of n IP Addresses. For example:

n IP addresses

192.168.45.120192.128.15.120100.192.168.45

...92.16.4.120

Thus, Why not use the chunks in the IP to sort the IP addresses(Think Columns)

n IP addresses

192.168.45.120192.128.15.120100.192.168.45

...92.16.4.120

22 / 34

Yes!!!

Yes!!!After all each chunk is a number between 0 and 255 i.e between 0 to28 − 1 ⇒ size chunks is r = 8We have then for each IP address a size of b = 32 bits

Lemma 2Given n b-bit numbers and any positive ingeter r ≤ b, RADIX-SORTcorrectly sort these numbers in Θ( b

r (n + 2r))time.

ProofAt the Board...

23 / 34

Yes!!!

r (n + 2r))time.

23 / 34

Yes!!!

r (n + 2r))time.

23 / 34

Yes!!!

r (n + 2r))time.

23 / 34

Final Remarks

Final RemarksLSD radix sorts have resurfaced as an alternative to high performancecomparison-based sorting algorithms (like heapsort and mergesort)that require O(n log n) comparisons. YES BIG DATA!!!For more, look at V. J. Duvanenko, "In-Place Hybrid Binary-RadixSort", Dr. Dobb’s Journal, 1 October 2009

24 / 34

Final Remarks

Final RemarksLSD radix sorts have resurfaced as an alternative to high performancecomparison-based sorting algorithms (like heapsort and mergesort)that require O(n log n) comparisons. YES BIG DATA!!!For more, look at V. J. Duvanenko, "In-Place Hybrid Binary-RadixSort", Dr. Dobb’s Journal, 1 October 2009

24 / 34

5 Exercises

25 / 34

Bucket Sort: Definitions

AssumptionThe keys are in the range [0, 1).

I Actually you can have a range [0, N ) and divide the Keys by N

You have n of them

Basic idea when you have1 Create n linked lists (buckets) to divide interval [0, 1) into

subintervals of size 1/n.2 Add each input element to the appropriate bucket.3 Concatenate the buckets.

ComplexityExpected total time is O(n), where n is the size of the original sequence.

26 / 34

You have n of them

26 / 34

You have n of them

26 / 34

You have n of them

26 / 34

You have n of them

26 / 34

You have n of them

26 / 34

You have n of them

26 / 34

5 Exercises

27 / 34

Bucket Sort Algorithm

Algorithm assumingBuket-Sort(A)

1 let B [0..n − 1] be a new array2 n = A.length3 for i = 0 to n − 14 make B [i] an empty list5 for i = 0 to n6 insert A [i] into list B [bnA [i]c]7 for i = 0 to n − 18 sort list B [i] with insertion sort9 concatenate the list B [0], B [1], ... , B [n − 1] together in order

28 / 34

5 Exercises

29 / 34

Bucket SortExample

30 / 34

5 Exercises

31 / 34

We have the following

We need to analyze the algoritmBut we have an insertion sort at each bucket!!!

ThereforeLet ni the random variable on the size of the bucket.

We get the following complexity function

T (n) = Θ (n) +n−1∑i=0

Look at the Board!!!

32 / 34

T (n) = Θ (n) +n−1∑i=0

32 / 34

T (n) = Θ (n) +n−1∑i=0

32 / 34

Final Complexity is

After using the expected value

E (T (n)) = Θ (n)

33 / 34

Exercises

From Cormen’s book solve the following8.1-18.1-38.2-28.2-38.3-28.3-48.4-2

34 / 34

06 analysis of algorithms: sorting in linear time

Engineering

c i c

k8 c i

k3 c i

sortcounting sort

length5 c

ifx yx yx

auxiliary array c

numbers x