sorting. objectives become familiar with the following sorting methods: insertion sort shell sort...
Post on 13-Dec-2015
268 Views
Preview:
TRANSCRIPT
Sorting
Objectives Become familiar with the following
sorting methods Insertion Sort Shell Sort Selection Sort Bubble Sort Quick Sort Heap Sort Merge Sort hellip more
Introduction One of the most common applications
in computer science is sorting the process through which data are arranged according to their values
If data were not ordered in some way we would spend an incredible amount of time trying to find the correct information
Introduction To appreciate this imagine trying
to find someonersquos number in the telephone book if the names were not sorted in some way
General Sorting Concepts Sorts are generally classified as either
internal or external An internal sort is a sort in which all of
the data is held in primary memory during the sorting process
An external sort uses primary memory for the data currently being sorted and secondary storage for any data that will not fit in primary memory
General Sorting Concepts For example a file of 20000
records may be sorted using an array that holds only 1000 records
Therefore only 1000 records are in primary memory at any given time
The other 19000 records are stored in secondary storage
Sort Order Data may be sorted in either
ascending or descending order The sort order identifies the
sequence of sorted data ascending or descending
If the order of the sort is not specified it is assumed to be ascending
Sort Stability Sort stability is an attribute of a
sort indicating that data elements with equal keys maintain their relative input order in the output
Consider the following example
Sort Stability
Note the unsorted data in (a) If we use a stable sort items with equal keys are
guaranteed to be sorted in the same order as they appear in the unsorted data
Sort Stability
If however we use an unstable sort data with equal keys can be sorted in any order (ie not necessarily in the same order as they appeared in the unsorted data)
Sort Efficiency Sort efficiency is a measure of
the relative efficiency of a sort It is usually an estimate of the
number of comparisons and moves required to order an unordered list
Passes During the sorting process the data is
traversed many times Each traversal of the data is referred to
as a sort pass Depending on the algorithm the sort
pass may traverse the whole list or just a section of the list
The sort pass may also include the placement of one or more elements into the sorted list
Types of Sorts We now discuss several sorting
methods Insertion Sort Shell Sort Selection Sort Bubble Sort Quick Sort Heap Sort Merge Sort
Insertion Sort In each pass of an insertion sort
one or more pieces of data are inserted into their correct location in an ordered list (just as a card player picks up cards and places them in his hand in order)
Insertion Sort In the insertion sort the list is divided into
2 parts Sorted Unsorted
In each pass the first element of the unsorted sublist is transferred to the sorted sublist by inserting it at the appropriate place
If we have a list of n elements it will take at most n-1 passes to sort the data
Insertion Sort
We can visualize this type of sort with the above figure
The first part of the list is the sorted portion which is separated by a ldquoconceptualrdquo wall from the unsorted portion of the list
Insertion SortHere we start with an unsorted list We leave the first data element alone and will start with the 2nd element of the list (the 1st element of the unsorted list)
On our first pass we look at the 2nd element of the list (thefirst element of the unsorted list) and place it in our sorted list in order
On our next pass we look at the 3rd element of the list (the1st element of the unsorted list) and place it in the sorted list in order
We continue in this fashion until the entire list has beensorted
Insertion Sort Example
In our first pass we compare the first 2 values Because theyare out of order we swap them
Now we look at the next 2 values Again they are out oforder so we swap them
Since we have swapped those values we need to comparethe previous 2 values to make sure that they are still in orderSince they are out of order we swap them and then continueon with the next 2 data valuesThese 2 values are out of order so we swap them and look atthe previous 2 values etc
Shell Sort Named after its creator Donald Shell the
shell sort is an improved version of the insertion sort
In the shell sort a list of N elements is divided into K segments where K is known as the increment
What this means is that instead of comparing adjacent values we will compare values that are a distance K apart
We will shrink K as we run through our algorithm
Shell SortPass Notes
1 77 62 14 9 30 21 80 25 70 55 Swap2 21 62 14 9 30 77 80 25 70 55 In order3 21 62 14 9 30 77 80 25 70 55 In order4 21 62 14 9 30 77 80 25 70 55 In order5 21 62 14 9 30 77 80 25 70 55 In order
List (K=5)
Just as in the straight insertion sort we compare 2 values andswap them if they are out of order However in the shell sort we compare values that are a distance K apartOnce we have completed going through the elements in our
list with K=5 we decrease K and continue the process
Shell SortPass Notes
6 21 62 14 9 30 77 80 25 70 55 Swap7 14 62 21 9 30 77 80 25 70 55 Swap8 14 9 21 62 30 77 80 25 70 55 In order9 14 9 21 62 30 77 80 25 70 55 In order10 14 9 21 62 30 77 80 25 70 55 In order11 14 9 21 62 30 77 80 25 70 55 Swap
14 9 21 62 30 25 80 77 70 55 Swap14 9 21 25 30 62 80 77 70 55 In order
12 14 9 21 25 30 62 80 77 70 55 Swap14 9 21 25 30 62 70 77 80 55 In order
13 14 9 21 25 30 62 70 77 80 55 Swap14 9 21 25 30 62 70 55 80 77 Swap14 9 21 25 30 55 70 62 80 77 In order
List (K=2)
Here we have reduced K to 2 Just as in the insertion sort if we swap 2 values we have to go back and compare theprevious 2 values to make sure they are still in order
Shell Sort
All shell sorts will terminate by running an insertion sort(ie K=1) However using the larger values of K first has helped to sort our list so that the straight insertion sort will runfaster
Shell Sort There are many schools of thought
on what the increment should be in the shell sort
Also note that just because an increment is optimal on one list it might not be optimal for another list
Insertion Sort vs Shell Sort Comparing the Big-O notation (for the
average case) we find that Insertion O(n2) Shell O(n125) empirically determined
Although this doesnrsquot seem like much of a gain it makes a big difference as n gets large
Note that in the worst case the Shell sort has an efficiency of O(n2)
However using a special incrementing technique this worst case can be reduced to O(n15)
Insertion Sort vs Shell Sort(Average Case)
Straight Insertion Shell25 625 55100 10000 316500 250000 2364
1000 1000000 56232000 4000000 13374
Number of Loopsn
Selection Sort Imagine some data that you can examine
all at once To sort it you could select the smallest
element and put it in its place select the next smallest and put it in its place etc
For a card player this process is analogous to looking at an entire hand of cards and ordering them by selecting cards one at a time and placing them in their proper order
Selection Sort The selection sort follows this idea Given a list of data to be sorted
we simply select the smallest item and place it in a sorted list
We then repeat these steps until the list is sorted
Selection Sort In the selection sort the list at any moment
is divided into 2 sublists sorted and unsorted separated by a ldquoconceptualrdquo wall
We select the smallest element from the unsorted sublist and exchange it with the element at the beginning of the unsorted data
After each selection and exchange the wall between the 2 sublists moves ndash increasing the number of sorted elements and decreasing the number of unsorted elements
Selection Sort
We start with an unsorted list We search this list for thesmallest element We then exchange the smallest element (8)with the first element in the unsorted list (23)
Again we search the unsorted list for the smallest element We then exchange the smallest element (23) with the first element in the unsorted list (78)
This process continues until the list is fully sorted
Bubble Sort In the bubble sort the list at any
moment is divided into 2 sublists sorted and unsorted
The smallest element is ldquobubbledrdquo from the unsorted sublist to the sorted sublist
Bubble Sort
23 78 45 8 56 3232 56
We start with 32 and compareit with 56 Because 32 is lessthan 56 we swap the twoand step down one element
We then compare 32 and 8Because 32 is not less than 8we do not swap these elements
We step down one element andcompare 45 and 8 They areout of sequence so we swapthem and step down again
8 45
We step down again and compare 8 with 78 These two elements are swapped
8 78
Finally 8 is compared with 23and swapped We thencontinue this process back with56 hellip
8 23
1 Pass of the Bubble Sort
Quick Sort In the bubble sort consecutive items
are compared and possibly exchanged on each pass through the list
This means that many exchanges may be needed to move an element to its correct position
Quick sort is more efficient than bubble sort because a typical exchange involves elements that are far apart so fewer exchanges are required to correctly position an element
Idea of Quick Sort1) Select pick an element
2) Divide rearrange elements so that x goes to its final position E
3) Recurse and Conquer recursively sort
Quick Sort Each iteration of the quick sort
selects an element known as the pivot and divides the list into 3 groups Elements whose keys are less than (or
equal to) the pivotrsquos key The pivot element Elements whose keys are greater than
(or equal to) the pivotrsquos key
Quick Sort The sorting then continues by
quick sorting the left partition followed by quick sorting the right partition
The basic algorithm is as follows
Quick Sort1) Partitioning Step Take an element in the
unsorted array and determine its final location in the sorted array This occurs when all values to the left of the element in the array are less than (or equal to) the element and all values to the right of the element are greater than (or equal to) the element We now have 1 element in its proper location and two unsorted subarrays
2) Recursive Step Perform step 1 on each unsorted subarray
Quick Sort Each time step 1 is performed on a
subarray another element is placed in its final location of the sorted array and two unsorted subarrays are created
When a subarray consists of one element that subarray is sorted
Therefore that element is in its final location
Quick Sort There are several partitioning strategies
used in practice (ie several ldquoversionsrdquo of quick sort) but the one we are about to describe is known to work well
For simplicity we will choose the last element to be the pivot element
We could also chose a different pivot element and swap it with the last element in the array
Quick Sort Below is the array we would like to
sort1 4 8 9 0 11 5 10 7 6
Quick Sort The index left starts at the first element
and right starts at the next-to-last element
We want to move all the elements smaller than the pivot to the left part of the array and all the elements larger than the pivot to the right part
1 4 8 9 0 11 5 10 7 6left righ
t
Quick Sort We move left to the right skipping
over elements that are smaller than the pivot
1 4 8 9 0 11 5 10 7 6left righ
t
Quick Sort We then move right to the left skipping
over elements that are greater than the pivot
When left and right have stopped left is on an element greater than (or equal to) the pivot and right is on an element smaller than (or equal to) the pivot
1 4 8 9 0 11 5 10 7 6left right
Quick Sort If left is to the left of right (or if left
= right) those elements are swapped
1 4 8 9 0 11 5 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
Quick Sort The effect is to push a large
element to the right and a small element to the left
We then repeat the process until left and right cross
Quick Sort
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6left righ
t
Quick Sort
1 4 5 0 9 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6righ
tleft
Quick Sort
At this point left and right have crossed so no swap is performed
The final part of the partitioning is to swap the pivot element with left
1 4 5 0 9 11 8 10 7 6righ
tleft
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort Note that all elements to the left of
the pivot are less than (or equal to) the pivot and all elements to the right of the pivot are greater than (or equal to) the pivot
Hence the pivot element has been placed in its final sorted position
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort We now repeat the process using
the sub-arrays to the left and right of the pivot
1 4 5 0 6 11 8 10 7 9
1 4 5 0 11 8 10 7 9
Analysis of quicksortmdashbest case
Suppose each partition operation divides the array almost exactly in half
Then the depth of the recursion in log2n Because thatrsquos how many times we can
halve n However there are many recursions at
each level How can we figure this out We note that
Each partition is linear over its subarray All the partitions at one level cover the array
Partitioning at various levels
Best case II We cut the array size in half each time So the depth of the recursion in log2n At each level of the recursion all the
partitions at that level do work that is linear in n
O(log2n) O(n) = O(n log2n)
What about the worst case
Worst case In the worst case partitioning always
divides the size n array into these three parts A length one part containing the pivot itself A length zero part and A length n-1 part containing everything else
We donrsquot recur on the zero-length part Recurring on the length n-1 part requires
(in the worst case) recurring to depth n-1
Worst case partitioning
Worst case for quicksort In the worst case recursion may be
O(n) levels deep (for an array of size n)
But the partitioning work done at each level is still O(n)
O(n) O(n) = O(n2) So worst case for Quicksort is O(n2) When does this happen
When the array is sorted to begin with
Typical case for quicksort If the array is sorted to begin with
Quicksort is terrible O(n2) It is possible to construct other bad
cases However Quicksort is usually O(n log2n) The constants are so good that Quicksort
is generally the fastest algorithm known Most real-world sorting is done by
Quicksort
Tweaking Quicksort Almost anything you can try to
ldquoimproverdquo Quicksort will actually slow it down
One good tweak is to switch to a different sorting method when the subarrays get small (say 10 or 12) Quicksort has too much overhead for small
array sizes For large arrays it might be a good
idea to check beforehand if the array is already sorted But there is a better tweak than this
Randomized Quick-Sort Select the pivot as a random element of the sequence The expected running time of randomized quick-sort on a
sequence of size n is O(n log n) The time spent at a level of the quick-sort tree is O(n) We show that the expected height of the quick-sort tree is
O(log n) good vs bad pivots
bull The probability of a good pivot is 12 thus we expect k2 good pivots out of k pivotsbull After a good pivot the size of each child sequence is at most 34 the size of the parent sequencebull After h pivots we expect (34)h2 n elementsbull The expected height h of the quick-sort tree is at most 2 log43n
Median of three Obviously it doesnrsquot make sense to sort the
array in order to find the median to use as a pivot
Instead compare just three elements of our (sub)arraymdashthe first the last and the middle Take the median (middle value) of these three as
pivot Itrsquos possible (but not easy) to construct cases
which will make this technique O(n2) Suppose we rearrange (sort) these three
numbers so that the smallest is in the first position the largest in the last position and the other in the middle This lets us simplify and speed up the partition
loop
Final comments Weissrsquos code shows some additional
optimizations on pp 246-247 Weiss chooses to stop both searches on
equality to pivot This design decision is debatable
Quicksort is the fastest known general sorting algorithm on average
For optimum speed the pivot must be chosen carefully
ldquoMedian of threerdquo is a good technique for choosing the pivot
There will be some cases where Quicksort runs in O(n2) time
Quick Sort A couple of notes about quick sort There are more optimal ways to
choose the pivot value (such as the median-of-three method)
Also when the subarrays get small it becomes more efficient to use the insertion sort as opposed to continued use of quick sort
Bubble Sort vs Quick Sort If we calculate the Big-O notation
we find that (in the average case) Bubble Sort O(n2) Quick Sort O(nlog2n)
(quick-sort runs in time O(n2) in the worst case)
Heap Sort Idea take the items that need to be
sorted and insert them into the heap By calling deleteHeap we remove the
smallest or largest element depending on whether or not we are working with a min- or max-heap respectively
Hence the elements are removed in ascending or descending order
Efficiency O(nlog2n)
Merge Sort Idea
Take the array you would like to sort and divide it in half to create 2 unsorted subarrays
Next sort each of the 2 subarrays Finally merge the 2 sorted subarrays
into 1 sorted array Efficiency O(nlog2n)
Merge Sort
Merge Sort Although the merge step produces
a sorted array we have overlooked a very important step
How did we sort the 2 halves before performing the merge step
We used merge sort
Merge Sort By continually calling the merge
sort algorithm we eventually get a subarray of size 1
Since an array with only 1 element is clearly sorted we can back out and merge 2 arrays of size 1
Merge Sort
Merge Sort The basic merging algorithm
consists of 2 input arrays (arrayA and arrayB) An ouput array (arrayC) 3 position holders (indexA indexB
indexC) which are initially set to the beginning of their respective arrays
Merge Sort The smaller of arrayA[indexA] and
arrayB[indexB] is copied into arrayC[indexC] and the appropriate position holders are advanced
When either input list is exhausted the remainder of the other list is copied into arrayC
Merge Sort
1 13 24 26index
A
indexC
arrayA
2 15 27 38index
B
arrayB
arrayC
We compare arrayA[indexA]with arrayB[indexB] Whichever value is smaller isplaced into arrayC[indexC]
1 lt 2 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1index
C
arrayA
2 15 27 38index
B
arrayB
arrayC
2 lt 13 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
13 lt 15 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
15 lt 24 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13 15indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
24 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
indexA
1 2 13 15 24indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
26 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
1 2 13 15 24 26indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
Since we have exhaustedone of the arrays arrayAwe simply copy the remaining items from theother array arrayB intoarrayC
Merge Sort
1 13 24
26
1 2 13 15 24 26 27 38
arrayA
2 15 27 38arrayB
arrayC
Efficiency Summary
Sort Worst Case Average Case
Insertion O(n2) O(n2)
Shell O(n15) O(n125)
Selection O(n2) O(n2)
Bubble O(n2) O(n2)
Quick O(n2) O(nlog2n)
Heap O(nlog2n) O(nlog2n)
Merge O(nlog2n) O(nlog2n)
More Sorting
radix sort bucket sort in-place sorting how fast can we sort
Radix Sort
Unlike other sorting methods radix sort considers the structure of the keys
Assume keys are represented in a base M number system (M = radix) ie if M = 2 the keys are represented in binary
Sorting is done by comparing bits in the same position
Extension to keys that are alphanumeric strings
Radix Exchange Sort
Examine bits from left to right1 Sort array with respect to leftmost bit
2 Partition array
3Recursionrecursively sort top subarray ignoring leftmost bitrecursively sort bottom subarray ignoring leftmost bit
Time to sort n b-bit numbers O(b n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Objectives Become familiar with the following
sorting methods Insertion Sort Shell Sort Selection Sort Bubble Sort Quick Sort Heap Sort Merge Sort hellip more
Introduction One of the most common applications
in computer science is sorting the process through which data are arranged according to their values
If data were not ordered in some way we would spend an incredible amount of time trying to find the correct information
Introduction To appreciate this imagine trying
to find someonersquos number in the telephone book if the names were not sorted in some way
General Sorting Concepts Sorts are generally classified as either
internal or external An internal sort is a sort in which all of
the data is held in primary memory during the sorting process
An external sort uses primary memory for the data currently being sorted and secondary storage for any data that will not fit in primary memory
General Sorting Concepts For example a file of 20000
records may be sorted using an array that holds only 1000 records
Therefore only 1000 records are in primary memory at any given time
The other 19000 records are stored in secondary storage
Sort Order Data may be sorted in either
ascending or descending order The sort order identifies the
sequence of sorted data ascending or descending
If the order of the sort is not specified it is assumed to be ascending
Sort Stability Sort stability is an attribute of a
sort indicating that data elements with equal keys maintain their relative input order in the output
Consider the following example
Sort Stability
Note the unsorted data in (a) If we use a stable sort items with equal keys are
guaranteed to be sorted in the same order as they appear in the unsorted data
Sort Stability
If however we use an unstable sort data with equal keys can be sorted in any order (ie not necessarily in the same order as they appeared in the unsorted data)
Sort Efficiency Sort efficiency is a measure of
the relative efficiency of a sort It is usually an estimate of the
number of comparisons and moves required to order an unordered list
Passes During the sorting process the data is
traversed many times Each traversal of the data is referred to
as a sort pass Depending on the algorithm the sort
pass may traverse the whole list or just a section of the list
The sort pass may also include the placement of one or more elements into the sorted list
Types of Sorts We now discuss several sorting
methods Insertion Sort Shell Sort Selection Sort Bubble Sort Quick Sort Heap Sort Merge Sort
Insertion Sort In each pass of an insertion sort
one or more pieces of data are inserted into their correct location in an ordered list (just as a card player picks up cards and places them in his hand in order)
Insertion Sort In the insertion sort the list is divided into
2 parts Sorted Unsorted
In each pass the first element of the unsorted sublist is transferred to the sorted sublist by inserting it at the appropriate place
If we have a list of n elements it will take at most n-1 passes to sort the data
Insertion Sort
We can visualize this type of sort with the above figure
The first part of the list is the sorted portion which is separated by a ldquoconceptualrdquo wall from the unsorted portion of the list
Insertion SortHere we start with an unsorted list We leave the first data element alone and will start with the 2nd element of the list (the 1st element of the unsorted list)
On our first pass we look at the 2nd element of the list (thefirst element of the unsorted list) and place it in our sorted list in order
On our next pass we look at the 3rd element of the list (the1st element of the unsorted list) and place it in the sorted list in order
We continue in this fashion until the entire list has beensorted
Insertion Sort Example
In our first pass we compare the first 2 values Because theyare out of order we swap them
Now we look at the next 2 values Again they are out oforder so we swap them
Since we have swapped those values we need to comparethe previous 2 values to make sure that they are still in orderSince they are out of order we swap them and then continueon with the next 2 data valuesThese 2 values are out of order so we swap them and look atthe previous 2 values etc
Shell Sort Named after its creator Donald Shell the
shell sort is an improved version of the insertion sort
In the shell sort a list of N elements is divided into K segments where K is known as the increment
What this means is that instead of comparing adjacent values we will compare values that are a distance K apart
We will shrink K as we run through our algorithm
Shell SortPass Notes
1 77 62 14 9 30 21 80 25 70 55 Swap2 21 62 14 9 30 77 80 25 70 55 In order3 21 62 14 9 30 77 80 25 70 55 In order4 21 62 14 9 30 77 80 25 70 55 In order5 21 62 14 9 30 77 80 25 70 55 In order
List (K=5)
Just as in the straight insertion sort we compare 2 values andswap them if they are out of order However in the shell sort we compare values that are a distance K apartOnce we have completed going through the elements in our
list with K=5 we decrease K and continue the process
Shell SortPass Notes
6 21 62 14 9 30 77 80 25 70 55 Swap7 14 62 21 9 30 77 80 25 70 55 Swap8 14 9 21 62 30 77 80 25 70 55 In order9 14 9 21 62 30 77 80 25 70 55 In order10 14 9 21 62 30 77 80 25 70 55 In order11 14 9 21 62 30 77 80 25 70 55 Swap
14 9 21 62 30 25 80 77 70 55 Swap14 9 21 25 30 62 80 77 70 55 In order
12 14 9 21 25 30 62 80 77 70 55 Swap14 9 21 25 30 62 70 77 80 55 In order
13 14 9 21 25 30 62 70 77 80 55 Swap14 9 21 25 30 62 70 55 80 77 Swap14 9 21 25 30 55 70 62 80 77 In order
List (K=2)
Here we have reduced K to 2 Just as in the insertion sort if we swap 2 values we have to go back and compare theprevious 2 values to make sure they are still in order
Shell Sort
All shell sorts will terminate by running an insertion sort(ie K=1) However using the larger values of K first has helped to sort our list so that the straight insertion sort will runfaster
Shell Sort There are many schools of thought
on what the increment should be in the shell sort
Also note that just because an increment is optimal on one list it might not be optimal for another list
Insertion Sort vs Shell Sort Comparing the Big-O notation (for the
average case) we find that Insertion O(n2) Shell O(n125) empirically determined
Although this doesnrsquot seem like much of a gain it makes a big difference as n gets large
Note that in the worst case the Shell sort has an efficiency of O(n2)
However using a special incrementing technique this worst case can be reduced to O(n15)
Insertion Sort vs Shell Sort(Average Case)
Straight Insertion Shell25 625 55100 10000 316500 250000 2364
1000 1000000 56232000 4000000 13374
Number of Loopsn
Selection Sort Imagine some data that you can examine
all at once To sort it you could select the smallest
element and put it in its place select the next smallest and put it in its place etc
For a card player this process is analogous to looking at an entire hand of cards and ordering them by selecting cards one at a time and placing them in their proper order
Selection Sort The selection sort follows this idea Given a list of data to be sorted
we simply select the smallest item and place it in a sorted list
We then repeat these steps until the list is sorted
Selection Sort In the selection sort the list at any moment
is divided into 2 sublists sorted and unsorted separated by a ldquoconceptualrdquo wall
We select the smallest element from the unsorted sublist and exchange it with the element at the beginning of the unsorted data
After each selection and exchange the wall between the 2 sublists moves ndash increasing the number of sorted elements and decreasing the number of unsorted elements
Selection Sort
We start with an unsorted list We search this list for thesmallest element We then exchange the smallest element (8)with the first element in the unsorted list (23)
Again we search the unsorted list for the smallest element We then exchange the smallest element (23) with the first element in the unsorted list (78)
This process continues until the list is fully sorted
Bubble Sort In the bubble sort the list at any
moment is divided into 2 sublists sorted and unsorted
The smallest element is ldquobubbledrdquo from the unsorted sublist to the sorted sublist
Bubble Sort
23 78 45 8 56 3232 56
We start with 32 and compareit with 56 Because 32 is lessthan 56 we swap the twoand step down one element
We then compare 32 and 8Because 32 is not less than 8we do not swap these elements
We step down one element andcompare 45 and 8 They areout of sequence so we swapthem and step down again
8 45
We step down again and compare 8 with 78 These two elements are swapped
8 78
Finally 8 is compared with 23and swapped We thencontinue this process back with56 hellip
8 23
1 Pass of the Bubble Sort
Quick Sort In the bubble sort consecutive items
are compared and possibly exchanged on each pass through the list
This means that many exchanges may be needed to move an element to its correct position
Quick sort is more efficient than bubble sort because a typical exchange involves elements that are far apart so fewer exchanges are required to correctly position an element
Idea of Quick Sort1) Select pick an element
2) Divide rearrange elements so that x goes to its final position E
3) Recurse and Conquer recursively sort
Quick Sort Each iteration of the quick sort
selects an element known as the pivot and divides the list into 3 groups Elements whose keys are less than (or
equal to) the pivotrsquos key The pivot element Elements whose keys are greater than
(or equal to) the pivotrsquos key
Quick Sort The sorting then continues by
quick sorting the left partition followed by quick sorting the right partition
The basic algorithm is as follows
Quick Sort1) Partitioning Step Take an element in the
unsorted array and determine its final location in the sorted array This occurs when all values to the left of the element in the array are less than (or equal to) the element and all values to the right of the element are greater than (or equal to) the element We now have 1 element in its proper location and two unsorted subarrays
2) Recursive Step Perform step 1 on each unsorted subarray
Quick Sort Each time step 1 is performed on a
subarray another element is placed in its final location of the sorted array and two unsorted subarrays are created
When a subarray consists of one element that subarray is sorted
Therefore that element is in its final location
Quick Sort There are several partitioning strategies
used in practice (ie several ldquoversionsrdquo of quick sort) but the one we are about to describe is known to work well
For simplicity we will choose the last element to be the pivot element
We could also chose a different pivot element and swap it with the last element in the array
Quick Sort Below is the array we would like to
sort1 4 8 9 0 11 5 10 7 6
Quick Sort The index left starts at the first element
and right starts at the next-to-last element
We want to move all the elements smaller than the pivot to the left part of the array and all the elements larger than the pivot to the right part
1 4 8 9 0 11 5 10 7 6left righ
t
Quick Sort We move left to the right skipping
over elements that are smaller than the pivot
1 4 8 9 0 11 5 10 7 6left righ
t
Quick Sort We then move right to the left skipping
over elements that are greater than the pivot
When left and right have stopped left is on an element greater than (or equal to) the pivot and right is on an element smaller than (or equal to) the pivot
1 4 8 9 0 11 5 10 7 6left right
Quick Sort If left is to the left of right (or if left
= right) those elements are swapped
1 4 8 9 0 11 5 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
Quick Sort The effect is to push a large
element to the right and a small element to the left
We then repeat the process until left and right cross
Quick Sort
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6left righ
t
Quick Sort
1 4 5 0 9 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6righ
tleft
Quick Sort
At this point left and right have crossed so no swap is performed
The final part of the partitioning is to swap the pivot element with left
1 4 5 0 9 11 8 10 7 6righ
tleft
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort Note that all elements to the left of
the pivot are less than (or equal to) the pivot and all elements to the right of the pivot are greater than (or equal to) the pivot
Hence the pivot element has been placed in its final sorted position
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort We now repeat the process using
the sub-arrays to the left and right of the pivot
1 4 5 0 6 11 8 10 7 9
1 4 5 0 11 8 10 7 9
Analysis of quicksortmdashbest case
Suppose each partition operation divides the array almost exactly in half
Then the depth of the recursion in log2n Because thatrsquos how many times we can
halve n However there are many recursions at
each level How can we figure this out We note that
Each partition is linear over its subarray All the partitions at one level cover the array
Partitioning at various levels
Best case II We cut the array size in half each time So the depth of the recursion in log2n At each level of the recursion all the
partitions at that level do work that is linear in n
O(log2n) O(n) = O(n log2n)
What about the worst case
Worst case In the worst case partitioning always
divides the size n array into these three parts A length one part containing the pivot itself A length zero part and A length n-1 part containing everything else
We donrsquot recur on the zero-length part Recurring on the length n-1 part requires
(in the worst case) recurring to depth n-1
Worst case partitioning
Worst case for quicksort In the worst case recursion may be
O(n) levels deep (for an array of size n)
But the partitioning work done at each level is still O(n)
O(n) O(n) = O(n2) So worst case for Quicksort is O(n2) When does this happen
When the array is sorted to begin with
Typical case for quicksort If the array is sorted to begin with
Quicksort is terrible O(n2) It is possible to construct other bad
cases However Quicksort is usually O(n log2n) The constants are so good that Quicksort
is generally the fastest algorithm known Most real-world sorting is done by
Quicksort
Tweaking Quicksort Almost anything you can try to
ldquoimproverdquo Quicksort will actually slow it down
One good tweak is to switch to a different sorting method when the subarrays get small (say 10 or 12) Quicksort has too much overhead for small
array sizes For large arrays it might be a good
idea to check beforehand if the array is already sorted But there is a better tweak than this
Randomized Quick-Sort Select the pivot as a random element of the sequence The expected running time of randomized quick-sort on a
sequence of size n is O(n log n) The time spent at a level of the quick-sort tree is O(n) We show that the expected height of the quick-sort tree is
O(log n) good vs bad pivots
bull The probability of a good pivot is 12 thus we expect k2 good pivots out of k pivotsbull After a good pivot the size of each child sequence is at most 34 the size of the parent sequencebull After h pivots we expect (34)h2 n elementsbull The expected height h of the quick-sort tree is at most 2 log43n
Median of three Obviously it doesnrsquot make sense to sort the
array in order to find the median to use as a pivot
Instead compare just three elements of our (sub)arraymdashthe first the last and the middle Take the median (middle value) of these three as
pivot Itrsquos possible (but not easy) to construct cases
which will make this technique O(n2) Suppose we rearrange (sort) these three
numbers so that the smallest is in the first position the largest in the last position and the other in the middle This lets us simplify and speed up the partition
loop
Final comments Weissrsquos code shows some additional
optimizations on pp 246-247 Weiss chooses to stop both searches on
equality to pivot This design decision is debatable
Quicksort is the fastest known general sorting algorithm on average
For optimum speed the pivot must be chosen carefully
ldquoMedian of threerdquo is a good technique for choosing the pivot
There will be some cases where Quicksort runs in O(n2) time
Quick Sort A couple of notes about quick sort There are more optimal ways to
choose the pivot value (such as the median-of-three method)
Also when the subarrays get small it becomes more efficient to use the insertion sort as opposed to continued use of quick sort
Bubble Sort vs Quick Sort If we calculate the Big-O notation
we find that (in the average case) Bubble Sort O(n2) Quick Sort O(nlog2n)
(quick-sort runs in time O(n2) in the worst case)
Heap Sort Idea take the items that need to be
sorted and insert them into the heap By calling deleteHeap we remove the
smallest or largest element depending on whether or not we are working with a min- or max-heap respectively
Hence the elements are removed in ascending or descending order
Efficiency O(nlog2n)
Merge Sort Idea
Take the array you would like to sort and divide it in half to create 2 unsorted subarrays
Next sort each of the 2 subarrays Finally merge the 2 sorted subarrays
into 1 sorted array Efficiency O(nlog2n)
Merge Sort
Merge Sort Although the merge step produces
a sorted array we have overlooked a very important step
How did we sort the 2 halves before performing the merge step
We used merge sort
Merge Sort By continually calling the merge
sort algorithm we eventually get a subarray of size 1
Since an array with only 1 element is clearly sorted we can back out and merge 2 arrays of size 1
Merge Sort
Merge Sort The basic merging algorithm
consists of 2 input arrays (arrayA and arrayB) An ouput array (arrayC) 3 position holders (indexA indexB
indexC) which are initially set to the beginning of their respective arrays
Merge Sort The smaller of arrayA[indexA] and
arrayB[indexB] is copied into arrayC[indexC] and the appropriate position holders are advanced
When either input list is exhausted the remainder of the other list is copied into arrayC
Merge Sort
1 13 24 26index
A
indexC
arrayA
2 15 27 38index
B
arrayB
arrayC
We compare arrayA[indexA]with arrayB[indexB] Whichever value is smaller isplaced into arrayC[indexC]
1 lt 2 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1index
C
arrayA
2 15 27 38index
B
arrayB
arrayC
2 lt 13 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
13 lt 15 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
15 lt 24 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13 15indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
24 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
indexA
1 2 13 15 24indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
26 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
1 2 13 15 24 26indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
Since we have exhaustedone of the arrays arrayAwe simply copy the remaining items from theother array arrayB intoarrayC
Merge Sort
1 13 24
26
1 2 13 15 24 26 27 38
arrayA
2 15 27 38arrayB
arrayC
Efficiency Summary
Sort Worst Case Average Case
Insertion O(n2) O(n2)
Shell O(n15) O(n125)
Selection O(n2) O(n2)
Bubble O(n2) O(n2)
Quick O(n2) O(nlog2n)
Heap O(nlog2n) O(nlog2n)
Merge O(nlog2n) O(nlog2n)
More Sorting
radix sort bucket sort in-place sorting how fast can we sort
Radix Sort
Unlike other sorting methods radix sort considers the structure of the keys
Assume keys are represented in a base M number system (M = radix) ie if M = 2 the keys are represented in binary
Sorting is done by comparing bits in the same position
Extension to keys that are alphanumeric strings
Radix Exchange Sort
Examine bits from left to right1 Sort array with respect to leftmost bit
2 Partition array
3Recursionrecursively sort top subarray ignoring leftmost bitrecursively sort bottom subarray ignoring leftmost bit
Time to sort n b-bit numbers O(b n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Introduction One of the most common applications
in computer science is sorting the process through which data are arranged according to their values
If data were not ordered in some way we would spend an incredible amount of time trying to find the correct information
Introduction To appreciate this imagine trying
to find someonersquos number in the telephone book if the names were not sorted in some way
General Sorting Concepts Sorts are generally classified as either
internal or external An internal sort is a sort in which all of
the data is held in primary memory during the sorting process
An external sort uses primary memory for the data currently being sorted and secondary storage for any data that will not fit in primary memory
General Sorting Concepts For example a file of 20000
records may be sorted using an array that holds only 1000 records
Therefore only 1000 records are in primary memory at any given time
The other 19000 records are stored in secondary storage
Sort Order Data may be sorted in either
ascending or descending order The sort order identifies the
sequence of sorted data ascending or descending
If the order of the sort is not specified it is assumed to be ascending
Sort Stability Sort stability is an attribute of a
sort indicating that data elements with equal keys maintain their relative input order in the output
Consider the following example
Sort Stability
Note the unsorted data in (a) If we use a stable sort items with equal keys are
guaranteed to be sorted in the same order as they appear in the unsorted data
Sort Stability
If however we use an unstable sort data with equal keys can be sorted in any order (ie not necessarily in the same order as they appeared in the unsorted data)
Sort Efficiency Sort efficiency is a measure of
the relative efficiency of a sort It is usually an estimate of the
number of comparisons and moves required to order an unordered list
Passes During the sorting process the data is
traversed many times Each traversal of the data is referred to
as a sort pass Depending on the algorithm the sort
pass may traverse the whole list or just a section of the list
The sort pass may also include the placement of one or more elements into the sorted list
Types of Sorts We now discuss several sorting
methods Insertion Sort Shell Sort Selection Sort Bubble Sort Quick Sort Heap Sort Merge Sort
Insertion Sort In each pass of an insertion sort
one or more pieces of data are inserted into their correct location in an ordered list (just as a card player picks up cards and places them in his hand in order)
Insertion Sort In the insertion sort the list is divided into
2 parts Sorted Unsorted
In each pass the first element of the unsorted sublist is transferred to the sorted sublist by inserting it at the appropriate place
If we have a list of n elements it will take at most n-1 passes to sort the data
Insertion Sort
We can visualize this type of sort with the above figure
The first part of the list is the sorted portion which is separated by a ldquoconceptualrdquo wall from the unsorted portion of the list
Insertion SortHere we start with an unsorted list We leave the first data element alone and will start with the 2nd element of the list (the 1st element of the unsorted list)
On our first pass we look at the 2nd element of the list (thefirst element of the unsorted list) and place it in our sorted list in order
On our next pass we look at the 3rd element of the list (the1st element of the unsorted list) and place it in the sorted list in order
We continue in this fashion until the entire list has beensorted
Insertion Sort Example
In our first pass we compare the first 2 values Because theyare out of order we swap them
Now we look at the next 2 values Again they are out oforder so we swap them
Since we have swapped those values we need to comparethe previous 2 values to make sure that they are still in orderSince they are out of order we swap them and then continueon with the next 2 data valuesThese 2 values are out of order so we swap them and look atthe previous 2 values etc
Shell Sort Named after its creator Donald Shell the
shell sort is an improved version of the insertion sort
In the shell sort a list of N elements is divided into K segments where K is known as the increment
What this means is that instead of comparing adjacent values we will compare values that are a distance K apart
We will shrink K as we run through our algorithm
Shell SortPass Notes
1 77 62 14 9 30 21 80 25 70 55 Swap2 21 62 14 9 30 77 80 25 70 55 In order3 21 62 14 9 30 77 80 25 70 55 In order4 21 62 14 9 30 77 80 25 70 55 In order5 21 62 14 9 30 77 80 25 70 55 In order
List (K=5)
Just as in the straight insertion sort we compare 2 values andswap them if they are out of order However in the shell sort we compare values that are a distance K apartOnce we have completed going through the elements in our
list with K=5 we decrease K and continue the process
Shell SortPass Notes
6 21 62 14 9 30 77 80 25 70 55 Swap7 14 62 21 9 30 77 80 25 70 55 Swap8 14 9 21 62 30 77 80 25 70 55 In order9 14 9 21 62 30 77 80 25 70 55 In order10 14 9 21 62 30 77 80 25 70 55 In order11 14 9 21 62 30 77 80 25 70 55 Swap
14 9 21 62 30 25 80 77 70 55 Swap14 9 21 25 30 62 80 77 70 55 In order
12 14 9 21 25 30 62 80 77 70 55 Swap14 9 21 25 30 62 70 77 80 55 In order
13 14 9 21 25 30 62 70 77 80 55 Swap14 9 21 25 30 62 70 55 80 77 Swap14 9 21 25 30 55 70 62 80 77 In order
List (K=2)
Here we have reduced K to 2 Just as in the insertion sort if we swap 2 values we have to go back and compare theprevious 2 values to make sure they are still in order
Shell Sort
All shell sorts will terminate by running an insertion sort(ie K=1) However using the larger values of K first has helped to sort our list so that the straight insertion sort will runfaster
Shell Sort There are many schools of thought
on what the increment should be in the shell sort
Also note that just because an increment is optimal on one list it might not be optimal for another list
Insertion Sort vs Shell Sort Comparing the Big-O notation (for the
average case) we find that Insertion O(n2) Shell O(n125) empirically determined
Although this doesnrsquot seem like much of a gain it makes a big difference as n gets large
Note that in the worst case the Shell sort has an efficiency of O(n2)
However using a special incrementing technique this worst case can be reduced to O(n15)
Insertion Sort vs Shell Sort(Average Case)
Straight Insertion Shell25 625 55100 10000 316500 250000 2364
1000 1000000 56232000 4000000 13374
Number of Loopsn
Selection Sort Imagine some data that you can examine
all at once To sort it you could select the smallest
element and put it in its place select the next smallest and put it in its place etc
For a card player this process is analogous to looking at an entire hand of cards and ordering them by selecting cards one at a time and placing them in their proper order
Selection Sort The selection sort follows this idea Given a list of data to be sorted
we simply select the smallest item and place it in a sorted list
We then repeat these steps until the list is sorted
Selection Sort In the selection sort the list at any moment
is divided into 2 sublists sorted and unsorted separated by a ldquoconceptualrdquo wall
We select the smallest element from the unsorted sublist and exchange it with the element at the beginning of the unsorted data
After each selection and exchange the wall between the 2 sublists moves ndash increasing the number of sorted elements and decreasing the number of unsorted elements
Selection Sort
We start with an unsorted list We search this list for thesmallest element We then exchange the smallest element (8)with the first element in the unsorted list (23)
Again we search the unsorted list for the smallest element We then exchange the smallest element (23) with the first element in the unsorted list (78)
This process continues until the list is fully sorted
Bubble Sort In the bubble sort the list at any
moment is divided into 2 sublists sorted and unsorted
The smallest element is ldquobubbledrdquo from the unsorted sublist to the sorted sublist
Bubble Sort
23 78 45 8 56 3232 56
We start with 32 and compareit with 56 Because 32 is lessthan 56 we swap the twoand step down one element
We then compare 32 and 8Because 32 is not less than 8we do not swap these elements
We step down one element andcompare 45 and 8 They areout of sequence so we swapthem and step down again
8 45
We step down again and compare 8 with 78 These two elements are swapped
8 78
Finally 8 is compared with 23and swapped We thencontinue this process back with56 hellip
8 23
1 Pass of the Bubble Sort
Quick Sort In the bubble sort consecutive items
are compared and possibly exchanged on each pass through the list
This means that many exchanges may be needed to move an element to its correct position
Quick sort is more efficient than bubble sort because a typical exchange involves elements that are far apart so fewer exchanges are required to correctly position an element
Idea of Quick Sort1) Select pick an element
2) Divide rearrange elements so that x goes to its final position E
3) Recurse and Conquer recursively sort
Quick Sort Each iteration of the quick sort
selects an element known as the pivot and divides the list into 3 groups Elements whose keys are less than (or
equal to) the pivotrsquos key The pivot element Elements whose keys are greater than
(or equal to) the pivotrsquos key
Quick Sort The sorting then continues by
quick sorting the left partition followed by quick sorting the right partition
The basic algorithm is as follows
Quick Sort1) Partitioning Step Take an element in the
unsorted array and determine its final location in the sorted array This occurs when all values to the left of the element in the array are less than (or equal to) the element and all values to the right of the element are greater than (or equal to) the element We now have 1 element in its proper location and two unsorted subarrays
2) Recursive Step Perform step 1 on each unsorted subarray
Quick Sort Each time step 1 is performed on a
subarray another element is placed in its final location of the sorted array and two unsorted subarrays are created
When a subarray consists of one element that subarray is sorted
Therefore that element is in its final location
Quick Sort There are several partitioning strategies
used in practice (ie several ldquoversionsrdquo of quick sort) but the one we are about to describe is known to work well
For simplicity we will choose the last element to be the pivot element
We could also chose a different pivot element and swap it with the last element in the array
Quick Sort Below is the array we would like to
sort1 4 8 9 0 11 5 10 7 6
Quick Sort The index left starts at the first element
and right starts at the next-to-last element
We want to move all the elements smaller than the pivot to the left part of the array and all the elements larger than the pivot to the right part
1 4 8 9 0 11 5 10 7 6left righ
t
Quick Sort We move left to the right skipping
over elements that are smaller than the pivot
1 4 8 9 0 11 5 10 7 6left righ
t
Quick Sort We then move right to the left skipping
over elements that are greater than the pivot
When left and right have stopped left is on an element greater than (or equal to) the pivot and right is on an element smaller than (or equal to) the pivot
1 4 8 9 0 11 5 10 7 6left right
Quick Sort If left is to the left of right (or if left
= right) those elements are swapped
1 4 8 9 0 11 5 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
Quick Sort The effect is to push a large
element to the right and a small element to the left
We then repeat the process until left and right cross
Quick Sort
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6left righ
t
Quick Sort
1 4 5 0 9 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6righ
tleft
Quick Sort
At this point left and right have crossed so no swap is performed
The final part of the partitioning is to swap the pivot element with left
1 4 5 0 9 11 8 10 7 6righ
tleft
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort Note that all elements to the left of
the pivot are less than (or equal to) the pivot and all elements to the right of the pivot are greater than (or equal to) the pivot
Hence the pivot element has been placed in its final sorted position
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort We now repeat the process using
the sub-arrays to the left and right of the pivot
1 4 5 0 6 11 8 10 7 9
1 4 5 0 11 8 10 7 9
Analysis of quicksortmdashbest case
Suppose each partition operation divides the array almost exactly in half
Then the depth of the recursion in log2n Because thatrsquos how many times we can
halve n However there are many recursions at
each level How can we figure this out We note that
Each partition is linear over its subarray All the partitions at one level cover the array
Partitioning at various levels
Best case II We cut the array size in half each time So the depth of the recursion in log2n At each level of the recursion all the
partitions at that level do work that is linear in n
O(log2n) O(n) = O(n log2n)
What about the worst case
Worst case In the worst case partitioning always
divides the size n array into these three parts A length one part containing the pivot itself A length zero part and A length n-1 part containing everything else
We donrsquot recur on the zero-length part Recurring on the length n-1 part requires
(in the worst case) recurring to depth n-1
Worst case partitioning
Worst case for quicksort In the worst case recursion may be
O(n) levels deep (for an array of size n)
But the partitioning work done at each level is still O(n)
O(n) O(n) = O(n2) So worst case for Quicksort is O(n2) When does this happen
When the array is sorted to begin with
Typical case for quicksort If the array is sorted to begin with
Quicksort is terrible O(n2) It is possible to construct other bad
cases However Quicksort is usually O(n log2n) The constants are so good that Quicksort
is generally the fastest algorithm known Most real-world sorting is done by
Quicksort
Tweaking Quicksort Almost anything you can try to
ldquoimproverdquo Quicksort will actually slow it down
One good tweak is to switch to a different sorting method when the subarrays get small (say 10 or 12) Quicksort has too much overhead for small
array sizes For large arrays it might be a good
idea to check beforehand if the array is already sorted But there is a better tweak than this
Randomized Quick-Sort Select the pivot as a random element of the sequence The expected running time of randomized quick-sort on a
sequence of size n is O(n log n) The time spent at a level of the quick-sort tree is O(n) We show that the expected height of the quick-sort tree is
O(log n) good vs bad pivots
bull The probability of a good pivot is 12 thus we expect k2 good pivots out of k pivotsbull After a good pivot the size of each child sequence is at most 34 the size of the parent sequencebull After h pivots we expect (34)h2 n elementsbull The expected height h of the quick-sort tree is at most 2 log43n
Median of three Obviously it doesnrsquot make sense to sort the
array in order to find the median to use as a pivot
Instead compare just three elements of our (sub)arraymdashthe first the last and the middle Take the median (middle value) of these three as
pivot Itrsquos possible (but not easy) to construct cases
which will make this technique O(n2) Suppose we rearrange (sort) these three
numbers so that the smallest is in the first position the largest in the last position and the other in the middle This lets us simplify and speed up the partition
loop
Final comments Weissrsquos code shows some additional
optimizations on pp 246-247 Weiss chooses to stop both searches on
equality to pivot This design decision is debatable
Quicksort is the fastest known general sorting algorithm on average
For optimum speed the pivot must be chosen carefully
ldquoMedian of threerdquo is a good technique for choosing the pivot
There will be some cases where Quicksort runs in O(n2) time
Quick Sort A couple of notes about quick sort There are more optimal ways to
choose the pivot value (such as the median-of-three method)
Also when the subarrays get small it becomes more efficient to use the insertion sort as opposed to continued use of quick sort
Bubble Sort vs Quick Sort If we calculate the Big-O notation
we find that (in the average case) Bubble Sort O(n2) Quick Sort O(nlog2n)
(quick-sort runs in time O(n2) in the worst case)
Heap Sort Idea take the items that need to be
sorted and insert them into the heap By calling deleteHeap we remove the
smallest or largest element depending on whether or not we are working with a min- or max-heap respectively
Hence the elements are removed in ascending or descending order
Efficiency O(nlog2n)
Merge Sort Idea
Take the array you would like to sort and divide it in half to create 2 unsorted subarrays
Next sort each of the 2 subarrays Finally merge the 2 sorted subarrays
into 1 sorted array Efficiency O(nlog2n)
Merge Sort
Merge Sort Although the merge step produces
a sorted array we have overlooked a very important step
How did we sort the 2 halves before performing the merge step
We used merge sort
Merge Sort By continually calling the merge
sort algorithm we eventually get a subarray of size 1
Since an array with only 1 element is clearly sorted we can back out and merge 2 arrays of size 1
Merge Sort
Merge Sort The basic merging algorithm
consists of 2 input arrays (arrayA and arrayB) An ouput array (arrayC) 3 position holders (indexA indexB
indexC) which are initially set to the beginning of their respective arrays
Merge Sort The smaller of arrayA[indexA] and
arrayB[indexB] is copied into arrayC[indexC] and the appropriate position holders are advanced
When either input list is exhausted the remainder of the other list is copied into arrayC
Merge Sort
1 13 24 26index
A
indexC
arrayA
2 15 27 38index
B
arrayB
arrayC
We compare arrayA[indexA]with arrayB[indexB] Whichever value is smaller isplaced into arrayC[indexC]
1 lt 2 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1index
C
arrayA
2 15 27 38index
B
arrayB
arrayC
2 lt 13 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
13 lt 15 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
15 lt 24 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13 15indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
24 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
indexA
1 2 13 15 24indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
26 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
1 2 13 15 24 26indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
Since we have exhaustedone of the arrays arrayAwe simply copy the remaining items from theother array arrayB intoarrayC
Merge Sort
1 13 24
26
1 2 13 15 24 26 27 38
arrayA
2 15 27 38arrayB
arrayC
Efficiency Summary
Sort Worst Case Average Case
Insertion O(n2) O(n2)
Shell O(n15) O(n125)
Selection O(n2) O(n2)
Bubble O(n2) O(n2)
Quick O(n2) O(nlog2n)
Heap O(nlog2n) O(nlog2n)
Merge O(nlog2n) O(nlog2n)
More Sorting
radix sort bucket sort in-place sorting how fast can we sort
Radix Sort
Unlike other sorting methods radix sort considers the structure of the keys
Assume keys are represented in a base M number system (M = radix) ie if M = 2 the keys are represented in binary
Sorting is done by comparing bits in the same position
Extension to keys that are alphanumeric strings
Radix Exchange Sort
Examine bits from left to right1 Sort array with respect to leftmost bit
2 Partition array
3Recursionrecursively sort top subarray ignoring leftmost bitrecursively sort bottom subarray ignoring leftmost bit
Time to sort n b-bit numbers O(b n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Introduction To appreciate this imagine trying
to find someonersquos number in the telephone book if the names were not sorted in some way
General Sorting Concepts Sorts are generally classified as either
internal or external An internal sort is a sort in which all of
the data is held in primary memory during the sorting process
An external sort uses primary memory for the data currently being sorted and secondary storage for any data that will not fit in primary memory
General Sorting Concepts For example a file of 20000
records may be sorted using an array that holds only 1000 records
Therefore only 1000 records are in primary memory at any given time
The other 19000 records are stored in secondary storage
Sort Order Data may be sorted in either
ascending or descending order The sort order identifies the
sequence of sorted data ascending or descending
If the order of the sort is not specified it is assumed to be ascending
Sort Stability Sort stability is an attribute of a
sort indicating that data elements with equal keys maintain their relative input order in the output
Consider the following example
Sort Stability
Note the unsorted data in (a) If we use a stable sort items with equal keys are
guaranteed to be sorted in the same order as they appear in the unsorted data
Sort Stability
If however we use an unstable sort data with equal keys can be sorted in any order (ie not necessarily in the same order as they appeared in the unsorted data)
Sort Efficiency Sort efficiency is a measure of
the relative efficiency of a sort It is usually an estimate of the
number of comparisons and moves required to order an unordered list
Passes During the sorting process the data is
traversed many times Each traversal of the data is referred to
as a sort pass Depending on the algorithm the sort
pass may traverse the whole list or just a section of the list
The sort pass may also include the placement of one or more elements into the sorted list
Types of Sorts We now discuss several sorting
methods Insertion Sort Shell Sort Selection Sort Bubble Sort Quick Sort Heap Sort Merge Sort
Insertion Sort In each pass of an insertion sort
one or more pieces of data are inserted into their correct location in an ordered list (just as a card player picks up cards and places them in his hand in order)
Insertion Sort In the insertion sort the list is divided into
2 parts Sorted Unsorted
In each pass the first element of the unsorted sublist is transferred to the sorted sublist by inserting it at the appropriate place
If we have a list of n elements it will take at most n-1 passes to sort the data
Insertion Sort
We can visualize this type of sort with the above figure
The first part of the list is the sorted portion which is separated by a ldquoconceptualrdquo wall from the unsorted portion of the list
Insertion SortHere we start with an unsorted list We leave the first data element alone and will start with the 2nd element of the list (the 1st element of the unsorted list)
On our first pass we look at the 2nd element of the list (thefirst element of the unsorted list) and place it in our sorted list in order
On our next pass we look at the 3rd element of the list (the1st element of the unsorted list) and place it in the sorted list in order
We continue in this fashion until the entire list has beensorted
Insertion Sort Example
In our first pass we compare the first 2 values Because theyare out of order we swap them
Now we look at the next 2 values Again they are out oforder so we swap them
Since we have swapped those values we need to comparethe previous 2 values to make sure that they are still in orderSince they are out of order we swap them and then continueon with the next 2 data valuesThese 2 values are out of order so we swap them and look atthe previous 2 values etc
Shell Sort Named after its creator Donald Shell the
shell sort is an improved version of the insertion sort
In the shell sort a list of N elements is divided into K segments where K is known as the increment
What this means is that instead of comparing adjacent values we will compare values that are a distance K apart
We will shrink K as we run through our algorithm
Shell SortPass Notes
1 77 62 14 9 30 21 80 25 70 55 Swap2 21 62 14 9 30 77 80 25 70 55 In order3 21 62 14 9 30 77 80 25 70 55 In order4 21 62 14 9 30 77 80 25 70 55 In order5 21 62 14 9 30 77 80 25 70 55 In order
List (K=5)
Just as in the straight insertion sort we compare 2 values andswap them if they are out of order However in the shell sort we compare values that are a distance K apartOnce we have completed going through the elements in our
list with K=5 we decrease K and continue the process
Shell SortPass Notes
6 21 62 14 9 30 77 80 25 70 55 Swap7 14 62 21 9 30 77 80 25 70 55 Swap8 14 9 21 62 30 77 80 25 70 55 In order9 14 9 21 62 30 77 80 25 70 55 In order10 14 9 21 62 30 77 80 25 70 55 In order11 14 9 21 62 30 77 80 25 70 55 Swap
14 9 21 62 30 25 80 77 70 55 Swap14 9 21 25 30 62 80 77 70 55 In order
12 14 9 21 25 30 62 80 77 70 55 Swap14 9 21 25 30 62 70 77 80 55 In order
13 14 9 21 25 30 62 70 77 80 55 Swap14 9 21 25 30 62 70 55 80 77 Swap14 9 21 25 30 55 70 62 80 77 In order
List (K=2)
Here we have reduced K to 2 Just as in the insertion sort if we swap 2 values we have to go back and compare theprevious 2 values to make sure they are still in order
Shell Sort
All shell sorts will terminate by running an insertion sort(ie K=1) However using the larger values of K first has helped to sort our list so that the straight insertion sort will runfaster
Shell Sort There are many schools of thought
on what the increment should be in the shell sort
Also note that just because an increment is optimal on one list it might not be optimal for another list
Insertion Sort vs Shell Sort Comparing the Big-O notation (for the
average case) we find that Insertion O(n2) Shell O(n125) empirically determined
Although this doesnrsquot seem like much of a gain it makes a big difference as n gets large
Note that in the worst case the Shell sort has an efficiency of O(n2)
However using a special incrementing technique this worst case can be reduced to O(n15)
Insertion Sort vs Shell Sort(Average Case)
Straight Insertion Shell25 625 55100 10000 316500 250000 2364
1000 1000000 56232000 4000000 13374
Number of Loopsn
Selection Sort Imagine some data that you can examine
all at once To sort it you could select the smallest
element and put it in its place select the next smallest and put it in its place etc
For a card player this process is analogous to looking at an entire hand of cards and ordering them by selecting cards one at a time and placing them in their proper order
Selection Sort The selection sort follows this idea Given a list of data to be sorted
we simply select the smallest item and place it in a sorted list
We then repeat these steps until the list is sorted
Selection Sort In the selection sort the list at any moment
is divided into 2 sublists sorted and unsorted separated by a ldquoconceptualrdquo wall
We select the smallest element from the unsorted sublist and exchange it with the element at the beginning of the unsorted data
After each selection and exchange the wall between the 2 sublists moves ndash increasing the number of sorted elements and decreasing the number of unsorted elements
Selection Sort
We start with an unsorted list We search this list for thesmallest element We then exchange the smallest element (8)with the first element in the unsorted list (23)
Again we search the unsorted list for the smallest element We then exchange the smallest element (23) with the first element in the unsorted list (78)
This process continues until the list is fully sorted
Bubble Sort In the bubble sort the list at any
moment is divided into 2 sublists sorted and unsorted
The smallest element is ldquobubbledrdquo from the unsorted sublist to the sorted sublist
Bubble Sort
23 78 45 8 56 3232 56
We start with 32 and compareit with 56 Because 32 is lessthan 56 we swap the twoand step down one element
We then compare 32 and 8Because 32 is not less than 8we do not swap these elements
We step down one element andcompare 45 and 8 They areout of sequence so we swapthem and step down again
8 45
We step down again and compare 8 with 78 These two elements are swapped
8 78
Finally 8 is compared with 23and swapped We thencontinue this process back with56 hellip
8 23
1 Pass of the Bubble Sort
Quick Sort In the bubble sort consecutive items
are compared and possibly exchanged on each pass through the list
This means that many exchanges may be needed to move an element to its correct position
Quick sort is more efficient than bubble sort because a typical exchange involves elements that are far apart so fewer exchanges are required to correctly position an element
Idea of Quick Sort1) Select pick an element
2) Divide rearrange elements so that x goes to its final position E
3) Recurse and Conquer recursively sort
Quick Sort Each iteration of the quick sort
selects an element known as the pivot and divides the list into 3 groups Elements whose keys are less than (or
equal to) the pivotrsquos key The pivot element Elements whose keys are greater than
(or equal to) the pivotrsquos key
Quick Sort The sorting then continues by
quick sorting the left partition followed by quick sorting the right partition
The basic algorithm is as follows
Quick Sort1) Partitioning Step Take an element in the
unsorted array and determine its final location in the sorted array This occurs when all values to the left of the element in the array are less than (or equal to) the element and all values to the right of the element are greater than (or equal to) the element We now have 1 element in its proper location and two unsorted subarrays
2) Recursive Step Perform step 1 on each unsorted subarray
Quick Sort Each time step 1 is performed on a
subarray another element is placed in its final location of the sorted array and two unsorted subarrays are created
When a subarray consists of one element that subarray is sorted
Therefore that element is in its final location
Quick Sort There are several partitioning strategies
used in practice (ie several ldquoversionsrdquo of quick sort) but the one we are about to describe is known to work well
For simplicity we will choose the last element to be the pivot element
We could also chose a different pivot element and swap it with the last element in the array
Quick Sort Below is the array we would like to
sort1 4 8 9 0 11 5 10 7 6
Quick Sort The index left starts at the first element
and right starts at the next-to-last element
We want to move all the elements smaller than the pivot to the left part of the array and all the elements larger than the pivot to the right part
1 4 8 9 0 11 5 10 7 6left righ
t
Quick Sort We move left to the right skipping
over elements that are smaller than the pivot
1 4 8 9 0 11 5 10 7 6left righ
t
Quick Sort We then move right to the left skipping
over elements that are greater than the pivot
When left and right have stopped left is on an element greater than (or equal to) the pivot and right is on an element smaller than (or equal to) the pivot
1 4 8 9 0 11 5 10 7 6left right
Quick Sort If left is to the left of right (or if left
= right) those elements are swapped
1 4 8 9 0 11 5 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
Quick Sort The effect is to push a large
element to the right and a small element to the left
We then repeat the process until left and right cross
Quick Sort
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6left righ
t
Quick Sort
1 4 5 0 9 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6righ
tleft
Quick Sort
At this point left and right have crossed so no swap is performed
The final part of the partitioning is to swap the pivot element with left
1 4 5 0 9 11 8 10 7 6righ
tleft
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort Note that all elements to the left of
the pivot are less than (or equal to) the pivot and all elements to the right of the pivot are greater than (or equal to) the pivot
Hence the pivot element has been placed in its final sorted position
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort We now repeat the process using
the sub-arrays to the left and right of the pivot
1 4 5 0 6 11 8 10 7 9
1 4 5 0 11 8 10 7 9
Analysis of quicksortmdashbest case
Suppose each partition operation divides the array almost exactly in half
Then the depth of the recursion in log2n Because thatrsquos how many times we can
halve n However there are many recursions at
each level How can we figure this out We note that
Each partition is linear over its subarray All the partitions at one level cover the array
Partitioning at various levels
Best case II We cut the array size in half each time So the depth of the recursion in log2n At each level of the recursion all the
partitions at that level do work that is linear in n
O(log2n) O(n) = O(n log2n)
What about the worst case
Worst case In the worst case partitioning always
divides the size n array into these three parts A length one part containing the pivot itself A length zero part and A length n-1 part containing everything else
We donrsquot recur on the zero-length part Recurring on the length n-1 part requires
(in the worst case) recurring to depth n-1
Worst case partitioning
Worst case for quicksort In the worst case recursion may be
O(n) levels deep (for an array of size n)
But the partitioning work done at each level is still O(n)
O(n) O(n) = O(n2) So worst case for Quicksort is O(n2) When does this happen
When the array is sorted to begin with
Typical case for quicksort If the array is sorted to begin with
Quicksort is terrible O(n2) It is possible to construct other bad
cases However Quicksort is usually O(n log2n) The constants are so good that Quicksort
is generally the fastest algorithm known Most real-world sorting is done by
Quicksort
Tweaking Quicksort Almost anything you can try to
ldquoimproverdquo Quicksort will actually slow it down
One good tweak is to switch to a different sorting method when the subarrays get small (say 10 or 12) Quicksort has too much overhead for small
array sizes For large arrays it might be a good
idea to check beforehand if the array is already sorted But there is a better tweak than this
Randomized Quick-Sort Select the pivot as a random element of the sequence The expected running time of randomized quick-sort on a
sequence of size n is O(n log n) The time spent at a level of the quick-sort tree is O(n) We show that the expected height of the quick-sort tree is
O(log n) good vs bad pivots
bull The probability of a good pivot is 12 thus we expect k2 good pivots out of k pivotsbull After a good pivot the size of each child sequence is at most 34 the size of the parent sequencebull After h pivots we expect (34)h2 n elementsbull The expected height h of the quick-sort tree is at most 2 log43n
Median of three Obviously it doesnrsquot make sense to sort the
array in order to find the median to use as a pivot
Instead compare just three elements of our (sub)arraymdashthe first the last and the middle Take the median (middle value) of these three as
pivot Itrsquos possible (but not easy) to construct cases
which will make this technique O(n2) Suppose we rearrange (sort) these three
numbers so that the smallest is in the first position the largest in the last position and the other in the middle This lets us simplify and speed up the partition
loop
Final comments Weissrsquos code shows some additional
optimizations on pp 246-247 Weiss chooses to stop both searches on
equality to pivot This design decision is debatable
Quicksort is the fastest known general sorting algorithm on average
For optimum speed the pivot must be chosen carefully
ldquoMedian of threerdquo is a good technique for choosing the pivot
There will be some cases where Quicksort runs in O(n2) time
Quick Sort A couple of notes about quick sort There are more optimal ways to
choose the pivot value (such as the median-of-three method)
Also when the subarrays get small it becomes more efficient to use the insertion sort as opposed to continued use of quick sort
Bubble Sort vs Quick Sort If we calculate the Big-O notation
we find that (in the average case) Bubble Sort O(n2) Quick Sort O(nlog2n)
(quick-sort runs in time O(n2) in the worst case)
Heap Sort Idea take the items that need to be
sorted and insert them into the heap By calling deleteHeap we remove the
smallest or largest element depending on whether or not we are working with a min- or max-heap respectively
Hence the elements are removed in ascending or descending order
Efficiency O(nlog2n)
Merge Sort Idea
Take the array you would like to sort and divide it in half to create 2 unsorted subarrays
Next sort each of the 2 subarrays Finally merge the 2 sorted subarrays
into 1 sorted array Efficiency O(nlog2n)
Merge Sort
Merge Sort Although the merge step produces
a sorted array we have overlooked a very important step
How did we sort the 2 halves before performing the merge step
We used merge sort
Merge Sort By continually calling the merge
sort algorithm we eventually get a subarray of size 1
Since an array with only 1 element is clearly sorted we can back out and merge 2 arrays of size 1
Merge Sort
Merge Sort The basic merging algorithm
consists of 2 input arrays (arrayA and arrayB) An ouput array (arrayC) 3 position holders (indexA indexB
indexC) which are initially set to the beginning of their respective arrays
Merge Sort The smaller of arrayA[indexA] and
arrayB[indexB] is copied into arrayC[indexC] and the appropriate position holders are advanced
When either input list is exhausted the remainder of the other list is copied into arrayC
Merge Sort
1 13 24 26index
A
indexC
arrayA
2 15 27 38index
B
arrayB
arrayC
We compare arrayA[indexA]with arrayB[indexB] Whichever value is smaller isplaced into arrayC[indexC]
1 lt 2 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1index
C
arrayA
2 15 27 38index
B
arrayB
arrayC
2 lt 13 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
13 lt 15 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
15 lt 24 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13 15indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
24 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
indexA
1 2 13 15 24indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
26 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
1 2 13 15 24 26indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
Since we have exhaustedone of the arrays arrayAwe simply copy the remaining items from theother array arrayB intoarrayC
Merge Sort
1 13 24
26
1 2 13 15 24 26 27 38
arrayA
2 15 27 38arrayB
arrayC
Efficiency Summary
Sort Worst Case Average Case
Insertion O(n2) O(n2)
Shell O(n15) O(n125)
Selection O(n2) O(n2)
Bubble O(n2) O(n2)
Quick O(n2) O(nlog2n)
Heap O(nlog2n) O(nlog2n)
Merge O(nlog2n) O(nlog2n)
More Sorting
radix sort bucket sort in-place sorting how fast can we sort
Radix Sort
Unlike other sorting methods radix sort considers the structure of the keys
Assume keys are represented in a base M number system (M = radix) ie if M = 2 the keys are represented in binary
Sorting is done by comparing bits in the same position
Extension to keys that are alphanumeric strings
Radix Exchange Sort
Examine bits from left to right1 Sort array with respect to leftmost bit
2 Partition array
3Recursionrecursively sort top subarray ignoring leftmost bitrecursively sort bottom subarray ignoring leftmost bit
Time to sort n b-bit numbers O(b n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
General Sorting Concepts Sorts are generally classified as either
internal or external An internal sort is a sort in which all of
the data is held in primary memory during the sorting process
An external sort uses primary memory for the data currently being sorted and secondary storage for any data that will not fit in primary memory
General Sorting Concepts For example a file of 20000
records may be sorted using an array that holds only 1000 records
Therefore only 1000 records are in primary memory at any given time
The other 19000 records are stored in secondary storage
Sort Order Data may be sorted in either
ascending or descending order The sort order identifies the
sequence of sorted data ascending or descending
If the order of the sort is not specified it is assumed to be ascending
Sort Stability Sort stability is an attribute of a
sort indicating that data elements with equal keys maintain their relative input order in the output
Consider the following example
Sort Stability
Note the unsorted data in (a) If we use a stable sort items with equal keys are
guaranteed to be sorted in the same order as they appear in the unsorted data
Sort Stability
If however we use an unstable sort data with equal keys can be sorted in any order (ie not necessarily in the same order as they appeared in the unsorted data)
Sort Efficiency Sort efficiency is a measure of
the relative efficiency of a sort It is usually an estimate of the
number of comparisons and moves required to order an unordered list
Passes During the sorting process the data is
traversed many times Each traversal of the data is referred to
as a sort pass Depending on the algorithm the sort
pass may traverse the whole list or just a section of the list
The sort pass may also include the placement of one or more elements into the sorted list
Types of Sorts We now discuss several sorting
methods Insertion Sort Shell Sort Selection Sort Bubble Sort Quick Sort Heap Sort Merge Sort
Insertion Sort In each pass of an insertion sort
one or more pieces of data are inserted into their correct location in an ordered list (just as a card player picks up cards and places them in his hand in order)
Insertion Sort In the insertion sort the list is divided into
2 parts Sorted Unsorted
In each pass the first element of the unsorted sublist is transferred to the sorted sublist by inserting it at the appropriate place
If we have a list of n elements it will take at most n-1 passes to sort the data
Insertion Sort
We can visualize this type of sort with the above figure
The first part of the list is the sorted portion which is separated by a ldquoconceptualrdquo wall from the unsorted portion of the list
Insertion SortHere we start with an unsorted list We leave the first data element alone and will start with the 2nd element of the list (the 1st element of the unsorted list)
On our first pass we look at the 2nd element of the list (thefirst element of the unsorted list) and place it in our sorted list in order
On our next pass we look at the 3rd element of the list (the1st element of the unsorted list) and place it in the sorted list in order
We continue in this fashion until the entire list has beensorted
Insertion Sort Example
In our first pass we compare the first 2 values Because theyare out of order we swap them
Now we look at the next 2 values Again they are out oforder so we swap them
Since we have swapped those values we need to comparethe previous 2 values to make sure that they are still in orderSince they are out of order we swap them and then continueon with the next 2 data valuesThese 2 values are out of order so we swap them and look atthe previous 2 values etc
Shell Sort Named after its creator Donald Shell the
shell sort is an improved version of the insertion sort
In the shell sort a list of N elements is divided into K segments where K is known as the increment
What this means is that instead of comparing adjacent values we will compare values that are a distance K apart
We will shrink K as we run through our algorithm
Shell SortPass Notes
1 77 62 14 9 30 21 80 25 70 55 Swap2 21 62 14 9 30 77 80 25 70 55 In order3 21 62 14 9 30 77 80 25 70 55 In order4 21 62 14 9 30 77 80 25 70 55 In order5 21 62 14 9 30 77 80 25 70 55 In order
List (K=5)
Just as in the straight insertion sort we compare 2 values andswap them if they are out of order However in the shell sort we compare values that are a distance K apartOnce we have completed going through the elements in our
list with K=5 we decrease K and continue the process
Shell SortPass Notes
6 21 62 14 9 30 77 80 25 70 55 Swap7 14 62 21 9 30 77 80 25 70 55 Swap8 14 9 21 62 30 77 80 25 70 55 In order9 14 9 21 62 30 77 80 25 70 55 In order10 14 9 21 62 30 77 80 25 70 55 In order11 14 9 21 62 30 77 80 25 70 55 Swap
14 9 21 62 30 25 80 77 70 55 Swap14 9 21 25 30 62 80 77 70 55 In order
12 14 9 21 25 30 62 80 77 70 55 Swap14 9 21 25 30 62 70 77 80 55 In order
13 14 9 21 25 30 62 70 77 80 55 Swap14 9 21 25 30 62 70 55 80 77 Swap14 9 21 25 30 55 70 62 80 77 In order
List (K=2)
Here we have reduced K to 2 Just as in the insertion sort if we swap 2 values we have to go back and compare theprevious 2 values to make sure they are still in order
Shell Sort
All shell sorts will terminate by running an insertion sort(ie K=1) However using the larger values of K first has helped to sort our list so that the straight insertion sort will runfaster
Shell Sort There are many schools of thought
on what the increment should be in the shell sort
Also note that just because an increment is optimal on one list it might not be optimal for another list
Insertion Sort vs Shell Sort Comparing the Big-O notation (for the
average case) we find that Insertion O(n2) Shell O(n125) empirically determined
Although this doesnrsquot seem like much of a gain it makes a big difference as n gets large
Note that in the worst case the Shell sort has an efficiency of O(n2)
However using a special incrementing technique this worst case can be reduced to O(n15)
Insertion Sort vs Shell Sort(Average Case)
Straight Insertion Shell25 625 55100 10000 316500 250000 2364
1000 1000000 56232000 4000000 13374
Number of Loopsn
Selection Sort Imagine some data that you can examine
all at once To sort it you could select the smallest
element and put it in its place select the next smallest and put it in its place etc
For a card player this process is analogous to looking at an entire hand of cards and ordering them by selecting cards one at a time and placing them in their proper order
Selection Sort The selection sort follows this idea Given a list of data to be sorted
we simply select the smallest item and place it in a sorted list
We then repeat these steps until the list is sorted
Selection Sort In the selection sort the list at any moment
is divided into 2 sublists sorted and unsorted separated by a ldquoconceptualrdquo wall
We select the smallest element from the unsorted sublist and exchange it with the element at the beginning of the unsorted data
After each selection and exchange the wall between the 2 sublists moves ndash increasing the number of sorted elements and decreasing the number of unsorted elements
Selection Sort
We start with an unsorted list We search this list for thesmallest element We then exchange the smallest element (8)with the first element in the unsorted list (23)
Again we search the unsorted list for the smallest element We then exchange the smallest element (23) with the first element in the unsorted list (78)
This process continues until the list is fully sorted
Bubble Sort In the bubble sort the list at any
moment is divided into 2 sublists sorted and unsorted
The smallest element is ldquobubbledrdquo from the unsorted sublist to the sorted sublist
Bubble Sort
23 78 45 8 56 3232 56
We start with 32 and compareit with 56 Because 32 is lessthan 56 we swap the twoand step down one element
We then compare 32 and 8Because 32 is not less than 8we do not swap these elements
We step down one element andcompare 45 and 8 They areout of sequence so we swapthem and step down again
8 45
We step down again and compare 8 with 78 These two elements are swapped
8 78
Finally 8 is compared with 23and swapped We thencontinue this process back with56 hellip
8 23
1 Pass of the Bubble Sort
Quick Sort In the bubble sort consecutive items
are compared and possibly exchanged on each pass through the list
This means that many exchanges may be needed to move an element to its correct position
Quick sort is more efficient than bubble sort because a typical exchange involves elements that are far apart so fewer exchanges are required to correctly position an element
Idea of Quick Sort1) Select pick an element
2) Divide rearrange elements so that x goes to its final position E
3) Recurse and Conquer recursively sort
Quick Sort Each iteration of the quick sort
selects an element known as the pivot and divides the list into 3 groups Elements whose keys are less than (or
equal to) the pivotrsquos key The pivot element Elements whose keys are greater than
(or equal to) the pivotrsquos key
Quick Sort The sorting then continues by
quick sorting the left partition followed by quick sorting the right partition
The basic algorithm is as follows
Quick Sort1) Partitioning Step Take an element in the
unsorted array and determine its final location in the sorted array This occurs when all values to the left of the element in the array are less than (or equal to) the element and all values to the right of the element are greater than (or equal to) the element We now have 1 element in its proper location and two unsorted subarrays
2) Recursive Step Perform step 1 on each unsorted subarray
Quick Sort Each time step 1 is performed on a
subarray another element is placed in its final location of the sorted array and two unsorted subarrays are created
When a subarray consists of one element that subarray is sorted
Therefore that element is in its final location
Quick Sort There are several partitioning strategies
used in practice (ie several ldquoversionsrdquo of quick sort) but the one we are about to describe is known to work well
For simplicity we will choose the last element to be the pivot element
We could also chose a different pivot element and swap it with the last element in the array
Quick Sort Below is the array we would like to
sort1 4 8 9 0 11 5 10 7 6
Quick Sort The index left starts at the first element
and right starts at the next-to-last element
We want to move all the elements smaller than the pivot to the left part of the array and all the elements larger than the pivot to the right part
1 4 8 9 0 11 5 10 7 6left righ
t
Quick Sort We move left to the right skipping
over elements that are smaller than the pivot
1 4 8 9 0 11 5 10 7 6left righ
t
Quick Sort We then move right to the left skipping
over elements that are greater than the pivot
When left and right have stopped left is on an element greater than (or equal to) the pivot and right is on an element smaller than (or equal to) the pivot
1 4 8 9 0 11 5 10 7 6left right
Quick Sort If left is to the left of right (or if left
= right) those elements are swapped
1 4 8 9 0 11 5 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
Quick Sort The effect is to push a large
element to the right and a small element to the left
We then repeat the process until left and right cross
Quick Sort
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6left righ
t
Quick Sort
1 4 5 0 9 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6righ
tleft
Quick Sort
At this point left and right have crossed so no swap is performed
The final part of the partitioning is to swap the pivot element with left
1 4 5 0 9 11 8 10 7 6righ
tleft
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort Note that all elements to the left of
the pivot are less than (or equal to) the pivot and all elements to the right of the pivot are greater than (or equal to) the pivot
Hence the pivot element has been placed in its final sorted position
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort We now repeat the process using
the sub-arrays to the left and right of the pivot
1 4 5 0 6 11 8 10 7 9
1 4 5 0 11 8 10 7 9
Analysis of quicksortmdashbest case
Suppose each partition operation divides the array almost exactly in half
Then the depth of the recursion in log2n Because thatrsquos how many times we can
halve n However there are many recursions at
each level How can we figure this out We note that
Each partition is linear over its subarray All the partitions at one level cover the array
Partitioning at various levels
Best case II We cut the array size in half each time So the depth of the recursion in log2n At each level of the recursion all the
partitions at that level do work that is linear in n
O(log2n) O(n) = O(n log2n)
What about the worst case
Worst case In the worst case partitioning always
divides the size n array into these three parts A length one part containing the pivot itself A length zero part and A length n-1 part containing everything else
We donrsquot recur on the zero-length part Recurring on the length n-1 part requires
(in the worst case) recurring to depth n-1
Worst case partitioning
Worst case for quicksort In the worst case recursion may be
O(n) levels deep (for an array of size n)
But the partitioning work done at each level is still O(n)
O(n) O(n) = O(n2) So worst case for Quicksort is O(n2) When does this happen
When the array is sorted to begin with
Typical case for quicksort If the array is sorted to begin with
Quicksort is terrible O(n2) It is possible to construct other bad
cases However Quicksort is usually O(n log2n) The constants are so good that Quicksort
is generally the fastest algorithm known Most real-world sorting is done by
Quicksort
Tweaking Quicksort Almost anything you can try to
ldquoimproverdquo Quicksort will actually slow it down
One good tweak is to switch to a different sorting method when the subarrays get small (say 10 or 12) Quicksort has too much overhead for small
array sizes For large arrays it might be a good
idea to check beforehand if the array is already sorted But there is a better tweak than this
Randomized Quick-Sort Select the pivot as a random element of the sequence The expected running time of randomized quick-sort on a
sequence of size n is O(n log n) The time spent at a level of the quick-sort tree is O(n) We show that the expected height of the quick-sort tree is
O(log n) good vs bad pivots
bull The probability of a good pivot is 12 thus we expect k2 good pivots out of k pivotsbull After a good pivot the size of each child sequence is at most 34 the size of the parent sequencebull After h pivots we expect (34)h2 n elementsbull The expected height h of the quick-sort tree is at most 2 log43n
Median of three Obviously it doesnrsquot make sense to sort the
array in order to find the median to use as a pivot
Instead compare just three elements of our (sub)arraymdashthe first the last and the middle Take the median (middle value) of these three as
pivot Itrsquos possible (but not easy) to construct cases
which will make this technique O(n2) Suppose we rearrange (sort) these three
numbers so that the smallest is in the first position the largest in the last position and the other in the middle This lets us simplify and speed up the partition
loop
Final comments Weissrsquos code shows some additional
optimizations on pp 246-247 Weiss chooses to stop both searches on
equality to pivot This design decision is debatable
Quicksort is the fastest known general sorting algorithm on average
For optimum speed the pivot must be chosen carefully
ldquoMedian of threerdquo is a good technique for choosing the pivot
There will be some cases where Quicksort runs in O(n2) time
Quick Sort A couple of notes about quick sort There are more optimal ways to
choose the pivot value (such as the median-of-three method)
Also when the subarrays get small it becomes more efficient to use the insertion sort as opposed to continued use of quick sort
Bubble Sort vs Quick Sort If we calculate the Big-O notation
we find that (in the average case) Bubble Sort O(n2) Quick Sort O(nlog2n)
(quick-sort runs in time O(n2) in the worst case)
Heap Sort Idea take the items that need to be
sorted and insert them into the heap By calling deleteHeap we remove the
smallest or largest element depending on whether or not we are working with a min- or max-heap respectively
Hence the elements are removed in ascending or descending order
Efficiency O(nlog2n)
Merge Sort Idea
Take the array you would like to sort and divide it in half to create 2 unsorted subarrays
Next sort each of the 2 subarrays Finally merge the 2 sorted subarrays
into 1 sorted array Efficiency O(nlog2n)
Merge Sort
Merge Sort Although the merge step produces
a sorted array we have overlooked a very important step
How did we sort the 2 halves before performing the merge step
We used merge sort
Merge Sort By continually calling the merge
sort algorithm we eventually get a subarray of size 1
Since an array with only 1 element is clearly sorted we can back out and merge 2 arrays of size 1
Merge Sort
Merge Sort The basic merging algorithm
consists of 2 input arrays (arrayA and arrayB) An ouput array (arrayC) 3 position holders (indexA indexB
indexC) which are initially set to the beginning of their respective arrays
Merge Sort The smaller of arrayA[indexA] and
arrayB[indexB] is copied into arrayC[indexC] and the appropriate position holders are advanced
When either input list is exhausted the remainder of the other list is copied into arrayC
Merge Sort
1 13 24 26index
A
indexC
arrayA
2 15 27 38index
B
arrayB
arrayC
We compare arrayA[indexA]with arrayB[indexB] Whichever value is smaller isplaced into arrayC[indexC]
1 lt 2 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1index
C
arrayA
2 15 27 38index
B
arrayB
arrayC
2 lt 13 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
13 lt 15 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
15 lt 24 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13 15indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
24 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
indexA
1 2 13 15 24indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
26 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
1 2 13 15 24 26indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
Since we have exhaustedone of the arrays arrayAwe simply copy the remaining items from theother array arrayB intoarrayC
Merge Sort
1 13 24
26
1 2 13 15 24 26 27 38
arrayA
2 15 27 38arrayB
arrayC
Efficiency Summary
Sort Worst Case Average Case
Insertion O(n2) O(n2)
Shell O(n15) O(n125)
Selection O(n2) O(n2)
Bubble O(n2) O(n2)
Quick O(n2) O(nlog2n)
Heap O(nlog2n) O(nlog2n)
Merge O(nlog2n) O(nlog2n)
More Sorting
radix sort bucket sort in-place sorting how fast can we sort
Radix Sort
Unlike other sorting methods radix sort considers the structure of the keys
Assume keys are represented in a base M number system (M = radix) ie if M = 2 the keys are represented in binary
Sorting is done by comparing bits in the same position
Extension to keys that are alphanumeric strings
Radix Exchange Sort
Examine bits from left to right1 Sort array with respect to leftmost bit
2 Partition array
3Recursionrecursively sort top subarray ignoring leftmost bitrecursively sort bottom subarray ignoring leftmost bit
Time to sort n b-bit numbers O(b n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
General Sorting Concepts For example a file of 20000
records may be sorted using an array that holds only 1000 records
Therefore only 1000 records are in primary memory at any given time
The other 19000 records are stored in secondary storage
Sort Order Data may be sorted in either
ascending or descending order The sort order identifies the
sequence of sorted data ascending or descending
If the order of the sort is not specified it is assumed to be ascending
Sort Stability Sort stability is an attribute of a
sort indicating that data elements with equal keys maintain their relative input order in the output
Consider the following example
Sort Stability
Note the unsorted data in (a) If we use a stable sort items with equal keys are
guaranteed to be sorted in the same order as they appear in the unsorted data
Sort Stability
If however we use an unstable sort data with equal keys can be sorted in any order (ie not necessarily in the same order as they appeared in the unsorted data)
Sort Efficiency Sort efficiency is a measure of
the relative efficiency of a sort It is usually an estimate of the
number of comparisons and moves required to order an unordered list
Passes During the sorting process the data is
traversed many times Each traversal of the data is referred to
as a sort pass Depending on the algorithm the sort
pass may traverse the whole list or just a section of the list
The sort pass may also include the placement of one or more elements into the sorted list
Types of Sorts We now discuss several sorting
methods Insertion Sort Shell Sort Selection Sort Bubble Sort Quick Sort Heap Sort Merge Sort
Insertion Sort In each pass of an insertion sort
one or more pieces of data are inserted into their correct location in an ordered list (just as a card player picks up cards and places them in his hand in order)
Insertion Sort In the insertion sort the list is divided into
2 parts Sorted Unsorted
In each pass the first element of the unsorted sublist is transferred to the sorted sublist by inserting it at the appropriate place
If we have a list of n elements it will take at most n-1 passes to sort the data
Insertion Sort
We can visualize this type of sort with the above figure
The first part of the list is the sorted portion which is separated by a ldquoconceptualrdquo wall from the unsorted portion of the list
Insertion SortHere we start with an unsorted list We leave the first data element alone and will start with the 2nd element of the list (the 1st element of the unsorted list)
On our first pass we look at the 2nd element of the list (thefirst element of the unsorted list) and place it in our sorted list in order
On our next pass we look at the 3rd element of the list (the1st element of the unsorted list) and place it in the sorted list in order
We continue in this fashion until the entire list has beensorted
Insertion Sort Example
In our first pass we compare the first 2 values Because theyare out of order we swap them
Now we look at the next 2 values Again they are out oforder so we swap them
Since we have swapped those values we need to comparethe previous 2 values to make sure that they are still in orderSince they are out of order we swap them and then continueon with the next 2 data valuesThese 2 values are out of order so we swap them and look atthe previous 2 values etc
Shell Sort Named after its creator Donald Shell the
shell sort is an improved version of the insertion sort
In the shell sort a list of N elements is divided into K segments where K is known as the increment
What this means is that instead of comparing adjacent values we will compare values that are a distance K apart
We will shrink K as we run through our algorithm
Shell SortPass Notes
1 77 62 14 9 30 21 80 25 70 55 Swap2 21 62 14 9 30 77 80 25 70 55 In order3 21 62 14 9 30 77 80 25 70 55 In order4 21 62 14 9 30 77 80 25 70 55 In order5 21 62 14 9 30 77 80 25 70 55 In order
List (K=5)
Just as in the straight insertion sort we compare 2 values andswap them if they are out of order However in the shell sort we compare values that are a distance K apartOnce we have completed going through the elements in our
list with K=5 we decrease K and continue the process
Shell SortPass Notes
6 21 62 14 9 30 77 80 25 70 55 Swap7 14 62 21 9 30 77 80 25 70 55 Swap8 14 9 21 62 30 77 80 25 70 55 In order9 14 9 21 62 30 77 80 25 70 55 In order10 14 9 21 62 30 77 80 25 70 55 In order11 14 9 21 62 30 77 80 25 70 55 Swap
14 9 21 62 30 25 80 77 70 55 Swap14 9 21 25 30 62 80 77 70 55 In order
12 14 9 21 25 30 62 80 77 70 55 Swap14 9 21 25 30 62 70 77 80 55 In order
13 14 9 21 25 30 62 70 77 80 55 Swap14 9 21 25 30 62 70 55 80 77 Swap14 9 21 25 30 55 70 62 80 77 In order
List (K=2)
Here we have reduced K to 2 Just as in the insertion sort if we swap 2 values we have to go back and compare theprevious 2 values to make sure they are still in order
Shell Sort
All shell sorts will terminate by running an insertion sort(ie K=1) However using the larger values of K first has helped to sort our list so that the straight insertion sort will runfaster
Shell Sort There are many schools of thought
on what the increment should be in the shell sort
Also note that just because an increment is optimal on one list it might not be optimal for another list
Insertion Sort vs Shell Sort Comparing the Big-O notation (for the
average case) we find that Insertion O(n2) Shell O(n125) empirically determined
Although this doesnrsquot seem like much of a gain it makes a big difference as n gets large
Note that in the worst case the Shell sort has an efficiency of O(n2)
However using a special incrementing technique this worst case can be reduced to O(n15)
Insertion Sort vs Shell Sort(Average Case)
Straight Insertion Shell25 625 55100 10000 316500 250000 2364
1000 1000000 56232000 4000000 13374
Number of Loopsn
Selection Sort Imagine some data that you can examine
all at once To sort it you could select the smallest
element and put it in its place select the next smallest and put it in its place etc
For a card player this process is analogous to looking at an entire hand of cards and ordering them by selecting cards one at a time and placing them in their proper order
Selection Sort The selection sort follows this idea Given a list of data to be sorted
we simply select the smallest item and place it in a sorted list
We then repeat these steps until the list is sorted
Selection Sort In the selection sort the list at any moment
is divided into 2 sublists sorted and unsorted separated by a ldquoconceptualrdquo wall
We select the smallest element from the unsorted sublist and exchange it with the element at the beginning of the unsorted data
After each selection and exchange the wall between the 2 sublists moves ndash increasing the number of sorted elements and decreasing the number of unsorted elements
Selection Sort
We start with an unsorted list We search this list for thesmallest element We then exchange the smallest element (8)with the first element in the unsorted list (23)
Again we search the unsorted list for the smallest element We then exchange the smallest element (23) with the first element in the unsorted list (78)
This process continues until the list is fully sorted
Bubble Sort In the bubble sort the list at any
moment is divided into 2 sublists sorted and unsorted
The smallest element is ldquobubbledrdquo from the unsorted sublist to the sorted sublist
Bubble Sort
23 78 45 8 56 3232 56
We start with 32 and compareit with 56 Because 32 is lessthan 56 we swap the twoand step down one element
We then compare 32 and 8Because 32 is not less than 8we do not swap these elements
We step down one element andcompare 45 and 8 They areout of sequence so we swapthem and step down again
8 45
We step down again and compare 8 with 78 These two elements are swapped
8 78
Finally 8 is compared with 23and swapped We thencontinue this process back with56 hellip
8 23
1 Pass of the Bubble Sort
Quick Sort In the bubble sort consecutive items
are compared and possibly exchanged on each pass through the list
This means that many exchanges may be needed to move an element to its correct position
Quick sort is more efficient than bubble sort because a typical exchange involves elements that are far apart so fewer exchanges are required to correctly position an element
Idea of Quick Sort1) Select pick an element
2) Divide rearrange elements so that x goes to its final position E
3) Recurse and Conquer recursively sort
Quick Sort Each iteration of the quick sort
selects an element known as the pivot and divides the list into 3 groups Elements whose keys are less than (or
equal to) the pivotrsquos key The pivot element Elements whose keys are greater than
(or equal to) the pivotrsquos key
Quick Sort The sorting then continues by
quick sorting the left partition followed by quick sorting the right partition
The basic algorithm is as follows
Quick Sort1) Partitioning Step Take an element in the
unsorted array and determine its final location in the sorted array This occurs when all values to the left of the element in the array are less than (or equal to) the element and all values to the right of the element are greater than (or equal to) the element We now have 1 element in its proper location and two unsorted subarrays
2) Recursive Step Perform step 1 on each unsorted subarray
Quick Sort Each time step 1 is performed on a
subarray another element is placed in its final location of the sorted array and two unsorted subarrays are created
When a subarray consists of one element that subarray is sorted
Therefore that element is in its final location
Quick Sort There are several partitioning strategies
used in practice (ie several ldquoversionsrdquo of quick sort) but the one we are about to describe is known to work well
For simplicity we will choose the last element to be the pivot element
We could also chose a different pivot element and swap it with the last element in the array
Quick Sort Below is the array we would like to
sort1 4 8 9 0 11 5 10 7 6
Quick Sort The index left starts at the first element
and right starts at the next-to-last element
We want to move all the elements smaller than the pivot to the left part of the array and all the elements larger than the pivot to the right part
1 4 8 9 0 11 5 10 7 6left righ
t
Quick Sort We move left to the right skipping
over elements that are smaller than the pivot
1 4 8 9 0 11 5 10 7 6left righ
t
Quick Sort We then move right to the left skipping
over elements that are greater than the pivot
When left and right have stopped left is on an element greater than (or equal to) the pivot and right is on an element smaller than (or equal to) the pivot
1 4 8 9 0 11 5 10 7 6left right
Quick Sort If left is to the left of right (or if left
= right) those elements are swapped
1 4 8 9 0 11 5 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
Quick Sort The effect is to push a large
element to the right and a small element to the left
We then repeat the process until left and right cross
Quick Sort
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6left righ
t
Quick Sort
1 4 5 0 9 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6righ
tleft
Quick Sort
At this point left and right have crossed so no swap is performed
The final part of the partitioning is to swap the pivot element with left
1 4 5 0 9 11 8 10 7 6righ
tleft
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort Note that all elements to the left of
the pivot are less than (or equal to) the pivot and all elements to the right of the pivot are greater than (or equal to) the pivot
Hence the pivot element has been placed in its final sorted position
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort We now repeat the process using
the sub-arrays to the left and right of the pivot
1 4 5 0 6 11 8 10 7 9
1 4 5 0 11 8 10 7 9
Analysis of quicksortmdashbest case
Suppose each partition operation divides the array almost exactly in half
Then the depth of the recursion in log2n Because thatrsquos how many times we can
halve n However there are many recursions at
each level How can we figure this out We note that
Each partition is linear over its subarray All the partitions at one level cover the array
Partitioning at various levels
Best case II We cut the array size in half each time So the depth of the recursion in log2n At each level of the recursion all the
partitions at that level do work that is linear in n
O(log2n) O(n) = O(n log2n)
What about the worst case
Worst case In the worst case partitioning always
divides the size n array into these three parts A length one part containing the pivot itself A length zero part and A length n-1 part containing everything else
We donrsquot recur on the zero-length part Recurring on the length n-1 part requires
(in the worst case) recurring to depth n-1
Worst case partitioning
Worst case for quicksort In the worst case recursion may be
O(n) levels deep (for an array of size n)
But the partitioning work done at each level is still O(n)
O(n) O(n) = O(n2) So worst case for Quicksort is O(n2) When does this happen
When the array is sorted to begin with
Typical case for quicksort If the array is sorted to begin with
Quicksort is terrible O(n2) It is possible to construct other bad
cases However Quicksort is usually O(n log2n) The constants are so good that Quicksort
is generally the fastest algorithm known Most real-world sorting is done by
Quicksort
Tweaking Quicksort Almost anything you can try to
ldquoimproverdquo Quicksort will actually slow it down
One good tweak is to switch to a different sorting method when the subarrays get small (say 10 or 12) Quicksort has too much overhead for small
array sizes For large arrays it might be a good
idea to check beforehand if the array is already sorted But there is a better tweak than this
Randomized Quick-Sort Select the pivot as a random element of the sequence The expected running time of randomized quick-sort on a
sequence of size n is O(n log n) The time spent at a level of the quick-sort tree is O(n) We show that the expected height of the quick-sort tree is
O(log n) good vs bad pivots
bull The probability of a good pivot is 12 thus we expect k2 good pivots out of k pivotsbull After a good pivot the size of each child sequence is at most 34 the size of the parent sequencebull After h pivots we expect (34)h2 n elementsbull The expected height h of the quick-sort tree is at most 2 log43n
Median of three Obviously it doesnrsquot make sense to sort the
array in order to find the median to use as a pivot
Instead compare just three elements of our (sub)arraymdashthe first the last and the middle Take the median (middle value) of these three as
pivot Itrsquos possible (but not easy) to construct cases
which will make this technique O(n2) Suppose we rearrange (sort) these three
numbers so that the smallest is in the first position the largest in the last position and the other in the middle This lets us simplify and speed up the partition
loop
Final comments Weissrsquos code shows some additional
optimizations on pp 246-247 Weiss chooses to stop both searches on
equality to pivot This design decision is debatable
Quicksort is the fastest known general sorting algorithm on average
For optimum speed the pivot must be chosen carefully
ldquoMedian of threerdquo is a good technique for choosing the pivot
There will be some cases where Quicksort runs in O(n2) time
Quick Sort A couple of notes about quick sort There are more optimal ways to
choose the pivot value (such as the median-of-three method)
Also when the subarrays get small it becomes more efficient to use the insertion sort as opposed to continued use of quick sort
Bubble Sort vs Quick Sort If we calculate the Big-O notation
we find that (in the average case) Bubble Sort O(n2) Quick Sort O(nlog2n)
(quick-sort runs in time O(n2) in the worst case)
Heap Sort Idea take the items that need to be
sorted and insert them into the heap By calling deleteHeap we remove the
smallest or largest element depending on whether or not we are working with a min- or max-heap respectively
Hence the elements are removed in ascending or descending order
Efficiency O(nlog2n)
Merge Sort Idea
Take the array you would like to sort and divide it in half to create 2 unsorted subarrays
Next sort each of the 2 subarrays Finally merge the 2 sorted subarrays
into 1 sorted array Efficiency O(nlog2n)
Merge Sort
Merge Sort Although the merge step produces
a sorted array we have overlooked a very important step
How did we sort the 2 halves before performing the merge step
We used merge sort
Merge Sort By continually calling the merge
sort algorithm we eventually get a subarray of size 1
Since an array with only 1 element is clearly sorted we can back out and merge 2 arrays of size 1
Merge Sort
Merge Sort The basic merging algorithm
consists of 2 input arrays (arrayA and arrayB) An ouput array (arrayC) 3 position holders (indexA indexB
indexC) which are initially set to the beginning of their respective arrays
Merge Sort The smaller of arrayA[indexA] and
arrayB[indexB] is copied into arrayC[indexC] and the appropriate position holders are advanced
When either input list is exhausted the remainder of the other list is copied into arrayC
Merge Sort
1 13 24 26index
A
indexC
arrayA
2 15 27 38index
B
arrayB
arrayC
We compare arrayA[indexA]with arrayB[indexB] Whichever value is smaller isplaced into arrayC[indexC]
1 lt 2 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1index
C
arrayA
2 15 27 38index
B
arrayB
arrayC
2 lt 13 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
13 lt 15 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
15 lt 24 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13 15indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
24 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
indexA
1 2 13 15 24indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
26 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
1 2 13 15 24 26indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
Since we have exhaustedone of the arrays arrayAwe simply copy the remaining items from theother array arrayB intoarrayC
Merge Sort
1 13 24
26
1 2 13 15 24 26 27 38
arrayA
2 15 27 38arrayB
arrayC
Efficiency Summary
Sort Worst Case Average Case
Insertion O(n2) O(n2)
Shell O(n15) O(n125)
Selection O(n2) O(n2)
Bubble O(n2) O(n2)
Quick O(n2) O(nlog2n)
Heap O(nlog2n) O(nlog2n)
Merge O(nlog2n) O(nlog2n)
More Sorting
radix sort bucket sort in-place sorting how fast can we sort
Radix Sort
Unlike other sorting methods radix sort considers the structure of the keys
Assume keys are represented in a base M number system (M = radix) ie if M = 2 the keys are represented in binary
Sorting is done by comparing bits in the same position
Extension to keys that are alphanumeric strings
Radix Exchange Sort
Examine bits from left to right1 Sort array with respect to leftmost bit
2 Partition array
3Recursionrecursively sort top subarray ignoring leftmost bitrecursively sort bottom subarray ignoring leftmost bit
Time to sort n b-bit numbers O(b n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Sort Order Data may be sorted in either
ascending or descending order The sort order identifies the
sequence of sorted data ascending or descending
If the order of the sort is not specified it is assumed to be ascending
Sort Stability Sort stability is an attribute of a
sort indicating that data elements with equal keys maintain their relative input order in the output
Consider the following example
Sort Stability
Note the unsorted data in (a) If we use a stable sort items with equal keys are
guaranteed to be sorted in the same order as they appear in the unsorted data
Sort Stability
If however we use an unstable sort data with equal keys can be sorted in any order (ie not necessarily in the same order as they appeared in the unsorted data)
Sort Efficiency Sort efficiency is a measure of
the relative efficiency of a sort It is usually an estimate of the
number of comparisons and moves required to order an unordered list
Passes During the sorting process the data is
traversed many times Each traversal of the data is referred to
as a sort pass Depending on the algorithm the sort
pass may traverse the whole list or just a section of the list
The sort pass may also include the placement of one or more elements into the sorted list
Types of Sorts We now discuss several sorting
methods Insertion Sort Shell Sort Selection Sort Bubble Sort Quick Sort Heap Sort Merge Sort
Insertion Sort In each pass of an insertion sort
one or more pieces of data are inserted into their correct location in an ordered list (just as a card player picks up cards and places them in his hand in order)
Insertion Sort In the insertion sort the list is divided into
2 parts Sorted Unsorted
In each pass the first element of the unsorted sublist is transferred to the sorted sublist by inserting it at the appropriate place
If we have a list of n elements it will take at most n-1 passes to sort the data
Insertion Sort
We can visualize this type of sort with the above figure
The first part of the list is the sorted portion which is separated by a ldquoconceptualrdquo wall from the unsorted portion of the list
Insertion SortHere we start with an unsorted list We leave the first data element alone and will start with the 2nd element of the list (the 1st element of the unsorted list)
On our first pass we look at the 2nd element of the list (thefirst element of the unsorted list) and place it in our sorted list in order
On our next pass we look at the 3rd element of the list (the1st element of the unsorted list) and place it in the sorted list in order
We continue in this fashion until the entire list has beensorted
Insertion Sort Example
In our first pass we compare the first 2 values Because theyare out of order we swap them
Now we look at the next 2 values Again they are out oforder so we swap them
Since we have swapped those values we need to comparethe previous 2 values to make sure that they are still in orderSince they are out of order we swap them and then continueon with the next 2 data valuesThese 2 values are out of order so we swap them and look atthe previous 2 values etc
Shell Sort Named after its creator Donald Shell the
shell sort is an improved version of the insertion sort
In the shell sort a list of N elements is divided into K segments where K is known as the increment
What this means is that instead of comparing adjacent values we will compare values that are a distance K apart
We will shrink K as we run through our algorithm
Shell SortPass Notes
1 77 62 14 9 30 21 80 25 70 55 Swap2 21 62 14 9 30 77 80 25 70 55 In order3 21 62 14 9 30 77 80 25 70 55 In order4 21 62 14 9 30 77 80 25 70 55 In order5 21 62 14 9 30 77 80 25 70 55 In order
List (K=5)
Just as in the straight insertion sort we compare 2 values andswap them if they are out of order However in the shell sort we compare values that are a distance K apartOnce we have completed going through the elements in our
list with K=5 we decrease K and continue the process
Shell SortPass Notes
6 21 62 14 9 30 77 80 25 70 55 Swap7 14 62 21 9 30 77 80 25 70 55 Swap8 14 9 21 62 30 77 80 25 70 55 In order9 14 9 21 62 30 77 80 25 70 55 In order10 14 9 21 62 30 77 80 25 70 55 In order11 14 9 21 62 30 77 80 25 70 55 Swap
14 9 21 62 30 25 80 77 70 55 Swap14 9 21 25 30 62 80 77 70 55 In order
12 14 9 21 25 30 62 80 77 70 55 Swap14 9 21 25 30 62 70 77 80 55 In order
13 14 9 21 25 30 62 70 77 80 55 Swap14 9 21 25 30 62 70 55 80 77 Swap14 9 21 25 30 55 70 62 80 77 In order
List (K=2)
Here we have reduced K to 2 Just as in the insertion sort if we swap 2 values we have to go back and compare theprevious 2 values to make sure they are still in order
Shell Sort
All shell sorts will terminate by running an insertion sort(ie K=1) However using the larger values of K first has helped to sort our list so that the straight insertion sort will runfaster
Shell Sort There are many schools of thought
on what the increment should be in the shell sort
Also note that just because an increment is optimal on one list it might not be optimal for another list
Insertion Sort vs Shell Sort Comparing the Big-O notation (for the
average case) we find that Insertion O(n2) Shell O(n125) empirically determined
Although this doesnrsquot seem like much of a gain it makes a big difference as n gets large
Note that in the worst case the Shell sort has an efficiency of O(n2)
However using a special incrementing technique this worst case can be reduced to O(n15)
Insertion Sort vs Shell Sort(Average Case)
Straight Insertion Shell25 625 55100 10000 316500 250000 2364
1000 1000000 56232000 4000000 13374
Number of Loopsn
Selection Sort Imagine some data that you can examine
all at once To sort it you could select the smallest
element and put it in its place select the next smallest and put it in its place etc
For a card player this process is analogous to looking at an entire hand of cards and ordering them by selecting cards one at a time and placing them in their proper order
Selection Sort The selection sort follows this idea Given a list of data to be sorted
we simply select the smallest item and place it in a sorted list
We then repeat these steps until the list is sorted
Selection Sort In the selection sort the list at any moment
is divided into 2 sublists sorted and unsorted separated by a ldquoconceptualrdquo wall
We select the smallest element from the unsorted sublist and exchange it with the element at the beginning of the unsorted data
After each selection and exchange the wall between the 2 sublists moves ndash increasing the number of sorted elements and decreasing the number of unsorted elements
Selection Sort
We start with an unsorted list We search this list for thesmallest element We then exchange the smallest element (8)with the first element in the unsorted list (23)
Again we search the unsorted list for the smallest element We then exchange the smallest element (23) with the first element in the unsorted list (78)
This process continues until the list is fully sorted
Bubble Sort In the bubble sort the list at any
moment is divided into 2 sublists sorted and unsorted
The smallest element is ldquobubbledrdquo from the unsorted sublist to the sorted sublist
Bubble Sort
23 78 45 8 56 3232 56
We start with 32 and compareit with 56 Because 32 is lessthan 56 we swap the twoand step down one element
We then compare 32 and 8Because 32 is not less than 8we do not swap these elements
We step down one element andcompare 45 and 8 They areout of sequence so we swapthem and step down again
8 45
We step down again and compare 8 with 78 These two elements are swapped
8 78
Finally 8 is compared with 23and swapped We thencontinue this process back with56 hellip
8 23
1 Pass of the Bubble Sort
Quick Sort In the bubble sort consecutive items
are compared and possibly exchanged on each pass through the list
This means that many exchanges may be needed to move an element to its correct position
Quick sort is more efficient than bubble sort because a typical exchange involves elements that are far apart so fewer exchanges are required to correctly position an element
Idea of Quick Sort1) Select pick an element
2) Divide rearrange elements so that x goes to its final position E
3) Recurse and Conquer recursively sort
Quick Sort Each iteration of the quick sort
selects an element known as the pivot and divides the list into 3 groups Elements whose keys are less than (or
equal to) the pivotrsquos key The pivot element Elements whose keys are greater than
(or equal to) the pivotrsquos key
Quick Sort The sorting then continues by
quick sorting the left partition followed by quick sorting the right partition
The basic algorithm is as follows
Quick Sort1) Partitioning Step Take an element in the
unsorted array and determine its final location in the sorted array This occurs when all values to the left of the element in the array are less than (or equal to) the element and all values to the right of the element are greater than (or equal to) the element We now have 1 element in its proper location and two unsorted subarrays
2) Recursive Step Perform step 1 on each unsorted subarray
Quick Sort Each time step 1 is performed on a
subarray another element is placed in its final location of the sorted array and two unsorted subarrays are created
When a subarray consists of one element that subarray is sorted
Therefore that element is in its final location
Quick Sort There are several partitioning strategies
used in practice (ie several ldquoversionsrdquo of quick sort) but the one we are about to describe is known to work well
For simplicity we will choose the last element to be the pivot element
We could also chose a different pivot element and swap it with the last element in the array
Quick Sort Below is the array we would like to
sort1 4 8 9 0 11 5 10 7 6
Quick Sort The index left starts at the first element
and right starts at the next-to-last element
We want to move all the elements smaller than the pivot to the left part of the array and all the elements larger than the pivot to the right part
1 4 8 9 0 11 5 10 7 6left righ
t
Quick Sort We move left to the right skipping
over elements that are smaller than the pivot
1 4 8 9 0 11 5 10 7 6left righ
t
Quick Sort We then move right to the left skipping
over elements that are greater than the pivot
When left and right have stopped left is on an element greater than (or equal to) the pivot and right is on an element smaller than (or equal to) the pivot
1 4 8 9 0 11 5 10 7 6left right
Quick Sort If left is to the left of right (or if left
= right) those elements are swapped
1 4 8 9 0 11 5 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
Quick Sort The effect is to push a large
element to the right and a small element to the left
We then repeat the process until left and right cross
Quick Sort
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6left righ
t
Quick Sort
1 4 5 0 9 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6righ
tleft
Quick Sort
At this point left and right have crossed so no swap is performed
The final part of the partitioning is to swap the pivot element with left
1 4 5 0 9 11 8 10 7 6righ
tleft
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort Note that all elements to the left of
the pivot are less than (or equal to) the pivot and all elements to the right of the pivot are greater than (or equal to) the pivot
Hence the pivot element has been placed in its final sorted position
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort We now repeat the process using
the sub-arrays to the left and right of the pivot
1 4 5 0 6 11 8 10 7 9
1 4 5 0 11 8 10 7 9
Analysis of quicksortmdashbest case
Suppose each partition operation divides the array almost exactly in half
Then the depth of the recursion in log2n Because thatrsquos how many times we can
halve n However there are many recursions at
each level How can we figure this out We note that
Each partition is linear over its subarray All the partitions at one level cover the array
Partitioning at various levels
Best case II We cut the array size in half each time So the depth of the recursion in log2n At each level of the recursion all the
partitions at that level do work that is linear in n
O(log2n) O(n) = O(n log2n)
What about the worst case
Worst case In the worst case partitioning always
divides the size n array into these three parts A length one part containing the pivot itself A length zero part and A length n-1 part containing everything else
We donrsquot recur on the zero-length part Recurring on the length n-1 part requires
(in the worst case) recurring to depth n-1
Worst case partitioning
Worst case for quicksort In the worst case recursion may be
O(n) levels deep (for an array of size n)
But the partitioning work done at each level is still O(n)
O(n) O(n) = O(n2) So worst case for Quicksort is O(n2) When does this happen
When the array is sorted to begin with
Typical case for quicksort If the array is sorted to begin with
Quicksort is terrible O(n2) It is possible to construct other bad
cases However Quicksort is usually O(n log2n) The constants are so good that Quicksort
is generally the fastest algorithm known Most real-world sorting is done by
Quicksort
Tweaking Quicksort Almost anything you can try to
ldquoimproverdquo Quicksort will actually slow it down
One good tweak is to switch to a different sorting method when the subarrays get small (say 10 or 12) Quicksort has too much overhead for small
array sizes For large arrays it might be a good
idea to check beforehand if the array is already sorted But there is a better tweak than this
Randomized Quick-Sort Select the pivot as a random element of the sequence The expected running time of randomized quick-sort on a
sequence of size n is O(n log n) The time spent at a level of the quick-sort tree is O(n) We show that the expected height of the quick-sort tree is
O(log n) good vs bad pivots
bull The probability of a good pivot is 12 thus we expect k2 good pivots out of k pivotsbull After a good pivot the size of each child sequence is at most 34 the size of the parent sequencebull After h pivots we expect (34)h2 n elementsbull The expected height h of the quick-sort tree is at most 2 log43n
Median of three Obviously it doesnrsquot make sense to sort the
array in order to find the median to use as a pivot
Instead compare just three elements of our (sub)arraymdashthe first the last and the middle Take the median (middle value) of these three as
pivot Itrsquos possible (but not easy) to construct cases
which will make this technique O(n2) Suppose we rearrange (sort) these three
numbers so that the smallest is in the first position the largest in the last position and the other in the middle This lets us simplify and speed up the partition
loop
Final comments Weissrsquos code shows some additional
optimizations on pp 246-247 Weiss chooses to stop both searches on
equality to pivot This design decision is debatable
Quicksort is the fastest known general sorting algorithm on average
For optimum speed the pivot must be chosen carefully
ldquoMedian of threerdquo is a good technique for choosing the pivot
There will be some cases where Quicksort runs in O(n2) time
Quick Sort A couple of notes about quick sort There are more optimal ways to
choose the pivot value (such as the median-of-three method)
Also when the subarrays get small it becomes more efficient to use the insertion sort as opposed to continued use of quick sort
Bubble Sort vs Quick Sort If we calculate the Big-O notation
we find that (in the average case) Bubble Sort O(n2) Quick Sort O(nlog2n)
(quick-sort runs in time O(n2) in the worst case)
Heap Sort Idea take the items that need to be
sorted and insert them into the heap By calling deleteHeap we remove the
smallest or largest element depending on whether or not we are working with a min- or max-heap respectively
Hence the elements are removed in ascending or descending order
Efficiency O(nlog2n)
Merge Sort Idea
Take the array you would like to sort and divide it in half to create 2 unsorted subarrays
Next sort each of the 2 subarrays Finally merge the 2 sorted subarrays
into 1 sorted array Efficiency O(nlog2n)
Merge Sort
Merge Sort Although the merge step produces
a sorted array we have overlooked a very important step
How did we sort the 2 halves before performing the merge step
We used merge sort
Merge Sort By continually calling the merge
sort algorithm we eventually get a subarray of size 1
Since an array with only 1 element is clearly sorted we can back out and merge 2 arrays of size 1
Merge Sort
Merge Sort The basic merging algorithm
consists of 2 input arrays (arrayA and arrayB) An ouput array (arrayC) 3 position holders (indexA indexB
indexC) which are initially set to the beginning of their respective arrays
Merge Sort The smaller of arrayA[indexA] and
arrayB[indexB] is copied into arrayC[indexC] and the appropriate position holders are advanced
When either input list is exhausted the remainder of the other list is copied into arrayC
Merge Sort
1 13 24 26index
A
indexC
arrayA
2 15 27 38index
B
arrayB
arrayC
We compare arrayA[indexA]with arrayB[indexB] Whichever value is smaller isplaced into arrayC[indexC]
1 lt 2 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1index
C
arrayA
2 15 27 38index
B
arrayB
arrayC
2 lt 13 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
13 lt 15 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
15 lt 24 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13 15indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
24 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
indexA
1 2 13 15 24indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
26 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
1 2 13 15 24 26indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
Since we have exhaustedone of the arrays arrayAwe simply copy the remaining items from theother array arrayB intoarrayC
Merge Sort
1 13 24
26
1 2 13 15 24 26 27 38
arrayA
2 15 27 38arrayB
arrayC
Efficiency Summary
Sort Worst Case Average Case
Insertion O(n2) O(n2)
Shell O(n15) O(n125)
Selection O(n2) O(n2)
Bubble O(n2) O(n2)
Quick O(n2) O(nlog2n)
Heap O(nlog2n) O(nlog2n)
Merge O(nlog2n) O(nlog2n)
More Sorting
radix sort bucket sort in-place sorting how fast can we sort
Radix Sort
Unlike other sorting methods radix sort considers the structure of the keys
Assume keys are represented in a base M number system (M = radix) ie if M = 2 the keys are represented in binary
Sorting is done by comparing bits in the same position
Extension to keys that are alphanumeric strings
Radix Exchange Sort
Examine bits from left to right1 Sort array with respect to leftmost bit
2 Partition array
3Recursionrecursively sort top subarray ignoring leftmost bitrecursively sort bottom subarray ignoring leftmost bit
Time to sort n b-bit numbers O(b n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Sort Stability Sort stability is an attribute of a
sort indicating that data elements with equal keys maintain their relative input order in the output
Consider the following example
Sort Stability
Note the unsorted data in (a) If we use a stable sort items with equal keys are
guaranteed to be sorted in the same order as they appear in the unsorted data
Sort Stability
If however we use an unstable sort data with equal keys can be sorted in any order (ie not necessarily in the same order as they appeared in the unsorted data)
Sort Efficiency Sort efficiency is a measure of
the relative efficiency of a sort It is usually an estimate of the
number of comparisons and moves required to order an unordered list
Passes During the sorting process the data is
traversed many times Each traversal of the data is referred to
as a sort pass Depending on the algorithm the sort
pass may traverse the whole list or just a section of the list
The sort pass may also include the placement of one or more elements into the sorted list
Types of Sorts We now discuss several sorting
methods Insertion Sort Shell Sort Selection Sort Bubble Sort Quick Sort Heap Sort Merge Sort
Insertion Sort In each pass of an insertion sort
one or more pieces of data are inserted into their correct location in an ordered list (just as a card player picks up cards and places them in his hand in order)
Insertion Sort In the insertion sort the list is divided into
2 parts Sorted Unsorted
In each pass the first element of the unsorted sublist is transferred to the sorted sublist by inserting it at the appropriate place
If we have a list of n elements it will take at most n-1 passes to sort the data
Insertion Sort
We can visualize this type of sort with the above figure
The first part of the list is the sorted portion which is separated by a ldquoconceptualrdquo wall from the unsorted portion of the list
Insertion SortHere we start with an unsorted list We leave the first data element alone and will start with the 2nd element of the list (the 1st element of the unsorted list)
On our first pass we look at the 2nd element of the list (thefirst element of the unsorted list) and place it in our sorted list in order
On our next pass we look at the 3rd element of the list (the1st element of the unsorted list) and place it in the sorted list in order
We continue in this fashion until the entire list has beensorted
Insertion Sort Example
In our first pass we compare the first 2 values Because theyare out of order we swap them
Now we look at the next 2 values Again they are out oforder so we swap them
Since we have swapped those values we need to comparethe previous 2 values to make sure that they are still in orderSince they are out of order we swap them and then continueon with the next 2 data valuesThese 2 values are out of order so we swap them and look atthe previous 2 values etc
Shell Sort Named after its creator Donald Shell the
shell sort is an improved version of the insertion sort
In the shell sort a list of N elements is divided into K segments where K is known as the increment
What this means is that instead of comparing adjacent values we will compare values that are a distance K apart
We will shrink K as we run through our algorithm
Shell SortPass Notes
1 77 62 14 9 30 21 80 25 70 55 Swap2 21 62 14 9 30 77 80 25 70 55 In order3 21 62 14 9 30 77 80 25 70 55 In order4 21 62 14 9 30 77 80 25 70 55 In order5 21 62 14 9 30 77 80 25 70 55 In order
List (K=5)
Just as in the straight insertion sort we compare 2 values andswap them if they are out of order However in the shell sort we compare values that are a distance K apartOnce we have completed going through the elements in our
list with K=5 we decrease K and continue the process
Shell SortPass Notes
6 21 62 14 9 30 77 80 25 70 55 Swap7 14 62 21 9 30 77 80 25 70 55 Swap8 14 9 21 62 30 77 80 25 70 55 In order9 14 9 21 62 30 77 80 25 70 55 In order10 14 9 21 62 30 77 80 25 70 55 In order11 14 9 21 62 30 77 80 25 70 55 Swap
14 9 21 62 30 25 80 77 70 55 Swap14 9 21 25 30 62 80 77 70 55 In order
12 14 9 21 25 30 62 80 77 70 55 Swap14 9 21 25 30 62 70 77 80 55 In order
13 14 9 21 25 30 62 70 77 80 55 Swap14 9 21 25 30 62 70 55 80 77 Swap14 9 21 25 30 55 70 62 80 77 In order
List (K=2)
Here we have reduced K to 2 Just as in the insertion sort if we swap 2 values we have to go back and compare theprevious 2 values to make sure they are still in order
Shell Sort
All shell sorts will terminate by running an insertion sort(ie K=1) However using the larger values of K first has helped to sort our list so that the straight insertion sort will runfaster
Shell Sort There are many schools of thought
on what the increment should be in the shell sort
Also note that just because an increment is optimal on one list it might not be optimal for another list
Insertion Sort vs Shell Sort Comparing the Big-O notation (for the
average case) we find that Insertion O(n2) Shell O(n125) empirically determined
Although this doesnrsquot seem like much of a gain it makes a big difference as n gets large
Note that in the worst case the Shell sort has an efficiency of O(n2)
However using a special incrementing technique this worst case can be reduced to O(n15)
Insertion Sort vs Shell Sort(Average Case)
Straight Insertion Shell25 625 55100 10000 316500 250000 2364
1000 1000000 56232000 4000000 13374
Number of Loopsn
Selection Sort Imagine some data that you can examine
all at once To sort it you could select the smallest
element and put it in its place select the next smallest and put it in its place etc
For a card player this process is analogous to looking at an entire hand of cards and ordering them by selecting cards one at a time and placing them in their proper order
Selection Sort The selection sort follows this idea Given a list of data to be sorted
we simply select the smallest item and place it in a sorted list
We then repeat these steps until the list is sorted
Selection Sort In the selection sort the list at any moment
is divided into 2 sublists sorted and unsorted separated by a ldquoconceptualrdquo wall
We select the smallest element from the unsorted sublist and exchange it with the element at the beginning of the unsorted data
After each selection and exchange the wall between the 2 sublists moves ndash increasing the number of sorted elements and decreasing the number of unsorted elements
Selection Sort
We start with an unsorted list We search this list for thesmallest element We then exchange the smallest element (8)with the first element in the unsorted list (23)
Again we search the unsorted list for the smallest element We then exchange the smallest element (23) with the first element in the unsorted list (78)
This process continues until the list is fully sorted
Bubble Sort In the bubble sort the list at any
moment is divided into 2 sublists sorted and unsorted
The smallest element is ldquobubbledrdquo from the unsorted sublist to the sorted sublist
Bubble Sort
23 78 45 8 56 3232 56
We start with 32 and compareit with 56 Because 32 is lessthan 56 we swap the twoand step down one element
We then compare 32 and 8Because 32 is not less than 8we do not swap these elements
We step down one element andcompare 45 and 8 They areout of sequence so we swapthem and step down again
8 45
We step down again and compare 8 with 78 These two elements are swapped
8 78
Finally 8 is compared with 23and swapped We thencontinue this process back with56 hellip
8 23
1 Pass of the Bubble Sort
Quick Sort In the bubble sort consecutive items
are compared and possibly exchanged on each pass through the list
This means that many exchanges may be needed to move an element to its correct position
Quick sort is more efficient than bubble sort because a typical exchange involves elements that are far apart so fewer exchanges are required to correctly position an element
Idea of Quick Sort1) Select pick an element
2) Divide rearrange elements so that x goes to its final position E
3) Recurse and Conquer recursively sort
Quick Sort Each iteration of the quick sort
selects an element known as the pivot and divides the list into 3 groups Elements whose keys are less than (or
equal to) the pivotrsquos key The pivot element Elements whose keys are greater than
(or equal to) the pivotrsquos key
Quick Sort The sorting then continues by
quick sorting the left partition followed by quick sorting the right partition
The basic algorithm is as follows
Quick Sort1) Partitioning Step Take an element in the
unsorted array and determine its final location in the sorted array This occurs when all values to the left of the element in the array are less than (or equal to) the element and all values to the right of the element are greater than (or equal to) the element We now have 1 element in its proper location and two unsorted subarrays
2) Recursive Step Perform step 1 on each unsorted subarray
Quick Sort Each time step 1 is performed on a
subarray another element is placed in its final location of the sorted array and two unsorted subarrays are created
When a subarray consists of one element that subarray is sorted
Therefore that element is in its final location
Quick Sort There are several partitioning strategies
used in practice (ie several ldquoversionsrdquo of quick sort) but the one we are about to describe is known to work well
For simplicity we will choose the last element to be the pivot element
We could also chose a different pivot element and swap it with the last element in the array
Quick Sort Below is the array we would like to
sort1 4 8 9 0 11 5 10 7 6
Quick Sort The index left starts at the first element
and right starts at the next-to-last element
We want to move all the elements smaller than the pivot to the left part of the array and all the elements larger than the pivot to the right part
1 4 8 9 0 11 5 10 7 6left righ
t
Quick Sort We move left to the right skipping
over elements that are smaller than the pivot
1 4 8 9 0 11 5 10 7 6left righ
t
Quick Sort We then move right to the left skipping
over elements that are greater than the pivot
When left and right have stopped left is on an element greater than (or equal to) the pivot and right is on an element smaller than (or equal to) the pivot
1 4 8 9 0 11 5 10 7 6left right
Quick Sort If left is to the left of right (or if left
= right) those elements are swapped
1 4 8 9 0 11 5 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
Quick Sort The effect is to push a large
element to the right and a small element to the left
We then repeat the process until left and right cross
Quick Sort
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6left righ
t
Quick Sort
1 4 5 0 9 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6righ
tleft
Quick Sort
At this point left and right have crossed so no swap is performed
The final part of the partitioning is to swap the pivot element with left
1 4 5 0 9 11 8 10 7 6righ
tleft
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort Note that all elements to the left of
the pivot are less than (or equal to) the pivot and all elements to the right of the pivot are greater than (or equal to) the pivot
Hence the pivot element has been placed in its final sorted position
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort We now repeat the process using
the sub-arrays to the left and right of the pivot
1 4 5 0 6 11 8 10 7 9
1 4 5 0 11 8 10 7 9
Analysis of quicksortmdashbest case
Suppose each partition operation divides the array almost exactly in half
Then the depth of the recursion in log2n Because thatrsquos how many times we can
halve n However there are many recursions at
each level How can we figure this out We note that
Each partition is linear over its subarray All the partitions at one level cover the array
Partitioning at various levels
Best case II We cut the array size in half each time So the depth of the recursion in log2n At each level of the recursion all the
partitions at that level do work that is linear in n
O(log2n) O(n) = O(n log2n)
What about the worst case
Worst case In the worst case partitioning always
divides the size n array into these three parts A length one part containing the pivot itself A length zero part and A length n-1 part containing everything else
We donrsquot recur on the zero-length part Recurring on the length n-1 part requires
(in the worst case) recurring to depth n-1
Worst case partitioning
Worst case for quicksort In the worst case recursion may be
O(n) levels deep (for an array of size n)
But the partitioning work done at each level is still O(n)
O(n) O(n) = O(n2) So worst case for Quicksort is O(n2) When does this happen
When the array is sorted to begin with
Typical case for quicksort If the array is sorted to begin with
Quicksort is terrible O(n2) It is possible to construct other bad
cases However Quicksort is usually O(n log2n) The constants are so good that Quicksort
is generally the fastest algorithm known Most real-world sorting is done by
Quicksort
Tweaking Quicksort Almost anything you can try to
ldquoimproverdquo Quicksort will actually slow it down
One good tweak is to switch to a different sorting method when the subarrays get small (say 10 or 12) Quicksort has too much overhead for small
array sizes For large arrays it might be a good
idea to check beforehand if the array is already sorted But there is a better tweak than this
Randomized Quick-Sort Select the pivot as a random element of the sequence The expected running time of randomized quick-sort on a
sequence of size n is O(n log n) The time spent at a level of the quick-sort tree is O(n) We show that the expected height of the quick-sort tree is
O(log n) good vs bad pivots
bull The probability of a good pivot is 12 thus we expect k2 good pivots out of k pivotsbull After a good pivot the size of each child sequence is at most 34 the size of the parent sequencebull After h pivots we expect (34)h2 n elementsbull The expected height h of the quick-sort tree is at most 2 log43n
Median of three Obviously it doesnrsquot make sense to sort the
array in order to find the median to use as a pivot
Instead compare just three elements of our (sub)arraymdashthe first the last and the middle Take the median (middle value) of these three as
pivot Itrsquos possible (but not easy) to construct cases
which will make this technique O(n2) Suppose we rearrange (sort) these three
numbers so that the smallest is in the first position the largest in the last position and the other in the middle This lets us simplify and speed up the partition
loop
Final comments Weissrsquos code shows some additional
optimizations on pp 246-247 Weiss chooses to stop both searches on
equality to pivot This design decision is debatable
Quicksort is the fastest known general sorting algorithm on average
For optimum speed the pivot must be chosen carefully
ldquoMedian of threerdquo is a good technique for choosing the pivot
There will be some cases where Quicksort runs in O(n2) time
Quick Sort A couple of notes about quick sort There are more optimal ways to
choose the pivot value (such as the median-of-three method)
Also when the subarrays get small it becomes more efficient to use the insertion sort as opposed to continued use of quick sort
Bubble Sort vs Quick Sort If we calculate the Big-O notation
we find that (in the average case) Bubble Sort O(n2) Quick Sort O(nlog2n)
(quick-sort runs in time O(n2) in the worst case)
Heap Sort Idea take the items that need to be
sorted and insert them into the heap By calling deleteHeap we remove the
smallest or largest element depending on whether or not we are working with a min- or max-heap respectively
Hence the elements are removed in ascending or descending order
Efficiency O(nlog2n)
Merge Sort Idea
Take the array you would like to sort and divide it in half to create 2 unsorted subarrays
Next sort each of the 2 subarrays Finally merge the 2 sorted subarrays
into 1 sorted array Efficiency O(nlog2n)
Merge Sort
Merge Sort Although the merge step produces
a sorted array we have overlooked a very important step
How did we sort the 2 halves before performing the merge step
We used merge sort
Merge Sort By continually calling the merge
sort algorithm we eventually get a subarray of size 1
Since an array with only 1 element is clearly sorted we can back out and merge 2 arrays of size 1
Merge Sort
Merge Sort The basic merging algorithm
consists of 2 input arrays (arrayA and arrayB) An ouput array (arrayC) 3 position holders (indexA indexB
indexC) which are initially set to the beginning of their respective arrays
Merge Sort The smaller of arrayA[indexA] and
arrayB[indexB] is copied into arrayC[indexC] and the appropriate position holders are advanced
When either input list is exhausted the remainder of the other list is copied into arrayC
Merge Sort
1 13 24 26index
A
indexC
arrayA
2 15 27 38index
B
arrayB
arrayC
We compare arrayA[indexA]with arrayB[indexB] Whichever value is smaller isplaced into arrayC[indexC]
1 lt 2 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1index
C
arrayA
2 15 27 38index
B
arrayB
arrayC
2 lt 13 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
13 lt 15 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
15 lt 24 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13 15indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
24 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
indexA
1 2 13 15 24indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
26 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
1 2 13 15 24 26indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
Since we have exhaustedone of the arrays arrayAwe simply copy the remaining items from theother array arrayB intoarrayC
Merge Sort
1 13 24
26
1 2 13 15 24 26 27 38
arrayA
2 15 27 38arrayB
arrayC
Efficiency Summary
Sort Worst Case Average Case
Insertion O(n2) O(n2)
Shell O(n15) O(n125)
Selection O(n2) O(n2)
Bubble O(n2) O(n2)
Quick O(n2) O(nlog2n)
Heap O(nlog2n) O(nlog2n)
Merge O(nlog2n) O(nlog2n)
More Sorting
radix sort bucket sort in-place sorting how fast can we sort
Radix Sort
Unlike other sorting methods radix sort considers the structure of the keys
Assume keys are represented in a base M number system (M = radix) ie if M = 2 the keys are represented in binary
Sorting is done by comparing bits in the same position
Extension to keys that are alphanumeric strings
Radix Exchange Sort
Examine bits from left to right1 Sort array with respect to leftmost bit
2 Partition array
3Recursionrecursively sort top subarray ignoring leftmost bitrecursively sort bottom subarray ignoring leftmost bit
Time to sort n b-bit numbers O(b n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Sort Stability
Note the unsorted data in (a) If we use a stable sort items with equal keys are
guaranteed to be sorted in the same order as they appear in the unsorted data
Sort Stability
If however we use an unstable sort data with equal keys can be sorted in any order (ie not necessarily in the same order as they appeared in the unsorted data)
Sort Efficiency Sort efficiency is a measure of
the relative efficiency of a sort It is usually an estimate of the
number of comparisons and moves required to order an unordered list
Passes During the sorting process the data is
traversed many times Each traversal of the data is referred to
as a sort pass Depending on the algorithm the sort
pass may traverse the whole list or just a section of the list
The sort pass may also include the placement of one or more elements into the sorted list
Types of Sorts We now discuss several sorting
methods Insertion Sort Shell Sort Selection Sort Bubble Sort Quick Sort Heap Sort Merge Sort
Insertion Sort In each pass of an insertion sort
one or more pieces of data are inserted into their correct location in an ordered list (just as a card player picks up cards and places them in his hand in order)
Insertion Sort In the insertion sort the list is divided into
2 parts Sorted Unsorted
In each pass the first element of the unsorted sublist is transferred to the sorted sublist by inserting it at the appropriate place
If we have a list of n elements it will take at most n-1 passes to sort the data
Insertion Sort
We can visualize this type of sort with the above figure
The first part of the list is the sorted portion which is separated by a ldquoconceptualrdquo wall from the unsorted portion of the list
Insertion SortHere we start with an unsorted list We leave the first data element alone and will start with the 2nd element of the list (the 1st element of the unsorted list)
On our first pass we look at the 2nd element of the list (thefirst element of the unsorted list) and place it in our sorted list in order
On our next pass we look at the 3rd element of the list (the1st element of the unsorted list) and place it in the sorted list in order
We continue in this fashion until the entire list has beensorted
Insertion Sort Example
In our first pass we compare the first 2 values Because theyare out of order we swap them
Now we look at the next 2 values Again they are out oforder so we swap them
Since we have swapped those values we need to comparethe previous 2 values to make sure that they are still in orderSince they are out of order we swap them and then continueon with the next 2 data valuesThese 2 values are out of order so we swap them and look atthe previous 2 values etc
Shell Sort Named after its creator Donald Shell the
shell sort is an improved version of the insertion sort
In the shell sort a list of N elements is divided into K segments where K is known as the increment
What this means is that instead of comparing adjacent values we will compare values that are a distance K apart
We will shrink K as we run through our algorithm
Shell SortPass Notes
1 77 62 14 9 30 21 80 25 70 55 Swap2 21 62 14 9 30 77 80 25 70 55 In order3 21 62 14 9 30 77 80 25 70 55 In order4 21 62 14 9 30 77 80 25 70 55 In order5 21 62 14 9 30 77 80 25 70 55 In order
List (K=5)
Just as in the straight insertion sort we compare 2 values andswap them if they are out of order However in the shell sort we compare values that are a distance K apartOnce we have completed going through the elements in our
list with K=5 we decrease K and continue the process
Shell SortPass Notes
6 21 62 14 9 30 77 80 25 70 55 Swap7 14 62 21 9 30 77 80 25 70 55 Swap8 14 9 21 62 30 77 80 25 70 55 In order9 14 9 21 62 30 77 80 25 70 55 In order10 14 9 21 62 30 77 80 25 70 55 In order11 14 9 21 62 30 77 80 25 70 55 Swap
14 9 21 62 30 25 80 77 70 55 Swap14 9 21 25 30 62 80 77 70 55 In order
12 14 9 21 25 30 62 80 77 70 55 Swap14 9 21 25 30 62 70 77 80 55 In order
13 14 9 21 25 30 62 70 77 80 55 Swap14 9 21 25 30 62 70 55 80 77 Swap14 9 21 25 30 55 70 62 80 77 In order
List (K=2)
Here we have reduced K to 2 Just as in the insertion sort if we swap 2 values we have to go back and compare theprevious 2 values to make sure they are still in order
Shell Sort
All shell sorts will terminate by running an insertion sort(ie K=1) However using the larger values of K first has helped to sort our list so that the straight insertion sort will runfaster
Shell Sort There are many schools of thought
on what the increment should be in the shell sort
Also note that just because an increment is optimal on one list it might not be optimal for another list
Insertion Sort vs Shell Sort Comparing the Big-O notation (for the
average case) we find that Insertion O(n2) Shell O(n125) empirically determined
Although this doesnrsquot seem like much of a gain it makes a big difference as n gets large
Note that in the worst case the Shell sort has an efficiency of O(n2)
However using a special incrementing technique this worst case can be reduced to O(n15)
Insertion Sort vs Shell Sort(Average Case)
Straight Insertion Shell25 625 55100 10000 316500 250000 2364
1000 1000000 56232000 4000000 13374
Number of Loopsn
Selection Sort Imagine some data that you can examine
all at once To sort it you could select the smallest
element and put it in its place select the next smallest and put it in its place etc
For a card player this process is analogous to looking at an entire hand of cards and ordering them by selecting cards one at a time and placing them in their proper order
Selection Sort The selection sort follows this idea Given a list of data to be sorted
we simply select the smallest item and place it in a sorted list
We then repeat these steps until the list is sorted
Selection Sort In the selection sort the list at any moment
is divided into 2 sublists sorted and unsorted separated by a ldquoconceptualrdquo wall
We select the smallest element from the unsorted sublist and exchange it with the element at the beginning of the unsorted data
After each selection and exchange the wall between the 2 sublists moves ndash increasing the number of sorted elements and decreasing the number of unsorted elements
Selection Sort
We start with an unsorted list We search this list for thesmallest element We then exchange the smallest element (8)with the first element in the unsorted list (23)
Again we search the unsorted list for the smallest element We then exchange the smallest element (23) with the first element in the unsorted list (78)
This process continues until the list is fully sorted
Bubble Sort In the bubble sort the list at any
moment is divided into 2 sublists sorted and unsorted
The smallest element is ldquobubbledrdquo from the unsorted sublist to the sorted sublist
Bubble Sort
23 78 45 8 56 3232 56
We start with 32 and compareit with 56 Because 32 is lessthan 56 we swap the twoand step down one element
We then compare 32 and 8Because 32 is not less than 8we do not swap these elements
We step down one element andcompare 45 and 8 They areout of sequence so we swapthem and step down again
8 45
We step down again and compare 8 with 78 These two elements are swapped
8 78
Finally 8 is compared with 23and swapped We thencontinue this process back with56 hellip
8 23
1 Pass of the Bubble Sort
Quick Sort In the bubble sort consecutive items
are compared and possibly exchanged on each pass through the list
This means that many exchanges may be needed to move an element to its correct position
Quick sort is more efficient than bubble sort because a typical exchange involves elements that are far apart so fewer exchanges are required to correctly position an element
Idea of Quick Sort1) Select pick an element
2) Divide rearrange elements so that x goes to its final position E
3) Recurse and Conquer recursively sort
Quick Sort Each iteration of the quick sort
selects an element known as the pivot and divides the list into 3 groups Elements whose keys are less than (or
equal to) the pivotrsquos key The pivot element Elements whose keys are greater than
(or equal to) the pivotrsquos key
Quick Sort The sorting then continues by
quick sorting the left partition followed by quick sorting the right partition
The basic algorithm is as follows
Quick Sort1) Partitioning Step Take an element in the
unsorted array and determine its final location in the sorted array This occurs when all values to the left of the element in the array are less than (or equal to) the element and all values to the right of the element are greater than (or equal to) the element We now have 1 element in its proper location and two unsorted subarrays
2) Recursive Step Perform step 1 on each unsorted subarray
Quick Sort Each time step 1 is performed on a
subarray another element is placed in its final location of the sorted array and two unsorted subarrays are created
When a subarray consists of one element that subarray is sorted
Therefore that element is in its final location
Quick Sort There are several partitioning strategies
used in practice (ie several ldquoversionsrdquo of quick sort) but the one we are about to describe is known to work well
For simplicity we will choose the last element to be the pivot element
We could also chose a different pivot element and swap it with the last element in the array
Quick Sort Below is the array we would like to
sort1 4 8 9 0 11 5 10 7 6
Quick Sort The index left starts at the first element
and right starts at the next-to-last element
We want to move all the elements smaller than the pivot to the left part of the array and all the elements larger than the pivot to the right part
1 4 8 9 0 11 5 10 7 6left righ
t
Quick Sort We move left to the right skipping
over elements that are smaller than the pivot
1 4 8 9 0 11 5 10 7 6left righ
t
Quick Sort We then move right to the left skipping
over elements that are greater than the pivot
When left and right have stopped left is on an element greater than (or equal to) the pivot and right is on an element smaller than (or equal to) the pivot
1 4 8 9 0 11 5 10 7 6left right
Quick Sort If left is to the left of right (or if left
= right) those elements are swapped
1 4 8 9 0 11 5 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
Quick Sort The effect is to push a large
element to the right and a small element to the left
We then repeat the process until left and right cross
Quick Sort
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6left righ
t
Quick Sort
1 4 5 0 9 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6righ
tleft
Quick Sort
At this point left and right have crossed so no swap is performed
The final part of the partitioning is to swap the pivot element with left
1 4 5 0 9 11 8 10 7 6righ
tleft
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort Note that all elements to the left of
the pivot are less than (or equal to) the pivot and all elements to the right of the pivot are greater than (or equal to) the pivot
Hence the pivot element has been placed in its final sorted position
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort We now repeat the process using
the sub-arrays to the left and right of the pivot
1 4 5 0 6 11 8 10 7 9
1 4 5 0 11 8 10 7 9
Analysis of quicksortmdashbest case
Suppose each partition operation divides the array almost exactly in half
Then the depth of the recursion in log2n Because thatrsquos how many times we can
halve n However there are many recursions at
each level How can we figure this out We note that
Each partition is linear over its subarray All the partitions at one level cover the array
Partitioning at various levels
Best case II We cut the array size in half each time So the depth of the recursion in log2n At each level of the recursion all the
partitions at that level do work that is linear in n
O(log2n) O(n) = O(n log2n)
What about the worst case
Worst case In the worst case partitioning always
divides the size n array into these three parts A length one part containing the pivot itself A length zero part and A length n-1 part containing everything else
We donrsquot recur on the zero-length part Recurring on the length n-1 part requires
(in the worst case) recurring to depth n-1
Worst case partitioning
Worst case for quicksort In the worst case recursion may be
O(n) levels deep (for an array of size n)
But the partitioning work done at each level is still O(n)
O(n) O(n) = O(n2) So worst case for Quicksort is O(n2) When does this happen
When the array is sorted to begin with
Typical case for quicksort If the array is sorted to begin with
Quicksort is terrible O(n2) It is possible to construct other bad
cases However Quicksort is usually O(n log2n) The constants are so good that Quicksort
is generally the fastest algorithm known Most real-world sorting is done by
Quicksort
Tweaking Quicksort Almost anything you can try to
ldquoimproverdquo Quicksort will actually slow it down
One good tweak is to switch to a different sorting method when the subarrays get small (say 10 or 12) Quicksort has too much overhead for small
array sizes For large arrays it might be a good
idea to check beforehand if the array is already sorted But there is a better tweak than this
Randomized Quick-Sort Select the pivot as a random element of the sequence The expected running time of randomized quick-sort on a
sequence of size n is O(n log n) The time spent at a level of the quick-sort tree is O(n) We show that the expected height of the quick-sort tree is
O(log n) good vs bad pivots
bull The probability of a good pivot is 12 thus we expect k2 good pivots out of k pivotsbull After a good pivot the size of each child sequence is at most 34 the size of the parent sequencebull After h pivots we expect (34)h2 n elementsbull The expected height h of the quick-sort tree is at most 2 log43n
Median of three Obviously it doesnrsquot make sense to sort the
array in order to find the median to use as a pivot
Instead compare just three elements of our (sub)arraymdashthe first the last and the middle Take the median (middle value) of these three as
pivot Itrsquos possible (but not easy) to construct cases
which will make this technique O(n2) Suppose we rearrange (sort) these three
numbers so that the smallest is in the first position the largest in the last position and the other in the middle This lets us simplify and speed up the partition
loop
Final comments Weissrsquos code shows some additional
optimizations on pp 246-247 Weiss chooses to stop both searches on
equality to pivot This design decision is debatable
Quicksort is the fastest known general sorting algorithm on average
For optimum speed the pivot must be chosen carefully
ldquoMedian of threerdquo is a good technique for choosing the pivot
There will be some cases where Quicksort runs in O(n2) time
Quick Sort A couple of notes about quick sort There are more optimal ways to
choose the pivot value (such as the median-of-three method)
Also when the subarrays get small it becomes more efficient to use the insertion sort as opposed to continued use of quick sort
Bubble Sort vs Quick Sort If we calculate the Big-O notation
we find that (in the average case) Bubble Sort O(n2) Quick Sort O(nlog2n)
(quick-sort runs in time O(n2) in the worst case)
Heap Sort Idea take the items that need to be
sorted and insert them into the heap By calling deleteHeap we remove the
smallest or largest element depending on whether or not we are working with a min- or max-heap respectively
Hence the elements are removed in ascending or descending order
Efficiency O(nlog2n)
Merge Sort Idea
Take the array you would like to sort and divide it in half to create 2 unsorted subarrays
Next sort each of the 2 subarrays Finally merge the 2 sorted subarrays
into 1 sorted array Efficiency O(nlog2n)
Merge Sort
Merge Sort Although the merge step produces
a sorted array we have overlooked a very important step
How did we sort the 2 halves before performing the merge step
We used merge sort
Merge Sort By continually calling the merge
sort algorithm we eventually get a subarray of size 1
Since an array with only 1 element is clearly sorted we can back out and merge 2 arrays of size 1
Merge Sort
Merge Sort The basic merging algorithm
consists of 2 input arrays (arrayA and arrayB) An ouput array (arrayC) 3 position holders (indexA indexB
indexC) which are initially set to the beginning of their respective arrays
Merge Sort The smaller of arrayA[indexA] and
arrayB[indexB] is copied into arrayC[indexC] and the appropriate position holders are advanced
When either input list is exhausted the remainder of the other list is copied into arrayC
Merge Sort
1 13 24 26index
A
indexC
arrayA
2 15 27 38index
B
arrayB
arrayC
We compare arrayA[indexA]with arrayB[indexB] Whichever value is smaller isplaced into arrayC[indexC]
1 lt 2 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1index
C
arrayA
2 15 27 38index
B
arrayB
arrayC
2 lt 13 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
13 lt 15 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
15 lt 24 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13 15indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
24 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
indexA
1 2 13 15 24indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
26 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
1 2 13 15 24 26indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
Since we have exhaustedone of the arrays arrayAwe simply copy the remaining items from theother array arrayB intoarrayC
Merge Sort
1 13 24
26
1 2 13 15 24 26 27 38
arrayA
2 15 27 38arrayB
arrayC
Efficiency Summary
Sort Worst Case Average Case
Insertion O(n2) O(n2)
Shell O(n15) O(n125)
Selection O(n2) O(n2)
Bubble O(n2) O(n2)
Quick O(n2) O(nlog2n)
Heap O(nlog2n) O(nlog2n)
Merge O(nlog2n) O(nlog2n)
More Sorting
radix sort bucket sort in-place sorting how fast can we sort
Radix Sort
Unlike other sorting methods radix sort considers the structure of the keys
Assume keys are represented in a base M number system (M = radix) ie if M = 2 the keys are represented in binary
Sorting is done by comparing bits in the same position
Extension to keys that are alphanumeric strings
Radix Exchange Sort
Examine bits from left to right1 Sort array with respect to leftmost bit
2 Partition array
3Recursionrecursively sort top subarray ignoring leftmost bitrecursively sort bottom subarray ignoring leftmost bit
Time to sort n b-bit numbers O(b n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Sort Stability
If however we use an unstable sort data with equal keys can be sorted in any order (ie not necessarily in the same order as they appeared in the unsorted data)
Sort Efficiency Sort efficiency is a measure of
the relative efficiency of a sort It is usually an estimate of the
number of comparisons and moves required to order an unordered list
Passes During the sorting process the data is
traversed many times Each traversal of the data is referred to
as a sort pass Depending on the algorithm the sort
pass may traverse the whole list or just a section of the list
The sort pass may also include the placement of one or more elements into the sorted list
Types of Sorts We now discuss several sorting
methods Insertion Sort Shell Sort Selection Sort Bubble Sort Quick Sort Heap Sort Merge Sort
Insertion Sort In each pass of an insertion sort
one or more pieces of data are inserted into their correct location in an ordered list (just as a card player picks up cards and places them in his hand in order)
Insertion Sort In the insertion sort the list is divided into
2 parts Sorted Unsorted
In each pass the first element of the unsorted sublist is transferred to the sorted sublist by inserting it at the appropriate place
If we have a list of n elements it will take at most n-1 passes to sort the data
Insertion Sort
We can visualize this type of sort with the above figure
The first part of the list is the sorted portion which is separated by a ldquoconceptualrdquo wall from the unsorted portion of the list
Insertion SortHere we start with an unsorted list We leave the first data element alone and will start with the 2nd element of the list (the 1st element of the unsorted list)
On our first pass we look at the 2nd element of the list (thefirst element of the unsorted list) and place it in our sorted list in order
On our next pass we look at the 3rd element of the list (the1st element of the unsorted list) and place it in the sorted list in order
We continue in this fashion until the entire list has beensorted
Insertion Sort Example
In our first pass we compare the first 2 values Because theyare out of order we swap them
Now we look at the next 2 values Again they are out oforder so we swap them
Since we have swapped those values we need to comparethe previous 2 values to make sure that they are still in orderSince they are out of order we swap them and then continueon with the next 2 data valuesThese 2 values are out of order so we swap them and look atthe previous 2 values etc
Shell Sort Named after its creator Donald Shell the
shell sort is an improved version of the insertion sort
In the shell sort a list of N elements is divided into K segments where K is known as the increment
What this means is that instead of comparing adjacent values we will compare values that are a distance K apart
We will shrink K as we run through our algorithm
Shell SortPass Notes
1 77 62 14 9 30 21 80 25 70 55 Swap2 21 62 14 9 30 77 80 25 70 55 In order3 21 62 14 9 30 77 80 25 70 55 In order4 21 62 14 9 30 77 80 25 70 55 In order5 21 62 14 9 30 77 80 25 70 55 In order
List (K=5)
Just as in the straight insertion sort we compare 2 values andswap them if they are out of order However in the shell sort we compare values that are a distance K apartOnce we have completed going through the elements in our
list with K=5 we decrease K and continue the process
Shell SortPass Notes
6 21 62 14 9 30 77 80 25 70 55 Swap7 14 62 21 9 30 77 80 25 70 55 Swap8 14 9 21 62 30 77 80 25 70 55 In order9 14 9 21 62 30 77 80 25 70 55 In order10 14 9 21 62 30 77 80 25 70 55 In order11 14 9 21 62 30 77 80 25 70 55 Swap
14 9 21 62 30 25 80 77 70 55 Swap14 9 21 25 30 62 80 77 70 55 In order
12 14 9 21 25 30 62 80 77 70 55 Swap14 9 21 25 30 62 70 77 80 55 In order
13 14 9 21 25 30 62 70 77 80 55 Swap14 9 21 25 30 62 70 55 80 77 Swap14 9 21 25 30 55 70 62 80 77 In order
List (K=2)
Here we have reduced K to 2 Just as in the insertion sort if we swap 2 values we have to go back and compare theprevious 2 values to make sure they are still in order
Shell Sort
All shell sorts will terminate by running an insertion sort(ie K=1) However using the larger values of K first has helped to sort our list so that the straight insertion sort will runfaster
Shell Sort There are many schools of thought
on what the increment should be in the shell sort
Also note that just because an increment is optimal on one list it might not be optimal for another list
Insertion Sort vs Shell Sort Comparing the Big-O notation (for the
average case) we find that Insertion O(n2) Shell O(n125) empirically determined
Although this doesnrsquot seem like much of a gain it makes a big difference as n gets large
Note that in the worst case the Shell sort has an efficiency of O(n2)
However using a special incrementing technique this worst case can be reduced to O(n15)
Insertion Sort vs Shell Sort(Average Case)
Straight Insertion Shell25 625 55100 10000 316500 250000 2364
1000 1000000 56232000 4000000 13374
Number of Loopsn
Selection Sort Imagine some data that you can examine
all at once To sort it you could select the smallest
element and put it in its place select the next smallest and put it in its place etc
For a card player this process is analogous to looking at an entire hand of cards and ordering them by selecting cards one at a time and placing them in their proper order
Selection Sort The selection sort follows this idea Given a list of data to be sorted
we simply select the smallest item and place it in a sorted list
We then repeat these steps until the list is sorted
Selection Sort In the selection sort the list at any moment
is divided into 2 sublists sorted and unsorted separated by a ldquoconceptualrdquo wall
We select the smallest element from the unsorted sublist and exchange it with the element at the beginning of the unsorted data
After each selection and exchange the wall between the 2 sublists moves ndash increasing the number of sorted elements and decreasing the number of unsorted elements
Selection Sort
We start with an unsorted list We search this list for thesmallest element We then exchange the smallest element (8)with the first element in the unsorted list (23)
Again we search the unsorted list for the smallest element We then exchange the smallest element (23) with the first element in the unsorted list (78)
This process continues until the list is fully sorted
Bubble Sort In the bubble sort the list at any
moment is divided into 2 sublists sorted and unsorted
The smallest element is ldquobubbledrdquo from the unsorted sublist to the sorted sublist
Bubble Sort
23 78 45 8 56 3232 56
We start with 32 and compareit with 56 Because 32 is lessthan 56 we swap the twoand step down one element
We then compare 32 and 8Because 32 is not less than 8we do not swap these elements
We step down one element andcompare 45 and 8 They areout of sequence so we swapthem and step down again
8 45
We step down again and compare 8 with 78 These two elements are swapped
8 78
Finally 8 is compared with 23and swapped We thencontinue this process back with56 hellip
8 23
1 Pass of the Bubble Sort
Quick Sort In the bubble sort consecutive items
are compared and possibly exchanged on each pass through the list
This means that many exchanges may be needed to move an element to its correct position
Quick sort is more efficient than bubble sort because a typical exchange involves elements that are far apart so fewer exchanges are required to correctly position an element
Idea of Quick Sort1) Select pick an element
2) Divide rearrange elements so that x goes to its final position E
3) Recurse and Conquer recursively sort
Quick Sort Each iteration of the quick sort
selects an element known as the pivot and divides the list into 3 groups Elements whose keys are less than (or
equal to) the pivotrsquos key The pivot element Elements whose keys are greater than
(or equal to) the pivotrsquos key
Quick Sort The sorting then continues by
quick sorting the left partition followed by quick sorting the right partition
The basic algorithm is as follows
Quick Sort1) Partitioning Step Take an element in the
unsorted array and determine its final location in the sorted array This occurs when all values to the left of the element in the array are less than (or equal to) the element and all values to the right of the element are greater than (or equal to) the element We now have 1 element in its proper location and two unsorted subarrays
2) Recursive Step Perform step 1 on each unsorted subarray
Quick Sort Each time step 1 is performed on a
subarray another element is placed in its final location of the sorted array and two unsorted subarrays are created
When a subarray consists of one element that subarray is sorted
Therefore that element is in its final location
Quick Sort There are several partitioning strategies
used in practice (ie several ldquoversionsrdquo of quick sort) but the one we are about to describe is known to work well
For simplicity we will choose the last element to be the pivot element
We could also chose a different pivot element and swap it with the last element in the array
Quick Sort Below is the array we would like to
sort1 4 8 9 0 11 5 10 7 6
Quick Sort The index left starts at the first element
and right starts at the next-to-last element
We want to move all the elements smaller than the pivot to the left part of the array and all the elements larger than the pivot to the right part
1 4 8 9 0 11 5 10 7 6left righ
t
Quick Sort We move left to the right skipping
over elements that are smaller than the pivot
1 4 8 9 0 11 5 10 7 6left righ
t
Quick Sort We then move right to the left skipping
over elements that are greater than the pivot
When left and right have stopped left is on an element greater than (or equal to) the pivot and right is on an element smaller than (or equal to) the pivot
1 4 8 9 0 11 5 10 7 6left right
Quick Sort If left is to the left of right (or if left
= right) those elements are swapped
1 4 8 9 0 11 5 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
Quick Sort The effect is to push a large
element to the right and a small element to the left
We then repeat the process until left and right cross
Quick Sort
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6left righ
t
Quick Sort
1 4 5 0 9 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6righ
tleft
Quick Sort
At this point left and right have crossed so no swap is performed
The final part of the partitioning is to swap the pivot element with left
1 4 5 0 9 11 8 10 7 6righ
tleft
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort Note that all elements to the left of
the pivot are less than (or equal to) the pivot and all elements to the right of the pivot are greater than (or equal to) the pivot
Hence the pivot element has been placed in its final sorted position
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort We now repeat the process using
the sub-arrays to the left and right of the pivot
1 4 5 0 6 11 8 10 7 9
1 4 5 0 11 8 10 7 9
Analysis of quicksortmdashbest case
Suppose each partition operation divides the array almost exactly in half
Then the depth of the recursion in log2n Because thatrsquos how many times we can
halve n However there are many recursions at
each level How can we figure this out We note that
Each partition is linear over its subarray All the partitions at one level cover the array
Partitioning at various levels
Best case II We cut the array size in half each time So the depth of the recursion in log2n At each level of the recursion all the
partitions at that level do work that is linear in n
O(log2n) O(n) = O(n log2n)
What about the worst case
Worst case In the worst case partitioning always
divides the size n array into these three parts A length one part containing the pivot itself A length zero part and A length n-1 part containing everything else
We donrsquot recur on the zero-length part Recurring on the length n-1 part requires
(in the worst case) recurring to depth n-1
Worst case partitioning
Worst case for quicksort In the worst case recursion may be
O(n) levels deep (for an array of size n)
But the partitioning work done at each level is still O(n)
O(n) O(n) = O(n2) So worst case for Quicksort is O(n2) When does this happen
When the array is sorted to begin with
Typical case for quicksort If the array is sorted to begin with
Quicksort is terrible O(n2) It is possible to construct other bad
cases However Quicksort is usually O(n log2n) The constants are so good that Quicksort
is generally the fastest algorithm known Most real-world sorting is done by
Quicksort
Tweaking Quicksort Almost anything you can try to
ldquoimproverdquo Quicksort will actually slow it down
One good tweak is to switch to a different sorting method when the subarrays get small (say 10 or 12) Quicksort has too much overhead for small
array sizes For large arrays it might be a good
idea to check beforehand if the array is already sorted But there is a better tweak than this
Randomized Quick-Sort Select the pivot as a random element of the sequence The expected running time of randomized quick-sort on a
sequence of size n is O(n log n) The time spent at a level of the quick-sort tree is O(n) We show that the expected height of the quick-sort tree is
O(log n) good vs bad pivots
bull The probability of a good pivot is 12 thus we expect k2 good pivots out of k pivotsbull After a good pivot the size of each child sequence is at most 34 the size of the parent sequencebull After h pivots we expect (34)h2 n elementsbull The expected height h of the quick-sort tree is at most 2 log43n
Median of three Obviously it doesnrsquot make sense to sort the
array in order to find the median to use as a pivot
Instead compare just three elements of our (sub)arraymdashthe first the last and the middle Take the median (middle value) of these three as
pivot Itrsquos possible (but not easy) to construct cases
which will make this technique O(n2) Suppose we rearrange (sort) these three
numbers so that the smallest is in the first position the largest in the last position and the other in the middle This lets us simplify and speed up the partition
loop
Final comments Weissrsquos code shows some additional
optimizations on pp 246-247 Weiss chooses to stop both searches on
equality to pivot This design decision is debatable
Quicksort is the fastest known general sorting algorithm on average
For optimum speed the pivot must be chosen carefully
ldquoMedian of threerdquo is a good technique for choosing the pivot
There will be some cases where Quicksort runs in O(n2) time
Quick Sort A couple of notes about quick sort There are more optimal ways to
choose the pivot value (such as the median-of-three method)
Also when the subarrays get small it becomes more efficient to use the insertion sort as opposed to continued use of quick sort
Bubble Sort vs Quick Sort If we calculate the Big-O notation
we find that (in the average case) Bubble Sort O(n2) Quick Sort O(nlog2n)
(quick-sort runs in time O(n2) in the worst case)
Heap Sort Idea take the items that need to be
sorted and insert them into the heap By calling deleteHeap we remove the
smallest or largest element depending on whether or not we are working with a min- or max-heap respectively
Hence the elements are removed in ascending or descending order
Efficiency O(nlog2n)
Merge Sort Idea
Take the array you would like to sort and divide it in half to create 2 unsorted subarrays
Next sort each of the 2 subarrays Finally merge the 2 sorted subarrays
into 1 sorted array Efficiency O(nlog2n)
Merge Sort
Merge Sort Although the merge step produces
a sorted array we have overlooked a very important step
How did we sort the 2 halves before performing the merge step
We used merge sort
Merge Sort By continually calling the merge
sort algorithm we eventually get a subarray of size 1
Since an array with only 1 element is clearly sorted we can back out and merge 2 arrays of size 1
Merge Sort
Merge Sort The basic merging algorithm
consists of 2 input arrays (arrayA and arrayB) An ouput array (arrayC) 3 position holders (indexA indexB
indexC) which are initially set to the beginning of their respective arrays
Merge Sort The smaller of arrayA[indexA] and
arrayB[indexB] is copied into arrayC[indexC] and the appropriate position holders are advanced
When either input list is exhausted the remainder of the other list is copied into arrayC
Merge Sort
1 13 24 26index
A
indexC
arrayA
2 15 27 38index
B
arrayB
arrayC
We compare arrayA[indexA]with arrayB[indexB] Whichever value is smaller isplaced into arrayC[indexC]
1 lt 2 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1index
C
arrayA
2 15 27 38index
B
arrayB
arrayC
2 lt 13 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
13 lt 15 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
15 lt 24 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13 15indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
24 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
indexA
1 2 13 15 24indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
26 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
1 2 13 15 24 26indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
Since we have exhaustedone of the arrays arrayAwe simply copy the remaining items from theother array arrayB intoarrayC
Merge Sort
1 13 24
26
1 2 13 15 24 26 27 38
arrayA
2 15 27 38arrayB
arrayC
Efficiency Summary
Sort Worst Case Average Case
Insertion O(n2) O(n2)
Shell O(n15) O(n125)
Selection O(n2) O(n2)
Bubble O(n2) O(n2)
Quick O(n2) O(nlog2n)
Heap O(nlog2n) O(nlog2n)
Merge O(nlog2n) O(nlog2n)
More Sorting
radix sort bucket sort in-place sorting how fast can we sort
Radix Sort
Unlike other sorting methods radix sort considers the structure of the keys
Assume keys are represented in a base M number system (M = radix) ie if M = 2 the keys are represented in binary
Sorting is done by comparing bits in the same position
Extension to keys that are alphanumeric strings
Radix Exchange Sort
Examine bits from left to right1 Sort array with respect to leftmost bit
2 Partition array
3Recursionrecursively sort top subarray ignoring leftmost bitrecursively sort bottom subarray ignoring leftmost bit
Time to sort n b-bit numbers O(b n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Sort Efficiency Sort efficiency is a measure of
the relative efficiency of a sort It is usually an estimate of the
number of comparisons and moves required to order an unordered list
Passes During the sorting process the data is
traversed many times Each traversal of the data is referred to
as a sort pass Depending on the algorithm the sort
pass may traverse the whole list or just a section of the list
The sort pass may also include the placement of one or more elements into the sorted list
Types of Sorts We now discuss several sorting
methods Insertion Sort Shell Sort Selection Sort Bubble Sort Quick Sort Heap Sort Merge Sort
Insertion Sort In each pass of an insertion sort
one or more pieces of data are inserted into their correct location in an ordered list (just as a card player picks up cards and places them in his hand in order)
Insertion Sort In the insertion sort the list is divided into
2 parts Sorted Unsorted
In each pass the first element of the unsorted sublist is transferred to the sorted sublist by inserting it at the appropriate place
If we have a list of n elements it will take at most n-1 passes to sort the data
Insertion Sort
We can visualize this type of sort with the above figure
The first part of the list is the sorted portion which is separated by a ldquoconceptualrdquo wall from the unsorted portion of the list
Insertion SortHere we start with an unsorted list We leave the first data element alone and will start with the 2nd element of the list (the 1st element of the unsorted list)
On our first pass we look at the 2nd element of the list (thefirst element of the unsorted list) and place it in our sorted list in order
On our next pass we look at the 3rd element of the list (the1st element of the unsorted list) and place it in the sorted list in order
We continue in this fashion until the entire list has beensorted
Insertion Sort Example
In our first pass we compare the first 2 values Because theyare out of order we swap them
Now we look at the next 2 values Again they are out oforder so we swap them
Since we have swapped those values we need to comparethe previous 2 values to make sure that they are still in orderSince they are out of order we swap them and then continueon with the next 2 data valuesThese 2 values are out of order so we swap them and look atthe previous 2 values etc
Shell Sort Named after its creator Donald Shell the
shell sort is an improved version of the insertion sort
In the shell sort a list of N elements is divided into K segments where K is known as the increment
What this means is that instead of comparing adjacent values we will compare values that are a distance K apart
We will shrink K as we run through our algorithm
Shell SortPass Notes
1 77 62 14 9 30 21 80 25 70 55 Swap2 21 62 14 9 30 77 80 25 70 55 In order3 21 62 14 9 30 77 80 25 70 55 In order4 21 62 14 9 30 77 80 25 70 55 In order5 21 62 14 9 30 77 80 25 70 55 In order
List (K=5)
Just as in the straight insertion sort we compare 2 values andswap them if they are out of order However in the shell sort we compare values that are a distance K apartOnce we have completed going through the elements in our
list with K=5 we decrease K and continue the process
Shell SortPass Notes
6 21 62 14 9 30 77 80 25 70 55 Swap7 14 62 21 9 30 77 80 25 70 55 Swap8 14 9 21 62 30 77 80 25 70 55 In order9 14 9 21 62 30 77 80 25 70 55 In order10 14 9 21 62 30 77 80 25 70 55 In order11 14 9 21 62 30 77 80 25 70 55 Swap
14 9 21 62 30 25 80 77 70 55 Swap14 9 21 25 30 62 80 77 70 55 In order
12 14 9 21 25 30 62 80 77 70 55 Swap14 9 21 25 30 62 70 77 80 55 In order
13 14 9 21 25 30 62 70 77 80 55 Swap14 9 21 25 30 62 70 55 80 77 Swap14 9 21 25 30 55 70 62 80 77 In order
List (K=2)
Here we have reduced K to 2 Just as in the insertion sort if we swap 2 values we have to go back and compare theprevious 2 values to make sure they are still in order
Shell Sort
All shell sorts will terminate by running an insertion sort(ie K=1) However using the larger values of K first has helped to sort our list so that the straight insertion sort will runfaster
Shell Sort There are many schools of thought
on what the increment should be in the shell sort
Also note that just because an increment is optimal on one list it might not be optimal for another list
Insertion Sort vs Shell Sort Comparing the Big-O notation (for the
average case) we find that Insertion O(n2) Shell O(n125) empirically determined
Although this doesnrsquot seem like much of a gain it makes a big difference as n gets large
Note that in the worst case the Shell sort has an efficiency of O(n2)
However using a special incrementing technique this worst case can be reduced to O(n15)
Insertion Sort vs Shell Sort(Average Case)
Straight Insertion Shell25 625 55100 10000 316500 250000 2364
1000 1000000 56232000 4000000 13374
Number of Loopsn
Selection Sort Imagine some data that you can examine
all at once To sort it you could select the smallest
element and put it in its place select the next smallest and put it in its place etc
For a card player this process is analogous to looking at an entire hand of cards and ordering them by selecting cards one at a time and placing them in their proper order
Selection Sort The selection sort follows this idea Given a list of data to be sorted
we simply select the smallest item and place it in a sorted list
We then repeat these steps until the list is sorted
Selection Sort In the selection sort the list at any moment
is divided into 2 sublists sorted and unsorted separated by a ldquoconceptualrdquo wall
We select the smallest element from the unsorted sublist and exchange it with the element at the beginning of the unsorted data
After each selection and exchange the wall between the 2 sublists moves ndash increasing the number of sorted elements and decreasing the number of unsorted elements
Selection Sort
We start with an unsorted list We search this list for thesmallest element We then exchange the smallest element (8)with the first element in the unsorted list (23)
Again we search the unsorted list for the smallest element We then exchange the smallest element (23) with the first element in the unsorted list (78)
This process continues until the list is fully sorted
Bubble Sort In the bubble sort the list at any
moment is divided into 2 sublists sorted and unsorted
The smallest element is ldquobubbledrdquo from the unsorted sublist to the sorted sublist
Bubble Sort
23 78 45 8 56 3232 56
We start with 32 and compareit with 56 Because 32 is lessthan 56 we swap the twoand step down one element
We then compare 32 and 8Because 32 is not less than 8we do not swap these elements
We step down one element andcompare 45 and 8 They areout of sequence so we swapthem and step down again
8 45
We step down again and compare 8 with 78 These two elements are swapped
8 78
Finally 8 is compared with 23and swapped We thencontinue this process back with56 hellip
8 23
1 Pass of the Bubble Sort
Quick Sort In the bubble sort consecutive items
are compared and possibly exchanged on each pass through the list
This means that many exchanges may be needed to move an element to its correct position
Quick sort is more efficient than bubble sort because a typical exchange involves elements that are far apart so fewer exchanges are required to correctly position an element
Idea of Quick Sort1) Select pick an element
2) Divide rearrange elements so that x goes to its final position E
3) Recurse and Conquer recursively sort
Quick Sort Each iteration of the quick sort
selects an element known as the pivot and divides the list into 3 groups Elements whose keys are less than (or
equal to) the pivotrsquos key The pivot element Elements whose keys are greater than
(or equal to) the pivotrsquos key
Quick Sort The sorting then continues by
quick sorting the left partition followed by quick sorting the right partition
The basic algorithm is as follows
Quick Sort1) Partitioning Step Take an element in the
unsorted array and determine its final location in the sorted array This occurs when all values to the left of the element in the array are less than (or equal to) the element and all values to the right of the element are greater than (or equal to) the element We now have 1 element in its proper location and two unsorted subarrays
2) Recursive Step Perform step 1 on each unsorted subarray
Quick Sort Each time step 1 is performed on a
subarray another element is placed in its final location of the sorted array and two unsorted subarrays are created
When a subarray consists of one element that subarray is sorted
Therefore that element is in its final location
Quick Sort There are several partitioning strategies
used in practice (ie several ldquoversionsrdquo of quick sort) but the one we are about to describe is known to work well
For simplicity we will choose the last element to be the pivot element
We could also chose a different pivot element and swap it with the last element in the array
Quick Sort Below is the array we would like to
sort1 4 8 9 0 11 5 10 7 6
Quick Sort The index left starts at the first element
and right starts at the next-to-last element
We want to move all the elements smaller than the pivot to the left part of the array and all the elements larger than the pivot to the right part
1 4 8 9 0 11 5 10 7 6left righ
t
Quick Sort We move left to the right skipping
over elements that are smaller than the pivot
1 4 8 9 0 11 5 10 7 6left righ
t
Quick Sort We then move right to the left skipping
over elements that are greater than the pivot
When left and right have stopped left is on an element greater than (or equal to) the pivot and right is on an element smaller than (or equal to) the pivot
1 4 8 9 0 11 5 10 7 6left right
Quick Sort If left is to the left of right (or if left
= right) those elements are swapped
1 4 8 9 0 11 5 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
Quick Sort The effect is to push a large
element to the right and a small element to the left
We then repeat the process until left and right cross
Quick Sort
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6left righ
t
Quick Sort
1 4 5 0 9 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6righ
tleft
Quick Sort
At this point left and right have crossed so no swap is performed
The final part of the partitioning is to swap the pivot element with left
1 4 5 0 9 11 8 10 7 6righ
tleft
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort Note that all elements to the left of
the pivot are less than (or equal to) the pivot and all elements to the right of the pivot are greater than (or equal to) the pivot
Hence the pivot element has been placed in its final sorted position
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort We now repeat the process using
the sub-arrays to the left and right of the pivot
1 4 5 0 6 11 8 10 7 9
1 4 5 0 11 8 10 7 9
Analysis of quicksortmdashbest case
Suppose each partition operation divides the array almost exactly in half
Then the depth of the recursion in log2n Because thatrsquos how many times we can
halve n However there are many recursions at
each level How can we figure this out We note that
Each partition is linear over its subarray All the partitions at one level cover the array
Partitioning at various levels
Best case II We cut the array size in half each time So the depth of the recursion in log2n At each level of the recursion all the
partitions at that level do work that is linear in n
O(log2n) O(n) = O(n log2n)
What about the worst case
Worst case In the worst case partitioning always
divides the size n array into these three parts A length one part containing the pivot itself A length zero part and A length n-1 part containing everything else
We donrsquot recur on the zero-length part Recurring on the length n-1 part requires
(in the worst case) recurring to depth n-1
Worst case partitioning
Worst case for quicksort In the worst case recursion may be
O(n) levels deep (for an array of size n)
But the partitioning work done at each level is still O(n)
O(n) O(n) = O(n2) So worst case for Quicksort is O(n2) When does this happen
When the array is sorted to begin with
Typical case for quicksort If the array is sorted to begin with
Quicksort is terrible O(n2) It is possible to construct other bad
cases However Quicksort is usually O(n log2n) The constants are so good that Quicksort
is generally the fastest algorithm known Most real-world sorting is done by
Quicksort
Tweaking Quicksort Almost anything you can try to
ldquoimproverdquo Quicksort will actually slow it down
One good tweak is to switch to a different sorting method when the subarrays get small (say 10 or 12) Quicksort has too much overhead for small
array sizes For large arrays it might be a good
idea to check beforehand if the array is already sorted But there is a better tweak than this
Randomized Quick-Sort Select the pivot as a random element of the sequence The expected running time of randomized quick-sort on a
sequence of size n is O(n log n) The time spent at a level of the quick-sort tree is O(n) We show that the expected height of the quick-sort tree is
O(log n) good vs bad pivots
bull The probability of a good pivot is 12 thus we expect k2 good pivots out of k pivotsbull After a good pivot the size of each child sequence is at most 34 the size of the parent sequencebull After h pivots we expect (34)h2 n elementsbull The expected height h of the quick-sort tree is at most 2 log43n
Median of three Obviously it doesnrsquot make sense to sort the
array in order to find the median to use as a pivot
Instead compare just three elements of our (sub)arraymdashthe first the last and the middle Take the median (middle value) of these three as
pivot Itrsquos possible (but not easy) to construct cases
which will make this technique O(n2) Suppose we rearrange (sort) these three
numbers so that the smallest is in the first position the largest in the last position and the other in the middle This lets us simplify and speed up the partition
loop
Final comments Weissrsquos code shows some additional
optimizations on pp 246-247 Weiss chooses to stop both searches on
equality to pivot This design decision is debatable
Quicksort is the fastest known general sorting algorithm on average
For optimum speed the pivot must be chosen carefully
ldquoMedian of threerdquo is a good technique for choosing the pivot
There will be some cases where Quicksort runs in O(n2) time
Quick Sort A couple of notes about quick sort There are more optimal ways to
choose the pivot value (such as the median-of-three method)
Also when the subarrays get small it becomes more efficient to use the insertion sort as opposed to continued use of quick sort
Bubble Sort vs Quick Sort If we calculate the Big-O notation
we find that (in the average case) Bubble Sort O(n2) Quick Sort O(nlog2n)
(quick-sort runs in time O(n2) in the worst case)
Heap Sort Idea take the items that need to be
sorted and insert them into the heap By calling deleteHeap we remove the
smallest or largest element depending on whether or not we are working with a min- or max-heap respectively
Hence the elements are removed in ascending or descending order
Efficiency O(nlog2n)
Merge Sort Idea
Take the array you would like to sort and divide it in half to create 2 unsorted subarrays
Next sort each of the 2 subarrays Finally merge the 2 sorted subarrays
into 1 sorted array Efficiency O(nlog2n)
Merge Sort
Merge Sort Although the merge step produces
a sorted array we have overlooked a very important step
How did we sort the 2 halves before performing the merge step
We used merge sort
Merge Sort By continually calling the merge
sort algorithm we eventually get a subarray of size 1
Since an array with only 1 element is clearly sorted we can back out and merge 2 arrays of size 1
Merge Sort
Merge Sort The basic merging algorithm
consists of 2 input arrays (arrayA and arrayB) An ouput array (arrayC) 3 position holders (indexA indexB
indexC) which are initially set to the beginning of their respective arrays
Merge Sort The smaller of arrayA[indexA] and
arrayB[indexB] is copied into arrayC[indexC] and the appropriate position holders are advanced
When either input list is exhausted the remainder of the other list is copied into arrayC
Merge Sort
1 13 24 26index
A
indexC
arrayA
2 15 27 38index
B
arrayB
arrayC
We compare arrayA[indexA]with arrayB[indexB] Whichever value is smaller isplaced into arrayC[indexC]
1 lt 2 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1index
C
arrayA
2 15 27 38index
B
arrayB
arrayC
2 lt 13 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
13 lt 15 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
15 lt 24 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13 15indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
24 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
indexA
1 2 13 15 24indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
26 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
1 2 13 15 24 26indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
Since we have exhaustedone of the arrays arrayAwe simply copy the remaining items from theother array arrayB intoarrayC
Merge Sort
1 13 24
26
1 2 13 15 24 26 27 38
arrayA
2 15 27 38arrayB
arrayC
Efficiency Summary
Sort Worst Case Average Case
Insertion O(n2) O(n2)
Shell O(n15) O(n125)
Selection O(n2) O(n2)
Bubble O(n2) O(n2)
Quick O(n2) O(nlog2n)
Heap O(nlog2n) O(nlog2n)
Merge O(nlog2n) O(nlog2n)
More Sorting
radix sort bucket sort in-place sorting how fast can we sort
Radix Sort
Unlike other sorting methods radix sort considers the structure of the keys
Assume keys are represented in a base M number system (M = radix) ie if M = 2 the keys are represented in binary
Sorting is done by comparing bits in the same position
Extension to keys that are alphanumeric strings
Radix Exchange Sort
Examine bits from left to right1 Sort array with respect to leftmost bit
2 Partition array
3Recursionrecursively sort top subarray ignoring leftmost bitrecursively sort bottom subarray ignoring leftmost bit
Time to sort n b-bit numbers O(b n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Passes During the sorting process the data is
traversed many times Each traversal of the data is referred to
as a sort pass Depending on the algorithm the sort
pass may traverse the whole list or just a section of the list
The sort pass may also include the placement of one or more elements into the sorted list
Types of Sorts We now discuss several sorting
methods Insertion Sort Shell Sort Selection Sort Bubble Sort Quick Sort Heap Sort Merge Sort
Insertion Sort In each pass of an insertion sort
one or more pieces of data are inserted into their correct location in an ordered list (just as a card player picks up cards and places them in his hand in order)
Insertion Sort In the insertion sort the list is divided into
2 parts Sorted Unsorted
In each pass the first element of the unsorted sublist is transferred to the sorted sublist by inserting it at the appropriate place
If we have a list of n elements it will take at most n-1 passes to sort the data
Insertion Sort
We can visualize this type of sort with the above figure
The first part of the list is the sorted portion which is separated by a ldquoconceptualrdquo wall from the unsorted portion of the list
Insertion SortHere we start with an unsorted list We leave the first data element alone and will start with the 2nd element of the list (the 1st element of the unsorted list)
On our first pass we look at the 2nd element of the list (thefirst element of the unsorted list) and place it in our sorted list in order
On our next pass we look at the 3rd element of the list (the1st element of the unsorted list) and place it in the sorted list in order
We continue in this fashion until the entire list has beensorted
Insertion Sort Example
In our first pass we compare the first 2 values Because theyare out of order we swap them
Now we look at the next 2 values Again they are out oforder so we swap them
Since we have swapped those values we need to comparethe previous 2 values to make sure that they are still in orderSince they are out of order we swap them and then continueon with the next 2 data valuesThese 2 values are out of order so we swap them and look atthe previous 2 values etc
Shell Sort Named after its creator Donald Shell the
shell sort is an improved version of the insertion sort
In the shell sort a list of N elements is divided into K segments where K is known as the increment
What this means is that instead of comparing adjacent values we will compare values that are a distance K apart
We will shrink K as we run through our algorithm
Shell SortPass Notes
1 77 62 14 9 30 21 80 25 70 55 Swap2 21 62 14 9 30 77 80 25 70 55 In order3 21 62 14 9 30 77 80 25 70 55 In order4 21 62 14 9 30 77 80 25 70 55 In order5 21 62 14 9 30 77 80 25 70 55 In order
List (K=5)
Just as in the straight insertion sort we compare 2 values andswap them if they are out of order However in the shell sort we compare values that are a distance K apartOnce we have completed going through the elements in our
list with K=5 we decrease K and continue the process
Shell SortPass Notes
6 21 62 14 9 30 77 80 25 70 55 Swap7 14 62 21 9 30 77 80 25 70 55 Swap8 14 9 21 62 30 77 80 25 70 55 In order9 14 9 21 62 30 77 80 25 70 55 In order10 14 9 21 62 30 77 80 25 70 55 In order11 14 9 21 62 30 77 80 25 70 55 Swap
14 9 21 62 30 25 80 77 70 55 Swap14 9 21 25 30 62 80 77 70 55 In order
12 14 9 21 25 30 62 80 77 70 55 Swap14 9 21 25 30 62 70 77 80 55 In order
13 14 9 21 25 30 62 70 77 80 55 Swap14 9 21 25 30 62 70 55 80 77 Swap14 9 21 25 30 55 70 62 80 77 In order
List (K=2)
Here we have reduced K to 2 Just as in the insertion sort if we swap 2 values we have to go back and compare theprevious 2 values to make sure they are still in order
Shell Sort
All shell sorts will terminate by running an insertion sort(ie K=1) However using the larger values of K first has helped to sort our list so that the straight insertion sort will runfaster
Shell Sort There are many schools of thought
on what the increment should be in the shell sort
Also note that just because an increment is optimal on one list it might not be optimal for another list
Insertion Sort vs Shell Sort Comparing the Big-O notation (for the
average case) we find that Insertion O(n2) Shell O(n125) empirically determined
Although this doesnrsquot seem like much of a gain it makes a big difference as n gets large
Note that in the worst case the Shell sort has an efficiency of O(n2)
However using a special incrementing technique this worst case can be reduced to O(n15)
Insertion Sort vs Shell Sort(Average Case)
Straight Insertion Shell25 625 55100 10000 316500 250000 2364
1000 1000000 56232000 4000000 13374
Number of Loopsn
Selection Sort Imagine some data that you can examine
all at once To sort it you could select the smallest
element and put it in its place select the next smallest and put it in its place etc
For a card player this process is analogous to looking at an entire hand of cards and ordering them by selecting cards one at a time and placing them in their proper order
Selection Sort The selection sort follows this idea Given a list of data to be sorted
we simply select the smallest item and place it in a sorted list
We then repeat these steps until the list is sorted
Selection Sort In the selection sort the list at any moment
is divided into 2 sublists sorted and unsorted separated by a ldquoconceptualrdquo wall
We select the smallest element from the unsorted sublist and exchange it with the element at the beginning of the unsorted data
After each selection and exchange the wall between the 2 sublists moves ndash increasing the number of sorted elements and decreasing the number of unsorted elements
Selection Sort
We start with an unsorted list We search this list for thesmallest element We then exchange the smallest element (8)with the first element in the unsorted list (23)
Again we search the unsorted list for the smallest element We then exchange the smallest element (23) with the first element in the unsorted list (78)
This process continues until the list is fully sorted
Bubble Sort In the bubble sort the list at any
moment is divided into 2 sublists sorted and unsorted
The smallest element is ldquobubbledrdquo from the unsorted sublist to the sorted sublist
Bubble Sort
23 78 45 8 56 3232 56
We start with 32 and compareit with 56 Because 32 is lessthan 56 we swap the twoand step down one element
We then compare 32 and 8Because 32 is not less than 8we do not swap these elements
We step down one element andcompare 45 and 8 They areout of sequence so we swapthem and step down again
8 45
We step down again and compare 8 with 78 These two elements are swapped
8 78
Finally 8 is compared with 23and swapped We thencontinue this process back with56 hellip
8 23
1 Pass of the Bubble Sort
Quick Sort In the bubble sort consecutive items
are compared and possibly exchanged on each pass through the list
This means that many exchanges may be needed to move an element to its correct position
Quick sort is more efficient than bubble sort because a typical exchange involves elements that are far apart so fewer exchanges are required to correctly position an element
Idea of Quick Sort1) Select pick an element
2) Divide rearrange elements so that x goes to its final position E
3) Recurse and Conquer recursively sort
Quick Sort Each iteration of the quick sort
selects an element known as the pivot and divides the list into 3 groups Elements whose keys are less than (or
equal to) the pivotrsquos key The pivot element Elements whose keys are greater than
(or equal to) the pivotrsquos key
Quick Sort The sorting then continues by
quick sorting the left partition followed by quick sorting the right partition
The basic algorithm is as follows
Quick Sort1) Partitioning Step Take an element in the
unsorted array and determine its final location in the sorted array This occurs when all values to the left of the element in the array are less than (or equal to) the element and all values to the right of the element are greater than (or equal to) the element We now have 1 element in its proper location and two unsorted subarrays
2) Recursive Step Perform step 1 on each unsorted subarray
Quick Sort Each time step 1 is performed on a
subarray another element is placed in its final location of the sorted array and two unsorted subarrays are created
When a subarray consists of one element that subarray is sorted
Therefore that element is in its final location
Quick Sort There are several partitioning strategies
used in practice (ie several ldquoversionsrdquo of quick sort) but the one we are about to describe is known to work well
For simplicity we will choose the last element to be the pivot element
We could also chose a different pivot element and swap it with the last element in the array
Quick Sort Below is the array we would like to
sort1 4 8 9 0 11 5 10 7 6
Quick Sort The index left starts at the first element
and right starts at the next-to-last element
We want to move all the elements smaller than the pivot to the left part of the array and all the elements larger than the pivot to the right part
1 4 8 9 0 11 5 10 7 6left righ
t
Quick Sort We move left to the right skipping
over elements that are smaller than the pivot
1 4 8 9 0 11 5 10 7 6left righ
t
Quick Sort We then move right to the left skipping
over elements that are greater than the pivot
When left and right have stopped left is on an element greater than (or equal to) the pivot and right is on an element smaller than (or equal to) the pivot
1 4 8 9 0 11 5 10 7 6left right
Quick Sort If left is to the left of right (or if left
= right) those elements are swapped
1 4 8 9 0 11 5 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
Quick Sort The effect is to push a large
element to the right and a small element to the left
We then repeat the process until left and right cross
Quick Sort
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6left righ
t
Quick Sort
1 4 5 0 9 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6righ
tleft
Quick Sort
At this point left and right have crossed so no swap is performed
The final part of the partitioning is to swap the pivot element with left
1 4 5 0 9 11 8 10 7 6righ
tleft
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort Note that all elements to the left of
the pivot are less than (or equal to) the pivot and all elements to the right of the pivot are greater than (or equal to) the pivot
Hence the pivot element has been placed in its final sorted position
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort We now repeat the process using
the sub-arrays to the left and right of the pivot
1 4 5 0 6 11 8 10 7 9
1 4 5 0 11 8 10 7 9
Analysis of quicksortmdashbest case
Suppose each partition operation divides the array almost exactly in half
Then the depth of the recursion in log2n Because thatrsquos how many times we can
halve n However there are many recursions at
each level How can we figure this out We note that
Each partition is linear over its subarray All the partitions at one level cover the array
Partitioning at various levels
Best case II We cut the array size in half each time So the depth of the recursion in log2n At each level of the recursion all the
partitions at that level do work that is linear in n
O(log2n) O(n) = O(n log2n)
What about the worst case
Worst case In the worst case partitioning always
divides the size n array into these three parts A length one part containing the pivot itself A length zero part and A length n-1 part containing everything else
We donrsquot recur on the zero-length part Recurring on the length n-1 part requires
(in the worst case) recurring to depth n-1
Worst case partitioning
Worst case for quicksort In the worst case recursion may be
O(n) levels deep (for an array of size n)
But the partitioning work done at each level is still O(n)
O(n) O(n) = O(n2) So worst case for Quicksort is O(n2) When does this happen
When the array is sorted to begin with
Typical case for quicksort If the array is sorted to begin with
Quicksort is terrible O(n2) It is possible to construct other bad
cases However Quicksort is usually O(n log2n) The constants are so good that Quicksort
is generally the fastest algorithm known Most real-world sorting is done by
Quicksort
Tweaking Quicksort Almost anything you can try to
ldquoimproverdquo Quicksort will actually slow it down
One good tweak is to switch to a different sorting method when the subarrays get small (say 10 or 12) Quicksort has too much overhead for small
array sizes For large arrays it might be a good
idea to check beforehand if the array is already sorted But there is a better tweak than this
Randomized Quick-Sort Select the pivot as a random element of the sequence The expected running time of randomized quick-sort on a
sequence of size n is O(n log n) The time spent at a level of the quick-sort tree is O(n) We show that the expected height of the quick-sort tree is
O(log n) good vs bad pivots
bull The probability of a good pivot is 12 thus we expect k2 good pivots out of k pivotsbull After a good pivot the size of each child sequence is at most 34 the size of the parent sequencebull After h pivots we expect (34)h2 n elementsbull The expected height h of the quick-sort tree is at most 2 log43n
Median of three Obviously it doesnrsquot make sense to sort the
array in order to find the median to use as a pivot
Instead compare just three elements of our (sub)arraymdashthe first the last and the middle Take the median (middle value) of these three as
pivot Itrsquos possible (but not easy) to construct cases
which will make this technique O(n2) Suppose we rearrange (sort) these three
numbers so that the smallest is in the first position the largest in the last position and the other in the middle This lets us simplify and speed up the partition
loop
Final comments Weissrsquos code shows some additional
optimizations on pp 246-247 Weiss chooses to stop both searches on
equality to pivot This design decision is debatable
Quicksort is the fastest known general sorting algorithm on average
For optimum speed the pivot must be chosen carefully
ldquoMedian of threerdquo is a good technique for choosing the pivot
There will be some cases where Quicksort runs in O(n2) time
Quick Sort A couple of notes about quick sort There are more optimal ways to
choose the pivot value (such as the median-of-three method)
Also when the subarrays get small it becomes more efficient to use the insertion sort as opposed to continued use of quick sort
Bubble Sort vs Quick Sort If we calculate the Big-O notation
we find that (in the average case) Bubble Sort O(n2) Quick Sort O(nlog2n)
(quick-sort runs in time O(n2) in the worst case)
Heap Sort Idea take the items that need to be
sorted and insert them into the heap By calling deleteHeap we remove the
smallest or largest element depending on whether or not we are working with a min- or max-heap respectively
Hence the elements are removed in ascending or descending order
Efficiency O(nlog2n)
Merge Sort Idea
Take the array you would like to sort and divide it in half to create 2 unsorted subarrays
Next sort each of the 2 subarrays Finally merge the 2 sorted subarrays
into 1 sorted array Efficiency O(nlog2n)
Merge Sort
Merge Sort Although the merge step produces
a sorted array we have overlooked a very important step
How did we sort the 2 halves before performing the merge step
We used merge sort
Merge Sort By continually calling the merge
sort algorithm we eventually get a subarray of size 1
Since an array with only 1 element is clearly sorted we can back out and merge 2 arrays of size 1
Merge Sort
Merge Sort The basic merging algorithm
consists of 2 input arrays (arrayA and arrayB) An ouput array (arrayC) 3 position holders (indexA indexB
indexC) which are initially set to the beginning of their respective arrays
Merge Sort The smaller of arrayA[indexA] and
arrayB[indexB] is copied into arrayC[indexC] and the appropriate position holders are advanced
When either input list is exhausted the remainder of the other list is copied into arrayC
Merge Sort
1 13 24 26index
A
indexC
arrayA
2 15 27 38index
B
arrayB
arrayC
We compare arrayA[indexA]with arrayB[indexB] Whichever value is smaller isplaced into arrayC[indexC]
1 lt 2 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1index
C
arrayA
2 15 27 38index
B
arrayB
arrayC
2 lt 13 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
13 lt 15 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
15 lt 24 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13 15indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
24 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
indexA
1 2 13 15 24indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
26 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
1 2 13 15 24 26indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
Since we have exhaustedone of the arrays arrayAwe simply copy the remaining items from theother array arrayB intoarrayC
Merge Sort
1 13 24
26
1 2 13 15 24 26 27 38
arrayA
2 15 27 38arrayB
arrayC
Efficiency Summary
Sort Worst Case Average Case
Insertion O(n2) O(n2)
Shell O(n15) O(n125)
Selection O(n2) O(n2)
Bubble O(n2) O(n2)
Quick O(n2) O(nlog2n)
Heap O(nlog2n) O(nlog2n)
Merge O(nlog2n) O(nlog2n)
More Sorting
radix sort bucket sort in-place sorting how fast can we sort
Radix Sort
Unlike other sorting methods radix sort considers the structure of the keys
Assume keys are represented in a base M number system (M = radix) ie if M = 2 the keys are represented in binary
Sorting is done by comparing bits in the same position
Extension to keys that are alphanumeric strings
Radix Exchange Sort
Examine bits from left to right1 Sort array with respect to leftmost bit
2 Partition array
3Recursionrecursively sort top subarray ignoring leftmost bitrecursively sort bottom subarray ignoring leftmost bit
Time to sort n b-bit numbers O(b n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Types of Sorts We now discuss several sorting
methods Insertion Sort Shell Sort Selection Sort Bubble Sort Quick Sort Heap Sort Merge Sort
Insertion Sort In each pass of an insertion sort
one or more pieces of data are inserted into their correct location in an ordered list (just as a card player picks up cards and places them in his hand in order)
Insertion Sort In the insertion sort the list is divided into
2 parts Sorted Unsorted
In each pass the first element of the unsorted sublist is transferred to the sorted sublist by inserting it at the appropriate place
If we have a list of n elements it will take at most n-1 passes to sort the data
Insertion Sort
We can visualize this type of sort with the above figure
The first part of the list is the sorted portion which is separated by a ldquoconceptualrdquo wall from the unsorted portion of the list
Insertion SortHere we start with an unsorted list We leave the first data element alone and will start with the 2nd element of the list (the 1st element of the unsorted list)
On our first pass we look at the 2nd element of the list (thefirst element of the unsorted list) and place it in our sorted list in order
On our next pass we look at the 3rd element of the list (the1st element of the unsorted list) and place it in the sorted list in order
We continue in this fashion until the entire list has beensorted
Insertion Sort Example
In our first pass we compare the first 2 values Because theyare out of order we swap them
Now we look at the next 2 values Again they are out oforder so we swap them
Since we have swapped those values we need to comparethe previous 2 values to make sure that they are still in orderSince they are out of order we swap them and then continueon with the next 2 data valuesThese 2 values are out of order so we swap them and look atthe previous 2 values etc
Shell Sort Named after its creator Donald Shell the
shell sort is an improved version of the insertion sort
In the shell sort a list of N elements is divided into K segments where K is known as the increment
What this means is that instead of comparing adjacent values we will compare values that are a distance K apart
We will shrink K as we run through our algorithm
Shell SortPass Notes
1 77 62 14 9 30 21 80 25 70 55 Swap2 21 62 14 9 30 77 80 25 70 55 In order3 21 62 14 9 30 77 80 25 70 55 In order4 21 62 14 9 30 77 80 25 70 55 In order5 21 62 14 9 30 77 80 25 70 55 In order
List (K=5)
Just as in the straight insertion sort we compare 2 values andswap them if they are out of order However in the shell sort we compare values that are a distance K apartOnce we have completed going through the elements in our
list with K=5 we decrease K and continue the process
Shell SortPass Notes
6 21 62 14 9 30 77 80 25 70 55 Swap7 14 62 21 9 30 77 80 25 70 55 Swap8 14 9 21 62 30 77 80 25 70 55 In order9 14 9 21 62 30 77 80 25 70 55 In order10 14 9 21 62 30 77 80 25 70 55 In order11 14 9 21 62 30 77 80 25 70 55 Swap
14 9 21 62 30 25 80 77 70 55 Swap14 9 21 25 30 62 80 77 70 55 In order
12 14 9 21 25 30 62 80 77 70 55 Swap14 9 21 25 30 62 70 77 80 55 In order
13 14 9 21 25 30 62 70 77 80 55 Swap14 9 21 25 30 62 70 55 80 77 Swap14 9 21 25 30 55 70 62 80 77 In order
List (K=2)
Here we have reduced K to 2 Just as in the insertion sort if we swap 2 values we have to go back and compare theprevious 2 values to make sure they are still in order
Shell Sort
All shell sorts will terminate by running an insertion sort(ie K=1) However using the larger values of K first has helped to sort our list so that the straight insertion sort will runfaster
Shell Sort There are many schools of thought
on what the increment should be in the shell sort
Also note that just because an increment is optimal on one list it might not be optimal for another list
Insertion Sort vs Shell Sort Comparing the Big-O notation (for the
average case) we find that Insertion O(n2) Shell O(n125) empirically determined
Although this doesnrsquot seem like much of a gain it makes a big difference as n gets large
Note that in the worst case the Shell sort has an efficiency of O(n2)
However using a special incrementing technique this worst case can be reduced to O(n15)
Insertion Sort vs Shell Sort(Average Case)
Straight Insertion Shell25 625 55100 10000 316500 250000 2364
1000 1000000 56232000 4000000 13374
Number of Loopsn
Selection Sort Imagine some data that you can examine
all at once To sort it you could select the smallest
element and put it in its place select the next smallest and put it in its place etc
For a card player this process is analogous to looking at an entire hand of cards and ordering them by selecting cards one at a time and placing them in their proper order
Selection Sort The selection sort follows this idea Given a list of data to be sorted
we simply select the smallest item and place it in a sorted list
We then repeat these steps until the list is sorted
Selection Sort In the selection sort the list at any moment
is divided into 2 sublists sorted and unsorted separated by a ldquoconceptualrdquo wall
We select the smallest element from the unsorted sublist and exchange it with the element at the beginning of the unsorted data
After each selection and exchange the wall between the 2 sublists moves ndash increasing the number of sorted elements and decreasing the number of unsorted elements
Selection Sort
We start with an unsorted list We search this list for thesmallest element We then exchange the smallest element (8)with the first element in the unsorted list (23)
Again we search the unsorted list for the smallest element We then exchange the smallest element (23) with the first element in the unsorted list (78)
This process continues until the list is fully sorted
Bubble Sort In the bubble sort the list at any
moment is divided into 2 sublists sorted and unsorted
The smallest element is ldquobubbledrdquo from the unsorted sublist to the sorted sublist
Bubble Sort
23 78 45 8 56 3232 56
We start with 32 and compareit with 56 Because 32 is lessthan 56 we swap the twoand step down one element
We then compare 32 and 8Because 32 is not less than 8we do not swap these elements
We step down one element andcompare 45 and 8 They areout of sequence so we swapthem and step down again
8 45
We step down again and compare 8 with 78 These two elements are swapped
8 78
Finally 8 is compared with 23and swapped We thencontinue this process back with56 hellip
8 23
1 Pass of the Bubble Sort
Quick Sort In the bubble sort consecutive items
are compared and possibly exchanged on each pass through the list
This means that many exchanges may be needed to move an element to its correct position
Quick sort is more efficient than bubble sort because a typical exchange involves elements that are far apart so fewer exchanges are required to correctly position an element
Idea of Quick Sort1) Select pick an element
2) Divide rearrange elements so that x goes to its final position E
3) Recurse and Conquer recursively sort
Quick Sort Each iteration of the quick sort
selects an element known as the pivot and divides the list into 3 groups Elements whose keys are less than (or
equal to) the pivotrsquos key The pivot element Elements whose keys are greater than
(or equal to) the pivotrsquos key
Quick Sort The sorting then continues by
quick sorting the left partition followed by quick sorting the right partition
The basic algorithm is as follows
Quick Sort1) Partitioning Step Take an element in the
unsorted array and determine its final location in the sorted array This occurs when all values to the left of the element in the array are less than (or equal to) the element and all values to the right of the element are greater than (or equal to) the element We now have 1 element in its proper location and two unsorted subarrays
2) Recursive Step Perform step 1 on each unsorted subarray
Quick Sort Each time step 1 is performed on a
subarray another element is placed in its final location of the sorted array and two unsorted subarrays are created
When a subarray consists of one element that subarray is sorted
Therefore that element is in its final location
Quick Sort There are several partitioning strategies
used in practice (ie several ldquoversionsrdquo of quick sort) but the one we are about to describe is known to work well
For simplicity we will choose the last element to be the pivot element
We could also chose a different pivot element and swap it with the last element in the array
Quick Sort Below is the array we would like to
sort1 4 8 9 0 11 5 10 7 6
Quick Sort The index left starts at the first element
and right starts at the next-to-last element
We want to move all the elements smaller than the pivot to the left part of the array and all the elements larger than the pivot to the right part
1 4 8 9 0 11 5 10 7 6left righ
t
Quick Sort We move left to the right skipping
over elements that are smaller than the pivot
1 4 8 9 0 11 5 10 7 6left righ
t
Quick Sort We then move right to the left skipping
over elements that are greater than the pivot
When left and right have stopped left is on an element greater than (or equal to) the pivot and right is on an element smaller than (or equal to) the pivot
1 4 8 9 0 11 5 10 7 6left right
Quick Sort If left is to the left of right (or if left
= right) those elements are swapped
1 4 8 9 0 11 5 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
Quick Sort The effect is to push a large
element to the right and a small element to the left
We then repeat the process until left and right cross
Quick Sort
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6left righ
t
Quick Sort
1 4 5 0 9 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6righ
tleft
Quick Sort
At this point left and right have crossed so no swap is performed
The final part of the partitioning is to swap the pivot element with left
1 4 5 0 9 11 8 10 7 6righ
tleft
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort Note that all elements to the left of
the pivot are less than (or equal to) the pivot and all elements to the right of the pivot are greater than (or equal to) the pivot
Hence the pivot element has been placed in its final sorted position
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort We now repeat the process using
the sub-arrays to the left and right of the pivot
1 4 5 0 6 11 8 10 7 9
1 4 5 0 11 8 10 7 9
Analysis of quicksortmdashbest case
Suppose each partition operation divides the array almost exactly in half
Then the depth of the recursion in log2n Because thatrsquos how many times we can
halve n However there are many recursions at
each level How can we figure this out We note that
Each partition is linear over its subarray All the partitions at one level cover the array
Partitioning at various levels
Best case II We cut the array size in half each time So the depth of the recursion in log2n At each level of the recursion all the
partitions at that level do work that is linear in n
O(log2n) O(n) = O(n log2n)
What about the worst case
Worst case In the worst case partitioning always
divides the size n array into these three parts A length one part containing the pivot itself A length zero part and A length n-1 part containing everything else
We donrsquot recur on the zero-length part Recurring on the length n-1 part requires
(in the worst case) recurring to depth n-1
Worst case partitioning
Worst case for quicksort In the worst case recursion may be
O(n) levels deep (for an array of size n)
But the partitioning work done at each level is still O(n)
O(n) O(n) = O(n2) So worst case for Quicksort is O(n2) When does this happen
When the array is sorted to begin with
Typical case for quicksort If the array is sorted to begin with
Quicksort is terrible O(n2) It is possible to construct other bad
cases However Quicksort is usually O(n log2n) The constants are so good that Quicksort
is generally the fastest algorithm known Most real-world sorting is done by
Quicksort
Tweaking Quicksort Almost anything you can try to
ldquoimproverdquo Quicksort will actually slow it down
One good tweak is to switch to a different sorting method when the subarrays get small (say 10 or 12) Quicksort has too much overhead for small
array sizes For large arrays it might be a good
idea to check beforehand if the array is already sorted But there is a better tweak than this
Randomized Quick-Sort Select the pivot as a random element of the sequence The expected running time of randomized quick-sort on a
sequence of size n is O(n log n) The time spent at a level of the quick-sort tree is O(n) We show that the expected height of the quick-sort tree is
O(log n) good vs bad pivots
bull The probability of a good pivot is 12 thus we expect k2 good pivots out of k pivotsbull After a good pivot the size of each child sequence is at most 34 the size of the parent sequencebull After h pivots we expect (34)h2 n elementsbull The expected height h of the quick-sort tree is at most 2 log43n
Median of three Obviously it doesnrsquot make sense to sort the
array in order to find the median to use as a pivot
Instead compare just three elements of our (sub)arraymdashthe first the last and the middle Take the median (middle value) of these three as
pivot Itrsquos possible (but not easy) to construct cases
which will make this technique O(n2) Suppose we rearrange (sort) these three
numbers so that the smallest is in the first position the largest in the last position and the other in the middle This lets us simplify and speed up the partition
loop
Final comments Weissrsquos code shows some additional
optimizations on pp 246-247 Weiss chooses to stop both searches on
equality to pivot This design decision is debatable
Quicksort is the fastest known general sorting algorithm on average
For optimum speed the pivot must be chosen carefully
ldquoMedian of threerdquo is a good technique for choosing the pivot
There will be some cases where Quicksort runs in O(n2) time
Quick Sort A couple of notes about quick sort There are more optimal ways to
choose the pivot value (such as the median-of-three method)
Also when the subarrays get small it becomes more efficient to use the insertion sort as opposed to continued use of quick sort
Bubble Sort vs Quick Sort If we calculate the Big-O notation
we find that (in the average case) Bubble Sort O(n2) Quick Sort O(nlog2n)
(quick-sort runs in time O(n2) in the worst case)
Heap Sort Idea take the items that need to be
sorted and insert them into the heap By calling deleteHeap we remove the
smallest or largest element depending on whether or not we are working with a min- or max-heap respectively
Hence the elements are removed in ascending or descending order
Efficiency O(nlog2n)
Merge Sort Idea
Take the array you would like to sort and divide it in half to create 2 unsorted subarrays
Next sort each of the 2 subarrays Finally merge the 2 sorted subarrays
into 1 sorted array Efficiency O(nlog2n)
Merge Sort
Merge Sort Although the merge step produces
a sorted array we have overlooked a very important step
How did we sort the 2 halves before performing the merge step
We used merge sort
Merge Sort By continually calling the merge
sort algorithm we eventually get a subarray of size 1
Since an array with only 1 element is clearly sorted we can back out and merge 2 arrays of size 1
Merge Sort
Merge Sort The basic merging algorithm
consists of 2 input arrays (arrayA and arrayB) An ouput array (arrayC) 3 position holders (indexA indexB
indexC) which are initially set to the beginning of their respective arrays
Merge Sort The smaller of arrayA[indexA] and
arrayB[indexB] is copied into arrayC[indexC] and the appropriate position holders are advanced
When either input list is exhausted the remainder of the other list is copied into arrayC
Merge Sort
1 13 24 26index
A
indexC
arrayA
2 15 27 38index
B
arrayB
arrayC
We compare arrayA[indexA]with arrayB[indexB] Whichever value is smaller isplaced into arrayC[indexC]
1 lt 2 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1index
C
arrayA
2 15 27 38index
B
arrayB
arrayC
2 lt 13 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
13 lt 15 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
15 lt 24 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13 15indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
24 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
indexA
1 2 13 15 24indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
26 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
1 2 13 15 24 26indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
Since we have exhaustedone of the arrays arrayAwe simply copy the remaining items from theother array arrayB intoarrayC
Merge Sort
1 13 24
26
1 2 13 15 24 26 27 38
arrayA
2 15 27 38arrayB
arrayC
Efficiency Summary
Sort Worst Case Average Case
Insertion O(n2) O(n2)
Shell O(n15) O(n125)
Selection O(n2) O(n2)
Bubble O(n2) O(n2)
Quick O(n2) O(nlog2n)
Heap O(nlog2n) O(nlog2n)
Merge O(nlog2n) O(nlog2n)
More Sorting
radix sort bucket sort in-place sorting how fast can we sort
Radix Sort
Unlike other sorting methods radix sort considers the structure of the keys
Assume keys are represented in a base M number system (M = radix) ie if M = 2 the keys are represented in binary
Sorting is done by comparing bits in the same position
Extension to keys that are alphanumeric strings
Radix Exchange Sort
Examine bits from left to right1 Sort array with respect to leftmost bit
2 Partition array
3Recursionrecursively sort top subarray ignoring leftmost bitrecursively sort bottom subarray ignoring leftmost bit
Time to sort n b-bit numbers O(b n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Insertion Sort In each pass of an insertion sort
one or more pieces of data are inserted into their correct location in an ordered list (just as a card player picks up cards and places them in his hand in order)
Insertion Sort In the insertion sort the list is divided into
2 parts Sorted Unsorted
In each pass the first element of the unsorted sublist is transferred to the sorted sublist by inserting it at the appropriate place
If we have a list of n elements it will take at most n-1 passes to sort the data
Insertion Sort
We can visualize this type of sort with the above figure
The first part of the list is the sorted portion which is separated by a ldquoconceptualrdquo wall from the unsorted portion of the list
Insertion SortHere we start with an unsorted list We leave the first data element alone and will start with the 2nd element of the list (the 1st element of the unsorted list)
On our first pass we look at the 2nd element of the list (thefirst element of the unsorted list) and place it in our sorted list in order
On our next pass we look at the 3rd element of the list (the1st element of the unsorted list) and place it in the sorted list in order
We continue in this fashion until the entire list has beensorted
Insertion Sort Example
In our first pass we compare the first 2 values Because theyare out of order we swap them
Now we look at the next 2 values Again they are out oforder so we swap them
Since we have swapped those values we need to comparethe previous 2 values to make sure that they are still in orderSince they are out of order we swap them and then continueon with the next 2 data valuesThese 2 values are out of order so we swap them and look atthe previous 2 values etc
Shell Sort Named after its creator Donald Shell the
shell sort is an improved version of the insertion sort
In the shell sort a list of N elements is divided into K segments where K is known as the increment
What this means is that instead of comparing adjacent values we will compare values that are a distance K apart
We will shrink K as we run through our algorithm
Shell SortPass Notes
1 77 62 14 9 30 21 80 25 70 55 Swap2 21 62 14 9 30 77 80 25 70 55 In order3 21 62 14 9 30 77 80 25 70 55 In order4 21 62 14 9 30 77 80 25 70 55 In order5 21 62 14 9 30 77 80 25 70 55 In order
List (K=5)
Just as in the straight insertion sort we compare 2 values andswap them if they are out of order However in the shell sort we compare values that are a distance K apartOnce we have completed going through the elements in our
list with K=5 we decrease K and continue the process
Shell SortPass Notes
6 21 62 14 9 30 77 80 25 70 55 Swap7 14 62 21 9 30 77 80 25 70 55 Swap8 14 9 21 62 30 77 80 25 70 55 In order9 14 9 21 62 30 77 80 25 70 55 In order10 14 9 21 62 30 77 80 25 70 55 In order11 14 9 21 62 30 77 80 25 70 55 Swap
14 9 21 62 30 25 80 77 70 55 Swap14 9 21 25 30 62 80 77 70 55 In order
12 14 9 21 25 30 62 80 77 70 55 Swap14 9 21 25 30 62 70 77 80 55 In order
13 14 9 21 25 30 62 70 77 80 55 Swap14 9 21 25 30 62 70 55 80 77 Swap14 9 21 25 30 55 70 62 80 77 In order
List (K=2)
Here we have reduced K to 2 Just as in the insertion sort if we swap 2 values we have to go back and compare theprevious 2 values to make sure they are still in order
Shell Sort
All shell sorts will terminate by running an insertion sort(ie K=1) However using the larger values of K first has helped to sort our list so that the straight insertion sort will runfaster
Shell Sort There are many schools of thought
on what the increment should be in the shell sort
Also note that just because an increment is optimal on one list it might not be optimal for another list
Insertion Sort vs Shell Sort Comparing the Big-O notation (for the
average case) we find that Insertion O(n2) Shell O(n125) empirically determined
Although this doesnrsquot seem like much of a gain it makes a big difference as n gets large
Note that in the worst case the Shell sort has an efficiency of O(n2)
However using a special incrementing technique this worst case can be reduced to O(n15)
Insertion Sort vs Shell Sort(Average Case)
Straight Insertion Shell25 625 55100 10000 316500 250000 2364
1000 1000000 56232000 4000000 13374
Number of Loopsn
Selection Sort Imagine some data that you can examine
all at once To sort it you could select the smallest
element and put it in its place select the next smallest and put it in its place etc
For a card player this process is analogous to looking at an entire hand of cards and ordering them by selecting cards one at a time and placing them in their proper order
Selection Sort The selection sort follows this idea Given a list of data to be sorted
we simply select the smallest item and place it in a sorted list
We then repeat these steps until the list is sorted
Selection Sort In the selection sort the list at any moment
is divided into 2 sublists sorted and unsorted separated by a ldquoconceptualrdquo wall
We select the smallest element from the unsorted sublist and exchange it with the element at the beginning of the unsorted data
After each selection and exchange the wall between the 2 sublists moves ndash increasing the number of sorted elements and decreasing the number of unsorted elements
Selection Sort
We start with an unsorted list We search this list for thesmallest element We then exchange the smallest element (8)with the first element in the unsorted list (23)
Again we search the unsorted list for the smallest element We then exchange the smallest element (23) with the first element in the unsorted list (78)
This process continues until the list is fully sorted
Bubble Sort In the bubble sort the list at any
moment is divided into 2 sublists sorted and unsorted
The smallest element is ldquobubbledrdquo from the unsorted sublist to the sorted sublist
Bubble Sort
23 78 45 8 56 3232 56
We start with 32 and compareit with 56 Because 32 is lessthan 56 we swap the twoand step down one element
We then compare 32 and 8Because 32 is not less than 8we do not swap these elements
We step down one element andcompare 45 and 8 They areout of sequence so we swapthem and step down again
8 45
We step down again and compare 8 with 78 These two elements are swapped
8 78
Finally 8 is compared with 23and swapped We thencontinue this process back with56 hellip
8 23
1 Pass of the Bubble Sort
Quick Sort In the bubble sort consecutive items
are compared and possibly exchanged on each pass through the list
This means that many exchanges may be needed to move an element to its correct position
Quick sort is more efficient than bubble sort because a typical exchange involves elements that are far apart so fewer exchanges are required to correctly position an element
Idea of Quick Sort1) Select pick an element
2) Divide rearrange elements so that x goes to its final position E
3) Recurse and Conquer recursively sort
Quick Sort Each iteration of the quick sort
selects an element known as the pivot and divides the list into 3 groups Elements whose keys are less than (or
equal to) the pivotrsquos key The pivot element Elements whose keys are greater than
(or equal to) the pivotrsquos key
Quick Sort The sorting then continues by
quick sorting the left partition followed by quick sorting the right partition
The basic algorithm is as follows
Quick Sort1) Partitioning Step Take an element in the
unsorted array and determine its final location in the sorted array This occurs when all values to the left of the element in the array are less than (or equal to) the element and all values to the right of the element are greater than (or equal to) the element We now have 1 element in its proper location and two unsorted subarrays
2) Recursive Step Perform step 1 on each unsorted subarray
Quick Sort Each time step 1 is performed on a
subarray another element is placed in its final location of the sorted array and two unsorted subarrays are created
When a subarray consists of one element that subarray is sorted
Therefore that element is in its final location
Quick Sort There are several partitioning strategies
used in practice (ie several ldquoversionsrdquo of quick sort) but the one we are about to describe is known to work well
For simplicity we will choose the last element to be the pivot element
We could also chose a different pivot element and swap it with the last element in the array
Quick Sort Below is the array we would like to
sort1 4 8 9 0 11 5 10 7 6
Quick Sort The index left starts at the first element
and right starts at the next-to-last element
We want to move all the elements smaller than the pivot to the left part of the array and all the elements larger than the pivot to the right part
1 4 8 9 0 11 5 10 7 6left righ
t
Quick Sort We move left to the right skipping
over elements that are smaller than the pivot
1 4 8 9 0 11 5 10 7 6left righ
t
Quick Sort We then move right to the left skipping
over elements that are greater than the pivot
When left and right have stopped left is on an element greater than (or equal to) the pivot and right is on an element smaller than (or equal to) the pivot
1 4 8 9 0 11 5 10 7 6left right
Quick Sort If left is to the left of right (or if left
= right) those elements are swapped
1 4 8 9 0 11 5 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
Quick Sort The effect is to push a large
element to the right and a small element to the left
We then repeat the process until left and right cross
Quick Sort
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6left righ
t
Quick Sort
1 4 5 0 9 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6righ
tleft
Quick Sort
At this point left and right have crossed so no swap is performed
The final part of the partitioning is to swap the pivot element with left
1 4 5 0 9 11 8 10 7 6righ
tleft
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort Note that all elements to the left of
the pivot are less than (or equal to) the pivot and all elements to the right of the pivot are greater than (or equal to) the pivot
Hence the pivot element has been placed in its final sorted position
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort We now repeat the process using
the sub-arrays to the left and right of the pivot
1 4 5 0 6 11 8 10 7 9
1 4 5 0 11 8 10 7 9
Analysis of quicksortmdashbest case
Suppose each partition operation divides the array almost exactly in half
Then the depth of the recursion in log2n Because thatrsquos how many times we can
halve n However there are many recursions at
each level How can we figure this out We note that
Each partition is linear over its subarray All the partitions at one level cover the array
Partitioning at various levels
Best case II We cut the array size in half each time So the depth of the recursion in log2n At each level of the recursion all the
partitions at that level do work that is linear in n
O(log2n) O(n) = O(n log2n)
What about the worst case
Worst case In the worst case partitioning always
divides the size n array into these three parts A length one part containing the pivot itself A length zero part and A length n-1 part containing everything else
We donrsquot recur on the zero-length part Recurring on the length n-1 part requires
(in the worst case) recurring to depth n-1
Worst case partitioning
Worst case for quicksort In the worst case recursion may be
O(n) levels deep (for an array of size n)
But the partitioning work done at each level is still O(n)
O(n) O(n) = O(n2) So worst case for Quicksort is O(n2) When does this happen
When the array is sorted to begin with
Typical case for quicksort If the array is sorted to begin with
Quicksort is terrible O(n2) It is possible to construct other bad
cases However Quicksort is usually O(n log2n) The constants are so good that Quicksort
is generally the fastest algorithm known Most real-world sorting is done by
Quicksort
Tweaking Quicksort Almost anything you can try to
ldquoimproverdquo Quicksort will actually slow it down
One good tweak is to switch to a different sorting method when the subarrays get small (say 10 or 12) Quicksort has too much overhead for small
array sizes For large arrays it might be a good
idea to check beforehand if the array is already sorted But there is a better tweak than this
Randomized Quick-Sort Select the pivot as a random element of the sequence The expected running time of randomized quick-sort on a
sequence of size n is O(n log n) The time spent at a level of the quick-sort tree is O(n) We show that the expected height of the quick-sort tree is
O(log n) good vs bad pivots
bull The probability of a good pivot is 12 thus we expect k2 good pivots out of k pivotsbull After a good pivot the size of each child sequence is at most 34 the size of the parent sequencebull After h pivots we expect (34)h2 n elementsbull The expected height h of the quick-sort tree is at most 2 log43n
Median of three Obviously it doesnrsquot make sense to sort the
array in order to find the median to use as a pivot
Instead compare just three elements of our (sub)arraymdashthe first the last and the middle Take the median (middle value) of these three as
pivot Itrsquos possible (but not easy) to construct cases
which will make this technique O(n2) Suppose we rearrange (sort) these three
numbers so that the smallest is in the first position the largest in the last position and the other in the middle This lets us simplify and speed up the partition
loop
Final comments Weissrsquos code shows some additional
optimizations on pp 246-247 Weiss chooses to stop both searches on
equality to pivot This design decision is debatable
Quicksort is the fastest known general sorting algorithm on average
For optimum speed the pivot must be chosen carefully
ldquoMedian of threerdquo is a good technique for choosing the pivot
There will be some cases where Quicksort runs in O(n2) time
Quick Sort A couple of notes about quick sort There are more optimal ways to
choose the pivot value (such as the median-of-three method)
Also when the subarrays get small it becomes more efficient to use the insertion sort as opposed to continued use of quick sort
Bubble Sort vs Quick Sort If we calculate the Big-O notation
we find that (in the average case) Bubble Sort O(n2) Quick Sort O(nlog2n)
(quick-sort runs in time O(n2) in the worst case)
Heap Sort Idea take the items that need to be
sorted and insert them into the heap By calling deleteHeap we remove the
smallest or largest element depending on whether or not we are working with a min- or max-heap respectively
Hence the elements are removed in ascending or descending order
Efficiency O(nlog2n)
Merge Sort Idea
Take the array you would like to sort and divide it in half to create 2 unsorted subarrays
Next sort each of the 2 subarrays Finally merge the 2 sorted subarrays
into 1 sorted array Efficiency O(nlog2n)
Merge Sort
Merge Sort Although the merge step produces
a sorted array we have overlooked a very important step
How did we sort the 2 halves before performing the merge step
We used merge sort
Merge Sort By continually calling the merge
sort algorithm we eventually get a subarray of size 1
Since an array with only 1 element is clearly sorted we can back out and merge 2 arrays of size 1
Merge Sort
Merge Sort The basic merging algorithm
consists of 2 input arrays (arrayA and arrayB) An ouput array (arrayC) 3 position holders (indexA indexB
indexC) which are initially set to the beginning of their respective arrays
Merge Sort The smaller of arrayA[indexA] and
arrayB[indexB] is copied into arrayC[indexC] and the appropriate position holders are advanced
When either input list is exhausted the remainder of the other list is copied into arrayC
Merge Sort
1 13 24 26index
A
indexC
arrayA
2 15 27 38index
B
arrayB
arrayC
We compare arrayA[indexA]with arrayB[indexB] Whichever value is smaller isplaced into arrayC[indexC]
1 lt 2 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1index
C
arrayA
2 15 27 38index
B
arrayB
arrayC
2 lt 13 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
13 lt 15 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
15 lt 24 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13 15indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
24 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
indexA
1 2 13 15 24indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
26 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
1 2 13 15 24 26indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
Since we have exhaustedone of the arrays arrayAwe simply copy the remaining items from theother array arrayB intoarrayC
Merge Sort
1 13 24
26
1 2 13 15 24 26 27 38
arrayA
2 15 27 38arrayB
arrayC
Efficiency Summary
Sort Worst Case Average Case
Insertion O(n2) O(n2)
Shell O(n15) O(n125)
Selection O(n2) O(n2)
Bubble O(n2) O(n2)
Quick O(n2) O(nlog2n)
Heap O(nlog2n) O(nlog2n)
Merge O(nlog2n) O(nlog2n)
More Sorting
radix sort bucket sort in-place sorting how fast can we sort
Radix Sort
Unlike other sorting methods radix sort considers the structure of the keys
Assume keys are represented in a base M number system (M = radix) ie if M = 2 the keys are represented in binary
Sorting is done by comparing bits in the same position
Extension to keys that are alphanumeric strings
Radix Exchange Sort
Examine bits from left to right1 Sort array with respect to leftmost bit
2 Partition array
3Recursionrecursively sort top subarray ignoring leftmost bitrecursively sort bottom subarray ignoring leftmost bit
Time to sort n b-bit numbers O(b n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Insertion Sort In the insertion sort the list is divided into
2 parts Sorted Unsorted
In each pass the first element of the unsorted sublist is transferred to the sorted sublist by inserting it at the appropriate place
If we have a list of n elements it will take at most n-1 passes to sort the data
Insertion Sort
We can visualize this type of sort with the above figure
The first part of the list is the sorted portion which is separated by a ldquoconceptualrdquo wall from the unsorted portion of the list
Insertion SortHere we start with an unsorted list We leave the first data element alone and will start with the 2nd element of the list (the 1st element of the unsorted list)
On our first pass we look at the 2nd element of the list (thefirst element of the unsorted list) and place it in our sorted list in order
On our next pass we look at the 3rd element of the list (the1st element of the unsorted list) and place it in the sorted list in order
We continue in this fashion until the entire list has beensorted
Insertion Sort Example
In our first pass we compare the first 2 values Because theyare out of order we swap them
Now we look at the next 2 values Again they are out oforder so we swap them
Since we have swapped those values we need to comparethe previous 2 values to make sure that they are still in orderSince they are out of order we swap them and then continueon with the next 2 data valuesThese 2 values are out of order so we swap them and look atthe previous 2 values etc
Shell Sort Named after its creator Donald Shell the
shell sort is an improved version of the insertion sort
In the shell sort a list of N elements is divided into K segments where K is known as the increment
What this means is that instead of comparing adjacent values we will compare values that are a distance K apart
We will shrink K as we run through our algorithm
Shell SortPass Notes
1 77 62 14 9 30 21 80 25 70 55 Swap2 21 62 14 9 30 77 80 25 70 55 In order3 21 62 14 9 30 77 80 25 70 55 In order4 21 62 14 9 30 77 80 25 70 55 In order5 21 62 14 9 30 77 80 25 70 55 In order
List (K=5)
Just as in the straight insertion sort we compare 2 values andswap them if they are out of order However in the shell sort we compare values that are a distance K apartOnce we have completed going through the elements in our
list with K=5 we decrease K and continue the process
Shell SortPass Notes
6 21 62 14 9 30 77 80 25 70 55 Swap7 14 62 21 9 30 77 80 25 70 55 Swap8 14 9 21 62 30 77 80 25 70 55 In order9 14 9 21 62 30 77 80 25 70 55 In order10 14 9 21 62 30 77 80 25 70 55 In order11 14 9 21 62 30 77 80 25 70 55 Swap
14 9 21 62 30 25 80 77 70 55 Swap14 9 21 25 30 62 80 77 70 55 In order
12 14 9 21 25 30 62 80 77 70 55 Swap14 9 21 25 30 62 70 77 80 55 In order
13 14 9 21 25 30 62 70 77 80 55 Swap14 9 21 25 30 62 70 55 80 77 Swap14 9 21 25 30 55 70 62 80 77 In order
List (K=2)
Here we have reduced K to 2 Just as in the insertion sort if we swap 2 values we have to go back and compare theprevious 2 values to make sure they are still in order
Shell Sort
All shell sorts will terminate by running an insertion sort(ie K=1) However using the larger values of K first has helped to sort our list so that the straight insertion sort will runfaster
Shell Sort There are many schools of thought
on what the increment should be in the shell sort
Also note that just because an increment is optimal on one list it might not be optimal for another list
Insertion Sort vs Shell Sort Comparing the Big-O notation (for the
average case) we find that Insertion O(n2) Shell O(n125) empirically determined
Although this doesnrsquot seem like much of a gain it makes a big difference as n gets large
Note that in the worst case the Shell sort has an efficiency of O(n2)
However using a special incrementing technique this worst case can be reduced to O(n15)
Insertion Sort vs Shell Sort(Average Case)
Straight Insertion Shell25 625 55100 10000 316500 250000 2364
1000 1000000 56232000 4000000 13374
Number of Loopsn
Selection Sort Imagine some data that you can examine
all at once To sort it you could select the smallest
element and put it in its place select the next smallest and put it in its place etc
For a card player this process is analogous to looking at an entire hand of cards and ordering them by selecting cards one at a time and placing them in their proper order
Selection Sort The selection sort follows this idea Given a list of data to be sorted
we simply select the smallest item and place it in a sorted list
We then repeat these steps until the list is sorted
Selection Sort In the selection sort the list at any moment
is divided into 2 sublists sorted and unsorted separated by a ldquoconceptualrdquo wall
We select the smallest element from the unsorted sublist and exchange it with the element at the beginning of the unsorted data
After each selection and exchange the wall between the 2 sublists moves ndash increasing the number of sorted elements and decreasing the number of unsorted elements
Selection Sort
We start with an unsorted list We search this list for thesmallest element We then exchange the smallest element (8)with the first element in the unsorted list (23)
Again we search the unsorted list for the smallest element We then exchange the smallest element (23) with the first element in the unsorted list (78)
This process continues until the list is fully sorted
Bubble Sort In the bubble sort the list at any
moment is divided into 2 sublists sorted and unsorted
The smallest element is ldquobubbledrdquo from the unsorted sublist to the sorted sublist
Bubble Sort
23 78 45 8 56 3232 56
We start with 32 and compareit with 56 Because 32 is lessthan 56 we swap the twoand step down one element
We then compare 32 and 8Because 32 is not less than 8we do not swap these elements
We step down one element andcompare 45 and 8 They areout of sequence so we swapthem and step down again
8 45
We step down again and compare 8 with 78 These two elements are swapped
8 78
Finally 8 is compared with 23and swapped We thencontinue this process back with56 hellip
8 23
1 Pass of the Bubble Sort
Quick Sort In the bubble sort consecutive items
are compared and possibly exchanged on each pass through the list
This means that many exchanges may be needed to move an element to its correct position
Quick sort is more efficient than bubble sort because a typical exchange involves elements that are far apart so fewer exchanges are required to correctly position an element
Idea of Quick Sort1) Select pick an element
2) Divide rearrange elements so that x goes to its final position E
3) Recurse and Conquer recursively sort
Quick Sort Each iteration of the quick sort
selects an element known as the pivot and divides the list into 3 groups Elements whose keys are less than (or
equal to) the pivotrsquos key The pivot element Elements whose keys are greater than
(or equal to) the pivotrsquos key
Quick Sort The sorting then continues by
quick sorting the left partition followed by quick sorting the right partition
The basic algorithm is as follows
Quick Sort1) Partitioning Step Take an element in the
unsorted array and determine its final location in the sorted array This occurs when all values to the left of the element in the array are less than (or equal to) the element and all values to the right of the element are greater than (or equal to) the element We now have 1 element in its proper location and two unsorted subarrays
2) Recursive Step Perform step 1 on each unsorted subarray
Quick Sort Each time step 1 is performed on a
subarray another element is placed in its final location of the sorted array and two unsorted subarrays are created
When a subarray consists of one element that subarray is sorted
Therefore that element is in its final location
Quick Sort There are several partitioning strategies
used in practice (ie several ldquoversionsrdquo of quick sort) but the one we are about to describe is known to work well
For simplicity we will choose the last element to be the pivot element
We could also chose a different pivot element and swap it with the last element in the array
Quick Sort Below is the array we would like to
sort1 4 8 9 0 11 5 10 7 6
Quick Sort The index left starts at the first element
and right starts at the next-to-last element
We want to move all the elements smaller than the pivot to the left part of the array and all the elements larger than the pivot to the right part
1 4 8 9 0 11 5 10 7 6left righ
t
Quick Sort We move left to the right skipping
over elements that are smaller than the pivot
1 4 8 9 0 11 5 10 7 6left righ
t
Quick Sort We then move right to the left skipping
over elements that are greater than the pivot
When left and right have stopped left is on an element greater than (or equal to) the pivot and right is on an element smaller than (or equal to) the pivot
1 4 8 9 0 11 5 10 7 6left right
Quick Sort If left is to the left of right (or if left
= right) those elements are swapped
1 4 8 9 0 11 5 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
Quick Sort The effect is to push a large
element to the right and a small element to the left
We then repeat the process until left and right cross
Quick Sort
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6left righ
t
Quick Sort
1 4 5 0 9 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6righ
tleft
Quick Sort
At this point left and right have crossed so no swap is performed
The final part of the partitioning is to swap the pivot element with left
1 4 5 0 9 11 8 10 7 6righ
tleft
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort Note that all elements to the left of
the pivot are less than (or equal to) the pivot and all elements to the right of the pivot are greater than (or equal to) the pivot
Hence the pivot element has been placed in its final sorted position
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort We now repeat the process using
the sub-arrays to the left and right of the pivot
1 4 5 0 6 11 8 10 7 9
1 4 5 0 11 8 10 7 9
Analysis of quicksortmdashbest case
Suppose each partition operation divides the array almost exactly in half
Then the depth of the recursion in log2n Because thatrsquos how many times we can
halve n However there are many recursions at
each level How can we figure this out We note that
Each partition is linear over its subarray All the partitions at one level cover the array
Partitioning at various levels
Best case II We cut the array size in half each time So the depth of the recursion in log2n At each level of the recursion all the
partitions at that level do work that is linear in n
O(log2n) O(n) = O(n log2n)
What about the worst case
Worst case In the worst case partitioning always
divides the size n array into these three parts A length one part containing the pivot itself A length zero part and A length n-1 part containing everything else
We donrsquot recur on the zero-length part Recurring on the length n-1 part requires
(in the worst case) recurring to depth n-1
Worst case partitioning
Worst case for quicksort In the worst case recursion may be
O(n) levels deep (for an array of size n)
But the partitioning work done at each level is still O(n)
O(n) O(n) = O(n2) So worst case for Quicksort is O(n2) When does this happen
When the array is sorted to begin with
Typical case for quicksort If the array is sorted to begin with
Quicksort is terrible O(n2) It is possible to construct other bad
cases However Quicksort is usually O(n log2n) The constants are so good that Quicksort
is generally the fastest algorithm known Most real-world sorting is done by
Quicksort
Tweaking Quicksort Almost anything you can try to
ldquoimproverdquo Quicksort will actually slow it down
One good tweak is to switch to a different sorting method when the subarrays get small (say 10 or 12) Quicksort has too much overhead for small
array sizes For large arrays it might be a good
idea to check beforehand if the array is already sorted But there is a better tweak than this
Randomized Quick-Sort Select the pivot as a random element of the sequence The expected running time of randomized quick-sort on a
sequence of size n is O(n log n) The time spent at a level of the quick-sort tree is O(n) We show that the expected height of the quick-sort tree is
O(log n) good vs bad pivots
bull The probability of a good pivot is 12 thus we expect k2 good pivots out of k pivotsbull After a good pivot the size of each child sequence is at most 34 the size of the parent sequencebull After h pivots we expect (34)h2 n elementsbull The expected height h of the quick-sort tree is at most 2 log43n
Median of three Obviously it doesnrsquot make sense to sort the
array in order to find the median to use as a pivot
Instead compare just three elements of our (sub)arraymdashthe first the last and the middle Take the median (middle value) of these three as
pivot Itrsquos possible (but not easy) to construct cases
which will make this technique O(n2) Suppose we rearrange (sort) these three
numbers so that the smallest is in the first position the largest in the last position and the other in the middle This lets us simplify and speed up the partition
loop
Final comments Weissrsquos code shows some additional
optimizations on pp 246-247 Weiss chooses to stop both searches on
equality to pivot This design decision is debatable
Quicksort is the fastest known general sorting algorithm on average
For optimum speed the pivot must be chosen carefully
ldquoMedian of threerdquo is a good technique for choosing the pivot
There will be some cases where Quicksort runs in O(n2) time
Quick Sort A couple of notes about quick sort There are more optimal ways to
choose the pivot value (such as the median-of-three method)
Also when the subarrays get small it becomes more efficient to use the insertion sort as opposed to continued use of quick sort
Bubble Sort vs Quick Sort If we calculate the Big-O notation
we find that (in the average case) Bubble Sort O(n2) Quick Sort O(nlog2n)
(quick-sort runs in time O(n2) in the worst case)
Heap Sort Idea take the items that need to be
sorted and insert them into the heap By calling deleteHeap we remove the
smallest or largest element depending on whether or not we are working with a min- or max-heap respectively
Hence the elements are removed in ascending or descending order
Efficiency O(nlog2n)
Merge Sort Idea
Take the array you would like to sort and divide it in half to create 2 unsorted subarrays
Next sort each of the 2 subarrays Finally merge the 2 sorted subarrays
into 1 sorted array Efficiency O(nlog2n)
Merge Sort
Merge Sort Although the merge step produces
a sorted array we have overlooked a very important step
How did we sort the 2 halves before performing the merge step
We used merge sort
Merge Sort By continually calling the merge
sort algorithm we eventually get a subarray of size 1
Since an array with only 1 element is clearly sorted we can back out and merge 2 arrays of size 1
Merge Sort
Merge Sort The basic merging algorithm
consists of 2 input arrays (arrayA and arrayB) An ouput array (arrayC) 3 position holders (indexA indexB
indexC) which are initially set to the beginning of their respective arrays
Merge Sort The smaller of arrayA[indexA] and
arrayB[indexB] is copied into arrayC[indexC] and the appropriate position holders are advanced
When either input list is exhausted the remainder of the other list is copied into arrayC
Merge Sort
1 13 24 26index
A
indexC
arrayA
2 15 27 38index
B
arrayB
arrayC
We compare arrayA[indexA]with arrayB[indexB] Whichever value is smaller isplaced into arrayC[indexC]
1 lt 2 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1index
C
arrayA
2 15 27 38index
B
arrayB
arrayC
2 lt 13 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
13 lt 15 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
15 lt 24 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13 15indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
24 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
indexA
1 2 13 15 24indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
26 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
1 2 13 15 24 26indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
Since we have exhaustedone of the arrays arrayAwe simply copy the remaining items from theother array arrayB intoarrayC
Merge Sort
1 13 24
26
1 2 13 15 24 26 27 38
arrayA
2 15 27 38arrayB
arrayC
Efficiency Summary
Sort Worst Case Average Case
Insertion O(n2) O(n2)
Shell O(n15) O(n125)
Selection O(n2) O(n2)
Bubble O(n2) O(n2)
Quick O(n2) O(nlog2n)
Heap O(nlog2n) O(nlog2n)
Merge O(nlog2n) O(nlog2n)
More Sorting
radix sort bucket sort in-place sorting how fast can we sort
Radix Sort
Unlike other sorting methods radix sort considers the structure of the keys
Assume keys are represented in a base M number system (M = radix) ie if M = 2 the keys are represented in binary
Sorting is done by comparing bits in the same position
Extension to keys that are alphanumeric strings
Radix Exchange Sort
Examine bits from left to right1 Sort array with respect to leftmost bit
2 Partition array
3Recursionrecursively sort top subarray ignoring leftmost bitrecursively sort bottom subarray ignoring leftmost bit
Time to sort n b-bit numbers O(b n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Insertion Sort
We can visualize this type of sort with the above figure
The first part of the list is the sorted portion which is separated by a ldquoconceptualrdquo wall from the unsorted portion of the list
Insertion SortHere we start with an unsorted list We leave the first data element alone and will start with the 2nd element of the list (the 1st element of the unsorted list)
On our first pass we look at the 2nd element of the list (thefirst element of the unsorted list) and place it in our sorted list in order
On our next pass we look at the 3rd element of the list (the1st element of the unsorted list) and place it in the sorted list in order
We continue in this fashion until the entire list has beensorted
Insertion Sort Example
In our first pass we compare the first 2 values Because theyare out of order we swap them
Now we look at the next 2 values Again they are out oforder so we swap them
Since we have swapped those values we need to comparethe previous 2 values to make sure that they are still in orderSince they are out of order we swap them and then continueon with the next 2 data valuesThese 2 values are out of order so we swap them and look atthe previous 2 values etc
Shell Sort Named after its creator Donald Shell the
shell sort is an improved version of the insertion sort
In the shell sort a list of N elements is divided into K segments where K is known as the increment
What this means is that instead of comparing adjacent values we will compare values that are a distance K apart
We will shrink K as we run through our algorithm
Shell SortPass Notes
1 77 62 14 9 30 21 80 25 70 55 Swap2 21 62 14 9 30 77 80 25 70 55 In order3 21 62 14 9 30 77 80 25 70 55 In order4 21 62 14 9 30 77 80 25 70 55 In order5 21 62 14 9 30 77 80 25 70 55 In order
List (K=5)
Just as in the straight insertion sort we compare 2 values andswap them if they are out of order However in the shell sort we compare values that are a distance K apartOnce we have completed going through the elements in our
list with K=5 we decrease K and continue the process
Shell SortPass Notes
6 21 62 14 9 30 77 80 25 70 55 Swap7 14 62 21 9 30 77 80 25 70 55 Swap8 14 9 21 62 30 77 80 25 70 55 In order9 14 9 21 62 30 77 80 25 70 55 In order10 14 9 21 62 30 77 80 25 70 55 In order11 14 9 21 62 30 77 80 25 70 55 Swap
14 9 21 62 30 25 80 77 70 55 Swap14 9 21 25 30 62 80 77 70 55 In order
12 14 9 21 25 30 62 80 77 70 55 Swap14 9 21 25 30 62 70 77 80 55 In order
13 14 9 21 25 30 62 70 77 80 55 Swap14 9 21 25 30 62 70 55 80 77 Swap14 9 21 25 30 55 70 62 80 77 In order
List (K=2)
Here we have reduced K to 2 Just as in the insertion sort if we swap 2 values we have to go back and compare theprevious 2 values to make sure they are still in order
Shell Sort
All shell sorts will terminate by running an insertion sort(ie K=1) However using the larger values of K first has helped to sort our list so that the straight insertion sort will runfaster
Shell Sort There are many schools of thought
on what the increment should be in the shell sort
Also note that just because an increment is optimal on one list it might not be optimal for another list
Insertion Sort vs Shell Sort Comparing the Big-O notation (for the
average case) we find that Insertion O(n2) Shell O(n125) empirically determined
Although this doesnrsquot seem like much of a gain it makes a big difference as n gets large
Note that in the worst case the Shell sort has an efficiency of O(n2)
However using a special incrementing technique this worst case can be reduced to O(n15)
Insertion Sort vs Shell Sort(Average Case)
Straight Insertion Shell25 625 55100 10000 316500 250000 2364
1000 1000000 56232000 4000000 13374
Number of Loopsn
Selection Sort Imagine some data that you can examine
all at once To sort it you could select the smallest
element and put it in its place select the next smallest and put it in its place etc
For a card player this process is analogous to looking at an entire hand of cards and ordering them by selecting cards one at a time and placing them in their proper order
Selection Sort The selection sort follows this idea Given a list of data to be sorted
we simply select the smallest item and place it in a sorted list
We then repeat these steps until the list is sorted
Selection Sort In the selection sort the list at any moment
is divided into 2 sublists sorted and unsorted separated by a ldquoconceptualrdquo wall
We select the smallest element from the unsorted sublist and exchange it with the element at the beginning of the unsorted data
After each selection and exchange the wall between the 2 sublists moves ndash increasing the number of sorted elements and decreasing the number of unsorted elements
Selection Sort
We start with an unsorted list We search this list for thesmallest element We then exchange the smallest element (8)with the first element in the unsorted list (23)
Again we search the unsorted list for the smallest element We then exchange the smallest element (23) with the first element in the unsorted list (78)
This process continues until the list is fully sorted
Bubble Sort In the bubble sort the list at any
moment is divided into 2 sublists sorted and unsorted
The smallest element is ldquobubbledrdquo from the unsorted sublist to the sorted sublist
Bubble Sort
23 78 45 8 56 3232 56
We start with 32 and compareit with 56 Because 32 is lessthan 56 we swap the twoand step down one element
We then compare 32 and 8Because 32 is not less than 8we do not swap these elements
We step down one element andcompare 45 and 8 They areout of sequence so we swapthem and step down again
8 45
We step down again and compare 8 with 78 These two elements are swapped
8 78
Finally 8 is compared with 23and swapped We thencontinue this process back with56 hellip
8 23
1 Pass of the Bubble Sort
Quick Sort In the bubble sort consecutive items
are compared and possibly exchanged on each pass through the list
This means that many exchanges may be needed to move an element to its correct position
Quick sort is more efficient than bubble sort because a typical exchange involves elements that are far apart so fewer exchanges are required to correctly position an element
Idea of Quick Sort1) Select pick an element
2) Divide rearrange elements so that x goes to its final position E
3) Recurse and Conquer recursively sort
Quick Sort Each iteration of the quick sort
selects an element known as the pivot and divides the list into 3 groups Elements whose keys are less than (or
equal to) the pivotrsquos key The pivot element Elements whose keys are greater than
(or equal to) the pivotrsquos key
Quick Sort The sorting then continues by
quick sorting the left partition followed by quick sorting the right partition
The basic algorithm is as follows
Quick Sort1) Partitioning Step Take an element in the
unsorted array and determine its final location in the sorted array This occurs when all values to the left of the element in the array are less than (or equal to) the element and all values to the right of the element are greater than (or equal to) the element We now have 1 element in its proper location and two unsorted subarrays
2) Recursive Step Perform step 1 on each unsorted subarray
Quick Sort Each time step 1 is performed on a
subarray another element is placed in its final location of the sorted array and two unsorted subarrays are created
When a subarray consists of one element that subarray is sorted
Therefore that element is in its final location
Quick Sort There are several partitioning strategies
used in practice (ie several ldquoversionsrdquo of quick sort) but the one we are about to describe is known to work well
For simplicity we will choose the last element to be the pivot element
We could also chose a different pivot element and swap it with the last element in the array
Quick Sort Below is the array we would like to
sort1 4 8 9 0 11 5 10 7 6
Quick Sort The index left starts at the first element
and right starts at the next-to-last element
We want to move all the elements smaller than the pivot to the left part of the array and all the elements larger than the pivot to the right part
1 4 8 9 0 11 5 10 7 6left righ
t
Quick Sort We move left to the right skipping
over elements that are smaller than the pivot
1 4 8 9 0 11 5 10 7 6left righ
t
Quick Sort We then move right to the left skipping
over elements that are greater than the pivot
When left and right have stopped left is on an element greater than (or equal to) the pivot and right is on an element smaller than (or equal to) the pivot
1 4 8 9 0 11 5 10 7 6left right
Quick Sort If left is to the left of right (or if left
= right) those elements are swapped
1 4 8 9 0 11 5 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
Quick Sort The effect is to push a large
element to the right and a small element to the left
We then repeat the process until left and right cross
Quick Sort
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6left righ
t
Quick Sort
1 4 5 0 9 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6righ
tleft
Quick Sort
At this point left and right have crossed so no swap is performed
The final part of the partitioning is to swap the pivot element with left
1 4 5 0 9 11 8 10 7 6righ
tleft
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort Note that all elements to the left of
the pivot are less than (or equal to) the pivot and all elements to the right of the pivot are greater than (or equal to) the pivot
Hence the pivot element has been placed in its final sorted position
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort We now repeat the process using
the sub-arrays to the left and right of the pivot
1 4 5 0 6 11 8 10 7 9
1 4 5 0 11 8 10 7 9
Analysis of quicksortmdashbest case
Suppose each partition operation divides the array almost exactly in half
Then the depth of the recursion in log2n Because thatrsquos how many times we can
halve n However there are many recursions at
each level How can we figure this out We note that
Each partition is linear over its subarray All the partitions at one level cover the array
Partitioning at various levels
Best case II We cut the array size in half each time So the depth of the recursion in log2n At each level of the recursion all the
partitions at that level do work that is linear in n
O(log2n) O(n) = O(n log2n)
What about the worst case
Worst case In the worst case partitioning always
divides the size n array into these three parts A length one part containing the pivot itself A length zero part and A length n-1 part containing everything else
We donrsquot recur on the zero-length part Recurring on the length n-1 part requires
(in the worst case) recurring to depth n-1
Worst case partitioning
Worst case for quicksort In the worst case recursion may be
O(n) levels deep (for an array of size n)
But the partitioning work done at each level is still O(n)
O(n) O(n) = O(n2) So worst case for Quicksort is O(n2) When does this happen
When the array is sorted to begin with
Typical case for quicksort If the array is sorted to begin with
Quicksort is terrible O(n2) It is possible to construct other bad
cases However Quicksort is usually O(n log2n) The constants are so good that Quicksort
is generally the fastest algorithm known Most real-world sorting is done by
Quicksort
Tweaking Quicksort Almost anything you can try to
ldquoimproverdquo Quicksort will actually slow it down
One good tweak is to switch to a different sorting method when the subarrays get small (say 10 or 12) Quicksort has too much overhead for small
array sizes For large arrays it might be a good
idea to check beforehand if the array is already sorted But there is a better tweak than this
Randomized Quick-Sort Select the pivot as a random element of the sequence The expected running time of randomized quick-sort on a
sequence of size n is O(n log n) The time spent at a level of the quick-sort tree is O(n) We show that the expected height of the quick-sort tree is
O(log n) good vs bad pivots
bull The probability of a good pivot is 12 thus we expect k2 good pivots out of k pivotsbull After a good pivot the size of each child sequence is at most 34 the size of the parent sequencebull After h pivots we expect (34)h2 n elementsbull The expected height h of the quick-sort tree is at most 2 log43n
Median of three Obviously it doesnrsquot make sense to sort the
array in order to find the median to use as a pivot
Instead compare just three elements of our (sub)arraymdashthe first the last and the middle Take the median (middle value) of these three as
pivot Itrsquos possible (but not easy) to construct cases
which will make this technique O(n2) Suppose we rearrange (sort) these three
numbers so that the smallest is in the first position the largest in the last position and the other in the middle This lets us simplify and speed up the partition
loop
Final comments Weissrsquos code shows some additional
optimizations on pp 246-247 Weiss chooses to stop both searches on
equality to pivot This design decision is debatable
Quicksort is the fastest known general sorting algorithm on average
For optimum speed the pivot must be chosen carefully
ldquoMedian of threerdquo is a good technique for choosing the pivot
There will be some cases where Quicksort runs in O(n2) time
Quick Sort A couple of notes about quick sort There are more optimal ways to
choose the pivot value (such as the median-of-three method)
Also when the subarrays get small it becomes more efficient to use the insertion sort as opposed to continued use of quick sort
Bubble Sort vs Quick Sort If we calculate the Big-O notation
we find that (in the average case) Bubble Sort O(n2) Quick Sort O(nlog2n)
(quick-sort runs in time O(n2) in the worst case)
Heap Sort Idea take the items that need to be
sorted and insert them into the heap By calling deleteHeap we remove the
smallest or largest element depending on whether or not we are working with a min- or max-heap respectively
Hence the elements are removed in ascending or descending order
Efficiency O(nlog2n)
Merge Sort Idea
Take the array you would like to sort and divide it in half to create 2 unsorted subarrays
Next sort each of the 2 subarrays Finally merge the 2 sorted subarrays
into 1 sorted array Efficiency O(nlog2n)
Merge Sort
Merge Sort Although the merge step produces
a sorted array we have overlooked a very important step
How did we sort the 2 halves before performing the merge step
We used merge sort
Merge Sort By continually calling the merge
sort algorithm we eventually get a subarray of size 1
Since an array with only 1 element is clearly sorted we can back out and merge 2 arrays of size 1
Merge Sort
Merge Sort The basic merging algorithm
consists of 2 input arrays (arrayA and arrayB) An ouput array (arrayC) 3 position holders (indexA indexB
indexC) which are initially set to the beginning of their respective arrays
Merge Sort The smaller of arrayA[indexA] and
arrayB[indexB] is copied into arrayC[indexC] and the appropriate position holders are advanced
When either input list is exhausted the remainder of the other list is copied into arrayC
Merge Sort
1 13 24 26index
A
indexC
arrayA
2 15 27 38index
B
arrayB
arrayC
We compare arrayA[indexA]with arrayB[indexB] Whichever value is smaller isplaced into arrayC[indexC]
1 lt 2 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1index
C
arrayA
2 15 27 38index
B
arrayB
arrayC
2 lt 13 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
13 lt 15 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
15 lt 24 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13 15indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
24 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
indexA
1 2 13 15 24indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
26 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
1 2 13 15 24 26indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
Since we have exhaustedone of the arrays arrayAwe simply copy the remaining items from theother array arrayB intoarrayC
Merge Sort
1 13 24
26
1 2 13 15 24 26 27 38
arrayA
2 15 27 38arrayB
arrayC
Efficiency Summary
Sort Worst Case Average Case
Insertion O(n2) O(n2)
Shell O(n15) O(n125)
Selection O(n2) O(n2)
Bubble O(n2) O(n2)
Quick O(n2) O(nlog2n)
Heap O(nlog2n) O(nlog2n)
Merge O(nlog2n) O(nlog2n)
More Sorting
radix sort bucket sort in-place sorting how fast can we sort
Radix Sort
Unlike other sorting methods radix sort considers the structure of the keys
Assume keys are represented in a base M number system (M = radix) ie if M = 2 the keys are represented in binary
Sorting is done by comparing bits in the same position
Extension to keys that are alphanumeric strings
Radix Exchange Sort
Examine bits from left to right1 Sort array with respect to leftmost bit
2 Partition array
3Recursionrecursively sort top subarray ignoring leftmost bitrecursively sort bottom subarray ignoring leftmost bit
Time to sort n b-bit numbers O(b n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Insertion SortHere we start with an unsorted list We leave the first data element alone and will start with the 2nd element of the list (the 1st element of the unsorted list)
On our first pass we look at the 2nd element of the list (thefirst element of the unsorted list) and place it in our sorted list in order
On our next pass we look at the 3rd element of the list (the1st element of the unsorted list) and place it in the sorted list in order
We continue in this fashion until the entire list has beensorted
Insertion Sort Example
In our first pass we compare the first 2 values Because theyare out of order we swap them
Now we look at the next 2 values Again they are out oforder so we swap them
Since we have swapped those values we need to comparethe previous 2 values to make sure that they are still in orderSince they are out of order we swap them and then continueon with the next 2 data valuesThese 2 values are out of order so we swap them and look atthe previous 2 values etc
Shell Sort Named after its creator Donald Shell the
shell sort is an improved version of the insertion sort
In the shell sort a list of N elements is divided into K segments where K is known as the increment
What this means is that instead of comparing adjacent values we will compare values that are a distance K apart
We will shrink K as we run through our algorithm
Shell SortPass Notes
1 77 62 14 9 30 21 80 25 70 55 Swap2 21 62 14 9 30 77 80 25 70 55 In order3 21 62 14 9 30 77 80 25 70 55 In order4 21 62 14 9 30 77 80 25 70 55 In order5 21 62 14 9 30 77 80 25 70 55 In order
List (K=5)
Just as in the straight insertion sort we compare 2 values andswap them if they are out of order However in the shell sort we compare values that are a distance K apartOnce we have completed going through the elements in our
list with K=5 we decrease K and continue the process
Shell SortPass Notes
6 21 62 14 9 30 77 80 25 70 55 Swap7 14 62 21 9 30 77 80 25 70 55 Swap8 14 9 21 62 30 77 80 25 70 55 In order9 14 9 21 62 30 77 80 25 70 55 In order10 14 9 21 62 30 77 80 25 70 55 In order11 14 9 21 62 30 77 80 25 70 55 Swap
14 9 21 62 30 25 80 77 70 55 Swap14 9 21 25 30 62 80 77 70 55 In order
12 14 9 21 25 30 62 80 77 70 55 Swap14 9 21 25 30 62 70 77 80 55 In order
13 14 9 21 25 30 62 70 77 80 55 Swap14 9 21 25 30 62 70 55 80 77 Swap14 9 21 25 30 55 70 62 80 77 In order
List (K=2)
Here we have reduced K to 2 Just as in the insertion sort if we swap 2 values we have to go back and compare theprevious 2 values to make sure they are still in order
Shell Sort
All shell sorts will terminate by running an insertion sort(ie K=1) However using the larger values of K first has helped to sort our list so that the straight insertion sort will runfaster
Shell Sort There are many schools of thought
on what the increment should be in the shell sort
Also note that just because an increment is optimal on one list it might not be optimal for another list
Insertion Sort vs Shell Sort Comparing the Big-O notation (for the
average case) we find that Insertion O(n2) Shell O(n125) empirically determined
Although this doesnrsquot seem like much of a gain it makes a big difference as n gets large
Note that in the worst case the Shell sort has an efficiency of O(n2)
However using a special incrementing technique this worst case can be reduced to O(n15)
Insertion Sort vs Shell Sort(Average Case)
Straight Insertion Shell25 625 55100 10000 316500 250000 2364
1000 1000000 56232000 4000000 13374
Number of Loopsn
Selection Sort Imagine some data that you can examine
all at once To sort it you could select the smallest
element and put it in its place select the next smallest and put it in its place etc
For a card player this process is analogous to looking at an entire hand of cards and ordering them by selecting cards one at a time and placing them in their proper order
Selection Sort The selection sort follows this idea Given a list of data to be sorted
we simply select the smallest item and place it in a sorted list
We then repeat these steps until the list is sorted
Selection Sort In the selection sort the list at any moment
is divided into 2 sublists sorted and unsorted separated by a ldquoconceptualrdquo wall
We select the smallest element from the unsorted sublist and exchange it with the element at the beginning of the unsorted data
After each selection and exchange the wall between the 2 sublists moves ndash increasing the number of sorted elements and decreasing the number of unsorted elements
Selection Sort
We start with an unsorted list We search this list for thesmallest element We then exchange the smallest element (8)with the first element in the unsorted list (23)
Again we search the unsorted list for the smallest element We then exchange the smallest element (23) with the first element in the unsorted list (78)
This process continues until the list is fully sorted
Bubble Sort In the bubble sort the list at any
moment is divided into 2 sublists sorted and unsorted
The smallest element is ldquobubbledrdquo from the unsorted sublist to the sorted sublist
Bubble Sort
23 78 45 8 56 3232 56
We start with 32 and compareit with 56 Because 32 is lessthan 56 we swap the twoand step down one element
We then compare 32 and 8Because 32 is not less than 8we do not swap these elements
We step down one element andcompare 45 and 8 They areout of sequence so we swapthem and step down again
8 45
We step down again and compare 8 with 78 These two elements are swapped
8 78
Finally 8 is compared with 23and swapped We thencontinue this process back with56 hellip
8 23
1 Pass of the Bubble Sort
Quick Sort In the bubble sort consecutive items
are compared and possibly exchanged on each pass through the list
This means that many exchanges may be needed to move an element to its correct position
Quick sort is more efficient than bubble sort because a typical exchange involves elements that are far apart so fewer exchanges are required to correctly position an element
Idea of Quick Sort1) Select pick an element
2) Divide rearrange elements so that x goes to its final position E
3) Recurse and Conquer recursively sort
Quick Sort Each iteration of the quick sort
selects an element known as the pivot and divides the list into 3 groups Elements whose keys are less than (or
equal to) the pivotrsquos key The pivot element Elements whose keys are greater than
(or equal to) the pivotrsquos key
Quick Sort The sorting then continues by
quick sorting the left partition followed by quick sorting the right partition
The basic algorithm is as follows
Quick Sort1) Partitioning Step Take an element in the
unsorted array and determine its final location in the sorted array This occurs when all values to the left of the element in the array are less than (or equal to) the element and all values to the right of the element are greater than (or equal to) the element We now have 1 element in its proper location and two unsorted subarrays
2) Recursive Step Perform step 1 on each unsorted subarray
Quick Sort Each time step 1 is performed on a
subarray another element is placed in its final location of the sorted array and two unsorted subarrays are created
When a subarray consists of one element that subarray is sorted
Therefore that element is in its final location
Quick Sort There are several partitioning strategies
used in practice (ie several ldquoversionsrdquo of quick sort) but the one we are about to describe is known to work well
For simplicity we will choose the last element to be the pivot element
We could also chose a different pivot element and swap it with the last element in the array
Quick Sort Below is the array we would like to
sort1 4 8 9 0 11 5 10 7 6
Quick Sort The index left starts at the first element
and right starts at the next-to-last element
We want to move all the elements smaller than the pivot to the left part of the array and all the elements larger than the pivot to the right part
1 4 8 9 0 11 5 10 7 6left righ
t
Quick Sort We move left to the right skipping
over elements that are smaller than the pivot
1 4 8 9 0 11 5 10 7 6left righ
t
Quick Sort We then move right to the left skipping
over elements that are greater than the pivot
When left and right have stopped left is on an element greater than (or equal to) the pivot and right is on an element smaller than (or equal to) the pivot
1 4 8 9 0 11 5 10 7 6left right
Quick Sort If left is to the left of right (or if left
= right) those elements are swapped
1 4 8 9 0 11 5 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
Quick Sort The effect is to push a large
element to the right and a small element to the left
We then repeat the process until left and right cross
Quick Sort
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6left righ
t
Quick Sort
1 4 5 0 9 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6righ
tleft
Quick Sort
At this point left and right have crossed so no swap is performed
The final part of the partitioning is to swap the pivot element with left
1 4 5 0 9 11 8 10 7 6righ
tleft
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort Note that all elements to the left of
the pivot are less than (or equal to) the pivot and all elements to the right of the pivot are greater than (or equal to) the pivot
Hence the pivot element has been placed in its final sorted position
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort We now repeat the process using
the sub-arrays to the left and right of the pivot
1 4 5 0 6 11 8 10 7 9
1 4 5 0 11 8 10 7 9
Analysis of quicksortmdashbest case
Suppose each partition operation divides the array almost exactly in half
Then the depth of the recursion in log2n Because thatrsquos how many times we can
halve n However there are many recursions at
each level How can we figure this out We note that
Each partition is linear over its subarray All the partitions at one level cover the array
Partitioning at various levels
Best case II We cut the array size in half each time So the depth of the recursion in log2n At each level of the recursion all the
partitions at that level do work that is linear in n
O(log2n) O(n) = O(n log2n)
What about the worst case
Worst case In the worst case partitioning always
divides the size n array into these three parts A length one part containing the pivot itself A length zero part and A length n-1 part containing everything else
We donrsquot recur on the zero-length part Recurring on the length n-1 part requires
(in the worst case) recurring to depth n-1
Worst case partitioning
Worst case for quicksort In the worst case recursion may be
O(n) levels deep (for an array of size n)
But the partitioning work done at each level is still O(n)
O(n) O(n) = O(n2) So worst case for Quicksort is O(n2) When does this happen
When the array is sorted to begin with
Typical case for quicksort If the array is sorted to begin with
Quicksort is terrible O(n2) It is possible to construct other bad
cases However Quicksort is usually O(n log2n) The constants are so good that Quicksort
is generally the fastest algorithm known Most real-world sorting is done by
Quicksort
Tweaking Quicksort Almost anything you can try to
ldquoimproverdquo Quicksort will actually slow it down
One good tweak is to switch to a different sorting method when the subarrays get small (say 10 or 12) Quicksort has too much overhead for small
array sizes For large arrays it might be a good
idea to check beforehand if the array is already sorted But there is a better tweak than this
Randomized Quick-Sort Select the pivot as a random element of the sequence The expected running time of randomized quick-sort on a
sequence of size n is O(n log n) The time spent at a level of the quick-sort tree is O(n) We show that the expected height of the quick-sort tree is
O(log n) good vs bad pivots
bull The probability of a good pivot is 12 thus we expect k2 good pivots out of k pivotsbull After a good pivot the size of each child sequence is at most 34 the size of the parent sequencebull After h pivots we expect (34)h2 n elementsbull The expected height h of the quick-sort tree is at most 2 log43n
Median of three Obviously it doesnrsquot make sense to sort the
array in order to find the median to use as a pivot
Instead compare just three elements of our (sub)arraymdashthe first the last and the middle Take the median (middle value) of these three as
pivot Itrsquos possible (but not easy) to construct cases
which will make this technique O(n2) Suppose we rearrange (sort) these three
numbers so that the smallest is in the first position the largest in the last position and the other in the middle This lets us simplify and speed up the partition
loop
Final comments Weissrsquos code shows some additional
optimizations on pp 246-247 Weiss chooses to stop both searches on
equality to pivot This design decision is debatable
Quicksort is the fastest known general sorting algorithm on average
For optimum speed the pivot must be chosen carefully
ldquoMedian of threerdquo is a good technique for choosing the pivot
There will be some cases where Quicksort runs in O(n2) time
Quick Sort A couple of notes about quick sort There are more optimal ways to
choose the pivot value (such as the median-of-three method)
Also when the subarrays get small it becomes more efficient to use the insertion sort as opposed to continued use of quick sort
Bubble Sort vs Quick Sort If we calculate the Big-O notation
we find that (in the average case) Bubble Sort O(n2) Quick Sort O(nlog2n)
(quick-sort runs in time O(n2) in the worst case)
Heap Sort Idea take the items that need to be
sorted and insert them into the heap By calling deleteHeap we remove the
smallest or largest element depending on whether or not we are working with a min- or max-heap respectively
Hence the elements are removed in ascending or descending order
Efficiency O(nlog2n)
Merge Sort Idea
Take the array you would like to sort and divide it in half to create 2 unsorted subarrays
Next sort each of the 2 subarrays Finally merge the 2 sorted subarrays
into 1 sorted array Efficiency O(nlog2n)
Merge Sort
Merge Sort Although the merge step produces
a sorted array we have overlooked a very important step
How did we sort the 2 halves before performing the merge step
We used merge sort
Merge Sort By continually calling the merge
sort algorithm we eventually get a subarray of size 1
Since an array with only 1 element is clearly sorted we can back out and merge 2 arrays of size 1
Merge Sort
Merge Sort The basic merging algorithm
consists of 2 input arrays (arrayA and arrayB) An ouput array (arrayC) 3 position holders (indexA indexB
indexC) which are initially set to the beginning of their respective arrays
Merge Sort The smaller of arrayA[indexA] and
arrayB[indexB] is copied into arrayC[indexC] and the appropriate position holders are advanced
When either input list is exhausted the remainder of the other list is copied into arrayC
Merge Sort
1 13 24 26index
A
indexC
arrayA
2 15 27 38index
B
arrayB
arrayC
We compare arrayA[indexA]with arrayB[indexB] Whichever value is smaller isplaced into arrayC[indexC]
1 lt 2 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1index
C
arrayA
2 15 27 38index
B
arrayB
arrayC
2 lt 13 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
13 lt 15 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
15 lt 24 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13 15indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
24 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
indexA
1 2 13 15 24indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
26 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
1 2 13 15 24 26indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
Since we have exhaustedone of the arrays arrayAwe simply copy the remaining items from theother array arrayB intoarrayC
Merge Sort
1 13 24
26
1 2 13 15 24 26 27 38
arrayA
2 15 27 38arrayB
arrayC
Efficiency Summary
Sort Worst Case Average Case
Insertion O(n2) O(n2)
Shell O(n15) O(n125)
Selection O(n2) O(n2)
Bubble O(n2) O(n2)
Quick O(n2) O(nlog2n)
Heap O(nlog2n) O(nlog2n)
Merge O(nlog2n) O(nlog2n)
More Sorting
radix sort bucket sort in-place sorting how fast can we sort
Radix Sort
Unlike other sorting methods radix sort considers the structure of the keys
Assume keys are represented in a base M number system (M = radix) ie if M = 2 the keys are represented in binary
Sorting is done by comparing bits in the same position
Extension to keys that are alphanumeric strings
Radix Exchange Sort
Examine bits from left to right1 Sort array with respect to leftmost bit
2 Partition array
3Recursionrecursively sort top subarray ignoring leftmost bitrecursively sort bottom subarray ignoring leftmost bit
Time to sort n b-bit numbers O(b n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Insertion Sort Example
In our first pass we compare the first 2 values Because theyare out of order we swap them
Now we look at the next 2 values Again they are out oforder so we swap them
Since we have swapped those values we need to comparethe previous 2 values to make sure that they are still in orderSince they are out of order we swap them and then continueon with the next 2 data valuesThese 2 values are out of order so we swap them and look atthe previous 2 values etc
Shell Sort Named after its creator Donald Shell the
shell sort is an improved version of the insertion sort
In the shell sort a list of N elements is divided into K segments where K is known as the increment
What this means is that instead of comparing adjacent values we will compare values that are a distance K apart
We will shrink K as we run through our algorithm
Shell SortPass Notes
1 77 62 14 9 30 21 80 25 70 55 Swap2 21 62 14 9 30 77 80 25 70 55 In order3 21 62 14 9 30 77 80 25 70 55 In order4 21 62 14 9 30 77 80 25 70 55 In order5 21 62 14 9 30 77 80 25 70 55 In order
List (K=5)
Just as in the straight insertion sort we compare 2 values andswap them if they are out of order However in the shell sort we compare values that are a distance K apartOnce we have completed going through the elements in our
list with K=5 we decrease K and continue the process
Shell SortPass Notes
6 21 62 14 9 30 77 80 25 70 55 Swap7 14 62 21 9 30 77 80 25 70 55 Swap8 14 9 21 62 30 77 80 25 70 55 In order9 14 9 21 62 30 77 80 25 70 55 In order10 14 9 21 62 30 77 80 25 70 55 In order11 14 9 21 62 30 77 80 25 70 55 Swap
14 9 21 62 30 25 80 77 70 55 Swap14 9 21 25 30 62 80 77 70 55 In order
12 14 9 21 25 30 62 80 77 70 55 Swap14 9 21 25 30 62 70 77 80 55 In order
13 14 9 21 25 30 62 70 77 80 55 Swap14 9 21 25 30 62 70 55 80 77 Swap14 9 21 25 30 55 70 62 80 77 In order
List (K=2)
Here we have reduced K to 2 Just as in the insertion sort if we swap 2 values we have to go back and compare theprevious 2 values to make sure they are still in order
Shell Sort
All shell sorts will terminate by running an insertion sort(ie K=1) However using the larger values of K first has helped to sort our list so that the straight insertion sort will runfaster
Shell Sort There are many schools of thought
on what the increment should be in the shell sort
Also note that just because an increment is optimal on one list it might not be optimal for another list
Insertion Sort vs Shell Sort Comparing the Big-O notation (for the
average case) we find that Insertion O(n2) Shell O(n125) empirically determined
Although this doesnrsquot seem like much of a gain it makes a big difference as n gets large
Note that in the worst case the Shell sort has an efficiency of O(n2)
However using a special incrementing technique this worst case can be reduced to O(n15)
Insertion Sort vs Shell Sort(Average Case)
Straight Insertion Shell25 625 55100 10000 316500 250000 2364
1000 1000000 56232000 4000000 13374
Number of Loopsn
Selection Sort Imagine some data that you can examine
all at once To sort it you could select the smallest
element and put it in its place select the next smallest and put it in its place etc
For a card player this process is analogous to looking at an entire hand of cards and ordering them by selecting cards one at a time and placing them in their proper order
Selection Sort The selection sort follows this idea Given a list of data to be sorted
we simply select the smallest item and place it in a sorted list
We then repeat these steps until the list is sorted
Selection Sort In the selection sort the list at any moment
is divided into 2 sublists sorted and unsorted separated by a ldquoconceptualrdquo wall
We select the smallest element from the unsorted sublist and exchange it with the element at the beginning of the unsorted data
After each selection and exchange the wall between the 2 sublists moves ndash increasing the number of sorted elements and decreasing the number of unsorted elements
Selection Sort
We start with an unsorted list We search this list for thesmallest element We then exchange the smallest element (8)with the first element in the unsorted list (23)
Again we search the unsorted list for the smallest element We then exchange the smallest element (23) with the first element in the unsorted list (78)
This process continues until the list is fully sorted
Bubble Sort In the bubble sort the list at any
moment is divided into 2 sublists sorted and unsorted
The smallest element is ldquobubbledrdquo from the unsorted sublist to the sorted sublist
Bubble Sort
23 78 45 8 56 3232 56
We start with 32 and compareit with 56 Because 32 is lessthan 56 we swap the twoand step down one element
We then compare 32 and 8Because 32 is not less than 8we do not swap these elements
We step down one element andcompare 45 and 8 They areout of sequence so we swapthem and step down again
8 45
We step down again and compare 8 with 78 These two elements are swapped
8 78
Finally 8 is compared with 23and swapped We thencontinue this process back with56 hellip
8 23
1 Pass of the Bubble Sort
Quick Sort In the bubble sort consecutive items
are compared and possibly exchanged on each pass through the list
This means that many exchanges may be needed to move an element to its correct position
Quick sort is more efficient than bubble sort because a typical exchange involves elements that are far apart so fewer exchanges are required to correctly position an element
Idea of Quick Sort1) Select pick an element
2) Divide rearrange elements so that x goes to its final position E
3) Recurse and Conquer recursively sort
Quick Sort Each iteration of the quick sort
selects an element known as the pivot and divides the list into 3 groups Elements whose keys are less than (or
equal to) the pivotrsquos key The pivot element Elements whose keys are greater than
(or equal to) the pivotrsquos key
Quick Sort The sorting then continues by
quick sorting the left partition followed by quick sorting the right partition
The basic algorithm is as follows
Quick Sort1) Partitioning Step Take an element in the
unsorted array and determine its final location in the sorted array This occurs when all values to the left of the element in the array are less than (or equal to) the element and all values to the right of the element are greater than (or equal to) the element We now have 1 element in its proper location and two unsorted subarrays
2) Recursive Step Perform step 1 on each unsorted subarray
Quick Sort Each time step 1 is performed on a
subarray another element is placed in its final location of the sorted array and two unsorted subarrays are created
When a subarray consists of one element that subarray is sorted
Therefore that element is in its final location
Quick Sort There are several partitioning strategies
used in practice (ie several ldquoversionsrdquo of quick sort) but the one we are about to describe is known to work well
For simplicity we will choose the last element to be the pivot element
We could also chose a different pivot element and swap it with the last element in the array
Quick Sort Below is the array we would like to
sort1 4 8 9 0 11 5 10 7 6
Quick Sort The index left starts at the first element
and right starts at the next-to-last element
We want to move all the elements smaller than the pivot to the left part of the array and all the elements larger than the pivot to the right part
1 4 8 9 0 11 5 10 7 6left righ
t
Quick Sort We move left to the right skipping
over elements that are smaller than the pivot
1 4 8 9 0 11 5 10 7 6left righ
t
Quick Sort We then move right to the left skipping
over elements that are greater than the pivot
When left and right have stopped left is on an element greater than (or equal to) the pivot and right is on an element smaller than (or equal to) the pivot
1 4 8 9 0 11 5 10 7 6left right
Quick Sort If left is to the left of right (or if left
= right) those elements are swapped
1 4 8 9 0 11 5 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
Quick Sort The effect is to push a large
element to the right and a small element to the left
We then repeat the process until left and right cross
Quick Sort
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6left righ
t
Quick Sort
1 4 5 0 9 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6righ
tleft
Quick Sort
At this point left and right have crossed so no swap is performed
The final part of the partitioning is to swap the pivot element with left
1 4 5 0 9 11 8 10 7 6righ
tleft
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort Note that all elements to the left of
the pivot are less than (or equal to) the pivot and all elements to the right of the pivot are greater than (or equal to) the pivot
Hence the pivot element has been placed in its final sorted position
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort We now repeat the process using
the sub-arrays to the left and right of the pivot
1 4 5 0 6 11 8 10 7 9
1 4 5 0 11 8 10 7 9
Analysis of quicksortmdashbest case
Suppose each partition operation divides the array almost exactly in half
Then the depth of the recursion in log2n Because thatrsquos how many times we can
halve n However there are many recursions at
each level How can we figure this out We note that
Each partition is linear over its subarray All the partitions at one level cover the array
Partitioning at various levels
Best case II We cut the array size in half each time So the depth of the recursion in log2n At each level of the recursion all the
partitions at that level do work that is linear in n
O(log2n) O(n) = O(n log2n)
What about the worst case
Worst case In the worst case partitioning always
divides the size n array into these three parts A length one part containing the pivot itself A length zero part and A length n-1 part containing everything else
We donrsquot recur on the zero-length part Recurring on the length n-1 part requires
(in the worst case) recurring to depth n-1
Worst case partitioning
Worst case for quicksort In the worst case recursion may be
O(n) levels deep (for an array of size n)
But the partitioning work done at each level is still O(n)
O(n) O(n) = O(n2) So worst case for Quicksort is O(n2) When does this happen
When the array is sorted to begin with
Typical case for quicksort If the array is sorted to begin with
Quicksort is terrible O(n2) It is possible to construct other bad
cases However Quicksort is usually O(n log2n) The constants are so good that Quicksort
is generally the fastest algorithm known Most real-world sorting is done by
Quicksort
Tweaking Quicksort Almost anything you can try to
ldquoimproverdquo Quicksort will actually slow it down
One good tweak is to switch to a different sorting method when the subarrays get small (say 10 or 12) Quicksort has too much overhead for small
array sizes For large arrays it might be a good
idea to check beforehand if the array is already sorted But there is a better tweak than this
Randomized Quick-Sort Select the pivot as a random element of the sequence The expected running time of randomized quick-sort on a
sequence of size n is O(n log n) The time spent at a level of the quick-sort tree is O(n) We show that the expected height of the quick-sort tree is
O(log n) good vs bad pivots
bull The probability of a good pivot is 12 thus we expect k2 good pivots out of k pivotsbull After a good pivot the size of each child sequence is at most 34 the size of the parent sequencebull After h pivots we expect (34)h2 n elementsbull The expected height h of the quick-sort tree is at most 2 log43n
Median of three Obviously it doesnrsquot make sense to sort the
array in order to find the median to use as a pivot
Instead compare just three elements of our (sub)arraymdashthe first the last and the middle Take the median (middle value) of these three as
pivot Itrsquos possible (but not easy) to construct cases
which will make this technique O(n2) Suppose we rearrange (sort) these three
numbers so that the smallest is in the first position the largest in the last position and the other in the middle This lets us simplify and speed up the partition
loop
Final comments Weissrsquos code shows some additional
optimizations on pp 246-247 Weiss chooses to stop both searches on
equality to pivot This design decision is debatable
Quicksort is the fastest known general sorting algorithm on average
For optimum speed the pivot must be chosen carefully
ldquoMedian of threerdquo is a good technique for choosing the pivot
There will be some cases where Quicksort runs in O(n2) time
Quick Sort A couple of notes about quick sort There are more optimal ways to
choose the pivot value (such as the median-of-three method)
Also when the subarrays get small it becomes more efficient to use the insertion sort as opposed to continued use of quick sort
Bubble Sort vs Quick Sort If we calculate the Big-O notation
we find that (in the average case) Bubble Sort O(n2) Quick Sort O(nlog2n)
(quick-sort runs in time O(n2) in the worst case)
Heap Sort Idea take the items that need to be
sorted and insert them into the heap By calling deleteHeap we remove the
smallest or largest element depending on whether or not we are working with a min- or max-heap respectively
Hence the elements are removed in ascending or descending order
Efficiency O(nlog2n)
Merge Sort Idea
Take the array you would like to sort and divide it in half to create 2 unsorted subarrays
Next sort each of the 2 subarrays Finally merge the 2 sorted subarrays
into 1 sorted array Efficiency O(nlog2n)
Merge Sort
Merge Sort Although the merge step produces
a sorted array we have overlooked a very important step
How did we sort the 2 halves before performing the merge step
We used merge sort
Merge Sort By continually calling the merge
sort algorithm we eventually get a subarray of size 1
Since an array with only 1 element is clearly sorted we can back out and merge 2 arrays of size 1
Merge Sort
Merge Sort The basic merging algorithm
consists of 2 input arrays (arrayA and arrayB) An ouput array (arrayC) 3 position holders (indexA indexB
indexC) which are initially set to the beginning of their respective arrays
Merge Sort The smaller of arrayA[indexA] and
arrayB[indexB] is copied into arrayC[indexC] and the appropriate position holders are advanced
When either input list is exhausted the remainder of the other list is copied into arrayC
Merge Sort
1 13 24 26index
A
indexC
arrayA
2 15 27 38index
B
arrayB
arrayC
We compare arrayA[indexA]with arrayB[indexB] Whichever value is smaller isplaced into arrayC[indexC]
1 lt 2 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1index
C
arrayA
2 15 27 38index
B
arrayB
arrayC
2 lt 13 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
13 lt 15 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
15 lt 24 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13 15indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
24 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
indexA
1 2 13 15 24indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
26 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
1 2 13 15 24 26indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
Since we have exhaustedone of the arrays arrayAwe simply copy the remaining items from theother array arrayB intoarrayC
Merge Sort
1 13 24
26
1 2 13 15 24 26 27 38
arrayA
2 15 27 38arrayB
arrayC
Efficiency Summary
Sort Worst Case Average Case
Insertion O(n2) O(n2)
Shell O(n15) O(n125)
Selection O(n2) O(n2)
Bubble O(n2) O(n2)
Quick O(n2) O(nlog2n)
Heap O(nlog2n) O(nlog2n)
Merge O(nlog2n) O(nlog2n)
More Sorting
radix sort bucket sort in-place sorting how fast can we sort
Radix Sort
Unlike other sorting methods radix sort considers the structure of the keys
Assume keys are represented in a base M number system (M = radix) ie if M = 2 the keys are represented in binary
Sorting is done by comparing bits in the same position
Extension to keys that are alphanumeric strings
Radix Exchange Sort
Examine bits from left to right1 Sort array with respect to leftmost bit
2 Partition array
3Recursionrecursively sort top subarray ignoring leftmost bitrecursively sort bottom subarray ignoring leftmost bit
Time to sort n b-bit numbers O(b n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Shell Sort Named after its creator Donald Shell the
shell sort is an improved version of the insertion sort
In the shell sort a list of N elements is divided into K segments where K is known as the increment
What this means is that instead of comparing adjacent values we will compare values that are a distance K apart
We will shrink K as we run through our algorithm
Shell SortPass Notes
1 77 62 14 9 30 21 80 25 70 55 Swap2 21 62 14 9 30 77 80 25 70 55 In order3 21 62 14 9 30 77 80 25 70 55 In order4 21 62 14 9 30 77 80 25 70 55 In order5 21 62 14 9 30 77 80 25 70 55 In order
List (K=5)
Just as in the straight insertion sort we compare 2 values andswap them if they are out of order However in the shell sort we compare values that are a distance K apartOnce we have completed going through the elements in our
list with K=5 we decrease K and continue the process
Shell SortPass Notes
6 21 62 14 9 30 77 80 25 70 55 Swap7 14 62 21 9 30 77 80 25 70 55 Swap8 14 9 21 62 30 77 80 25 70 55 In order9 14 9 21 62 30 77 80 25 70 55 In order10 14 9 21 62 30 77 80 25 70 55 In order11 14 9 21 62 30 77 80 25 70 55 Swap
14 9 21 62 30 25 80 77 70 55 Swap14 9 21 25 30 62 80 77 70 55 In order
12 14 9 21 25 30 62 80 77 70 55 Swap14 9 21 25 30 62 70 77 80 55 In order
13 14 9 21 25 30 62 70 77 80 55 Swap14 9 21 25 30 62 70 55 80 77 Swap14 9 21 25 30 55 70 62 80 77 In order
List (K=2)
Here we have reduced K to 2 Just as in the insertion sort if we swap 2 values we have to go back and compare theprevious 2 values to make sure they are still in order
Shell Sort
All shell sorts will terminate by running an insertion sort(ie K=1) However using the larger values of K first has helped to sort our list so that the straight insertion sort will runfaster
Shell Sort There are many schools of thought
on what the increment should be in the shell sort
Also note that just because an increment is optimal on one list it might not be optimal for another list
Insertion Sort vs Shell Sort Comparing the Big-O notation (for the
average case) we find that Insertion O(n2) Shell O(n125) empirically determined
Although this doesnrsquot seem like much of a gain it makes a big difference as n gets large
Note that in the worst case the Shell sort has an efficiency of O(n2)
However using a special incrementing technique this worst case can be reduced to O(n15)
Insertion Sort vs Shell Sort(Average Case)
Straight Insertion Shell25 625 55100 10000 316500 250000 2364
1000 1000000 56232000 4000000 13374
Number of Loopsn
Selection Sort Imagine some data that you can examine
all at once To sort it you could select the smallest
element and put it in its place select the next smallest and put it in its place etc
For a card player this process is analogous to looking at an entire hand of cards and ordering them by selecting cards one at a time and placing them in their proper order
Selection Sort The selection sort follows this idea Given a list of data to be sorted
we simply select the smallest item and place it in a sorted list
We then repeat these steps until the list is sorted
Selection Sort In the selection sort the list at any moment
is divided into 2 sublists sorted and unsorted separated by a ldquoconceptualrdquo wall
We select the smallest element from the unsorted sublist and exchange it with the element at the beginning of the unsorted data
After each selection and exchange the wall between the 2 sublists moves ndash increasing the number of sorted elements and decreasing the number of unsorted elements
Selection Sort
We start with an unsorted list We search this list for thesmallest element We then exchange the smallest element (8)with the first element in the unsorted list (23)
Again we search the unsorted list for the smallest element We then exchange the smallest element (23) with the first element in the unsorted list (78)
This process continues until the list is fully sorted
Bubble Sort In the bubble sort the list at any
moment is divided into 2 sublists sorted and unsorted
The smallest element is ldquobubbledrdquo from the unsorted sublist to the sorted sublist
Bubble Sort
23 78 45 8 56 3232 56
We start with 32 and compareit with 56 Because 32 is lessthan 56 we swap the twoand step down one element
We then compare 32 and 8Because 32 is not less than 8we do not swap these elements
We step down one element andcompare 45 and 8 They areout of sequence so we swapthem and step down again
8 45
We step down again and compare 8 with 78 These two elements are swapped
8 78
Finally 8 is compared with 23and swapped We thencontinue this process back with56 hellip
8 23
1 Pass of the Bubble Sort
Quick Sort In the bubble sort consecutive items
are compared and possibly exchanged on each pass through the list
This means that many exchanges may be needed to move an element to its correct position
Quick sort is more efficient than bubble sort because a typical exchange involves elements that are far apart so fewer exchanges are required to correctly position an element
Idea of Quick Sort1) Select pick an element
2) Divide rearrange elements so that x goes to its final position E
3) Recurse and Conquer recursively sort
Quick Sort Each iteration of the quick sort
selects an element known as the pivot and divides the list into 3 groups Elements whose keys are less than (or
equal to) the pivotrsquos key The pivot element Elements whose keys are greater than
(or equal to) the pivotrsquos key
Quick Sort The sorting then continues by
quick sorting the left partition followed by quick sorting the right partition
The basic algorithm is as follows
Quick Sort1) Partitioning Step Take an element in the
unsorted array and determine its final location in the sorted array This occurs when all values to the left of the element in the array are less than (or equal to) the element and all values to the right of the element are greater than (or equal to) the element We now have 1 element in its proper location and two unsorted subarrays
2) Recursive Step Perform step 1 on each unsorted subarray
Quick Sort Each time step 1 is performed on a
subarray another element is placed in its final location of the sorted array and two unsorted subarrays are created
When a subarray consists of one element that subarray is sorted
Therefore that element is in its final location
Quick Sort There are several partitioning strategies
used in practice (ie several ldquoversionsrdquo of quick sort) but the one we are about to describe is known to work well
For simplicity we will choose the last element to be the pivot element
We could also chose a different pivot element and swap it with the last element in the array
Quick Sort Below is the array we would like to
sort1 4 8 9 0 11 5 10 7 6
Quick Sort The index left starts at the first element
and right starts at the next-to-last element
We want to move all the elements smaller than the pivot to the left part of the array and all the elements larger than the pivot to the right part
1 4 8 9 0 11 5 10 7 6left righ
t
Quick Sort We move left to the right skipping
over elements that are smaller than the pivot
1 4 8 9 0 11 5 10 7 6left righ
t
Quick Sort We then move right to the left skipping
over elements that are greater than the pivot
When left and right have stopped left is on an element greater than (or equal to) the pivot and right is on an element smaller than (or equal to) the pivot
1 4 8 9 0 11 5 10 7 6left right
Quick Sort If left is to the left of right (or if left
= right) those elements are swapped
1 4 8 9 0 11 5 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
Quick Sort The effect is to push a large
element to the right and a small element to the left
We then repeat the process until left and right cross
Quick Sort
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6left righ
t
Quick Sort
1 4 5 0 9 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6righ
tleft
Quick Sort
At this point left and right have crossed so no swap is performed
The final part of the partitioning is to swap the pivot element with left
1 4 5 0 9 11 8 10 7 6righ
tleft
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort Note that all elements to the left of
the pivot are less than (or equal to) the pivot and all elements to the right of the pivot are greater than (or equal to) the pivot
Hence the pivot element has been placed in its final sorted position
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort We now repeat the process using
the sub-arrays to the left and right of the pivot
1 4 5 0 6 11 8 10 7 9
1 4 5 0 11 8 10 7 9
Analysis of quicksortmdashbest case
Suppose each partition operation divides the array almost exactly in half
Then the depth of the recursion in log2n Because thatrsquos how many times we can
halve n However there are many recursions at
each level How can we figure this out We note that
Each partition is linear over its subarray All the partitions at one level cover the array
Partitioning at various levels
Best case II We cut the array size in half each time So the depth of the recursion in log2n At each level of the recursion all the
partitions at that level do work that is linear in n
O(log2n) O(n) = O(n log2n)
What about the worst case
Worst case In the worst case partitioning always
divides the size n array into these three parts A length one part containing the pivot itself A length zero part and A length n-1 part containing everything else
We donrsquot recur on the zero-length part Recurring on the length n-1 part requires
(in the worst case) recurring to depth n-1
Worst case partitioning
Worst case for quicksort In the worst case recursion may be
O(n) levels deep (for an array of size n)
But the partitioning work done at each level is still O(n)
O(n) O(n) = O(n2) So worst case for Quicksort is O(n2) When does this happen
When the array is sorted to begin with
Typical case for quicksort If the array is sorted to begin with
Quicksort is terrible O(n2) It is possible to construct other bad
cases However Quicksort is usually O(n log2n) The constants are so good that Quicksort
is generally the fastest algorithm known Most real-world sorting is done by
Quicksort
Tweaking Quicksort Almost anything you can try to
ldquoimproverdquo Quicksort will actually slow it down
One good tweak is to switch to a different sorting method when the subarrays get small (say 10 or 12) Quicksort has too much overhead for small
array sizes For large arrays it might be a good
idea to check beforehand if the array is already sorted But there is a better tweak than this
Randomized Quick-Sort Select the pivot as a random element of the sequence The expected running time of randomized quick-sort on a
sequence of size n is O(n log n) The time spent at a level of the quick-sort tree is O(n) We show that the expected height of the quick-sort tree is
O(log n) good vs bad pivots
bull The probability of a good pivot is 12 thus we expect k2 good pivots out of k pivotsbull After a good pivot the size of each child sequence is at most 34 the size of the parent sequencebull After h pivots we expect (34)h2 n elementsbull The expected height h of the quick-sort tree is at most 2 log43n
Median of three Obviously it doesnrsquot make sense to sort the
array in order to find the median to use as a pivot
Instead compare just three elements of our (sub)arraymdashthe first the last and the middle Take the median (middle value) of these three as
pivot Itrsquos possible (but not easy) to construct cases
which will make this technique O(n2) Suppose we rearrange (sort) these three
numbers so that the smallest is in the first position the largest in the last position and the other in the middle This lets us simplify and speed up the partition
loop
Final comments Weissrsquos code shows some additional
optimizations on pp 246-247 Weiss chooses to stop both searches on
equality to pivot This design decision is debatable
Quicksort is the fastest known general sorting algorithm on average
For optimum speed the pivot must be chosen carefully
ldquoMedian of threerdquo is a good technique for choosing the pivot
There will be some cases where Quicksort runs in O(n2) time
Quick Sort A couple of notes about quick sort There are more optimal ways to
choose the pivot value (such as the median-of-three method)
Also when the subarrays get small it becomes more efficient to use the insertion sort as opposed to continued use of quick sort
Bubble Sort vs Quick Sort If we calculate the Big-O notation
we find that (in the average case) Bubble Sort O(n2) Quick Sort O(nlog2n)
(quick-sort runs in time O(n2) in the worst case)
Heap Sort Idea take the items that need to be
sorted and insert them into the heap By calling deleteHeap we remove the
smallest or largest element depending on whether or not we are working with a min- or max-heap respectively
Hence the elements are removed in ascending or descending order
Efficiency O(nlog2n)
Merge Sort Idea
Take the array you would like to sort and divide it in half to create 2 unsorted subarrays
Next sort each of the 2 subarrays Finally merge the 2 sorted subarrays
into 1 sorted array Efficiency O(nlog2n)
Merge Sort
Merge Sort Although the merge step produces
a sorted array we have overlooked a very important step
How did we sort the 2 halves before performing the merge step
We used merge sort
Merge Sort By continually calling the merge
sort algorithm we eventually get a subarray of size 1
Since an array with only 1 element is clearly sorted we can back out and merge 2 arrays of size 1
Merge Sort
Merge Sort The basic merging algorithm
consists of 2 input arrays (arrayA and arrayB) An ouput array (arrayC) 3 position holders (indexA indexB
indexC) which are initially set to the beginning of their respective arrays
Merge Sort The smaller of arrayA[indexA] and
arrayB[indexB] is copied into arrayC[indexC] and the appropriate position holders are advanced
When either input list is exhausted the remainder of the other list is copied into arrayC
Merge Sort
1 13 24 26index
A
indexC
arrayA
2 15 27 38index
B
arrayB
arrayC
We compare arrayA[indexA]with arrayB[indexB] Whichever value is smaller isplaced into arrayC[indexC]
1 lt 2 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1index
C
arrayA
2 15 27 38index
B
arrayB
arrayC
2 lt 13 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
13 lt 15 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
15 lt 24 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13 15indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
24 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
indexA
1 2 13 15 24indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
26 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
1 2 13 15 24 26indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
Since we have exhaustedone of the arrays arrayAwe simply copy the remaining items from theother array arrayB intoarrayC
Merge Sort
1 13 24
26
1 2 13 15 24 26 27 38
arrayA
2 15 27 38arrayB
arrayC
Efficiency Summary
Sort Worst Case Average Case
Insertion O(n2) O(n2)
Shell O(n15) O(n125)
Selection O(n2) O(n2)
Bubble O(n2) O(n2)
Quick O(n2) O(nlog2n)
Heap O(nlog2n) O(nlog2n)
Merge O(nlog2n) O(nlog2n)
More Sorting
radix sort bucket sort in-place sorting how fast can we sort
Radix Sort
Unlike other sorting methods radix sort considers the structure of the keys
Assume keys are represented in a base M number system (M = radix) ie if M = 2 the keys are represented in binary
Sorting is done by comparing bits in the same position
Extension to keys that are alphanumeric strings
Radix Exchange Sort
Examine bits from left to right1 Sort array with respect to leftmost bit
2 Partition array
3Recursionrecursively sort top subarray ignoring leftmost bitrecursively sort bottom subarray ignoring leftmost bit
Time to sort n b-bit numbers O(b n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Shell SortPass Notes
1 77 62 14 9 30 21 80 25 70 55 Swap2 21 62 14 9 30 77 80 25 70 55 In order3 21 62 14 9 30 77 80 25 70 55 In order4 21 62 14 9 30 77 80 25 70 55 In order5 21 62 14 9 30 77 80 25 70 55 In order
List (K=5)
Just as in the straight insertion sort we compare 2 values andswap them if they are out of order However in the shell sort we compare values that are a distance K apartOnce we have completed going through the elements in our
list with K=5 we decrease K and continue the process
Shell SortPass Notes
6 21 62 14 9 30 77 80 25 70 55 Swap7 14 62 21 9 30 77 80 25 70 55 Swap8 14 9 21 62 30 77 80 25 70 55 In order9 14 9 21 62 30 77 80 25 70 55 In order10 14 9 21 62 30 77 80 25 70 55 In order11 14 9 21 62 30 77 80 25 70 55 Swap
14 9 21 62 30 25 80 77 70 55 Swap14 9 21 25 30 62 80 77 70 55 In order
12 14 9 21 25 30 62 80 77 70 55 Swap14 9 21 25 30 62 70 77 80 55 In order
13 14 9 21 25 30 62 70 77 80 55 Swap14 9 21 25 30 62 70 55 80 77 Swap14 9 21 25 30 55 70 62 80 77 In order
List (K=2)
Here we have reduced K to 2 Just as in the insertion sort if we swap 2 values we have to go back and compare theprevious 2 values to make sure they are still in order
Shell Sort
All shell sorts will terminate by running an insertion sort(ie K=1) However using the larger values of K first has helped to sort our list so that the straight insertion sort will runfaster
Shell Sort There are many schools of thought
on what the increment should be in the shell sort
Also note that just because an increment is optimal on one list it might not be optimal for another list
Insertion Sort vs Shell Sort Comparing the Big-O notation (for the
average case) we find that Insertion O(n2) Shell O(n125) empirically determined
Although this doesnrsquot seem like much of a gain it makes a big difference as n gets large
Note that in the worst case the Shell sort has an efficiency of O(n2)
However using a special incrementing technique this worst case can be reduced to O(n15)
Insertion Sort vs Shell Sort(Average Case)
Straight Insertion Shell25 625 55100 10000 316500 250000 2364
1000 1000000 56232000 4000000 13374
Number of Loopsn
Selection Sort Imagine some data that you can examine
all at once To sort it you could select the smallest
element and put it in its place select the next smallest and put it in its place etc
For a card player this process is analogous to looking at an entire hand of cards and ordering them by selecting cards one at a time and placing them in their proper order
Selection Sort The selection sort follows this idea Given a list of data to be sorted
we simply select the smallest item and place it in a sorted list
We then repeat these steps until the list is sorted
Selection Sort In the selection sort the list at any moment
is divided into 2 sublists sorted and unsorted separated by a ldquoconceptualrdquo wall
We select the smallest element from the unsorted sublist and exchange it with the element at the beginning of the unsorted data
After each selection and exchange the wall between the 2 sublists moves ndash increasing the number of sorted elements and decreasing the number of unsorted elements
Selection Sort
We start with an unsorted list We search this list for thesmallest element We then exchange the smallest element (8)with the first element in the unsorted list (23)
Again we search the unsorted list for the smallest element We then exchange the smallest element (23) with the first element in the unsorted list (78)
This process continues until the list is fully sorted
Bubble Sort In the bubble sort the list at any
moment is divided into 2 sublists sorted and unsorted
The smallest element is ldquobubbledrdquo from the unsorted sublist to the sorted sublist
Bubble Sort
23 78 45 8 56 3232 56
We start with 32 and compareit with 56 Because 32 is lessthan 56 we swap the twoand step down one element
We then compare 32 and 8Because 32 is not less than 8we do not swap these elements
We step down one element andcompare 45 and 8 They areout of sequence so we swapthem and step down again
8 45
We step down again and compare 8 with 78 These two elements are swapped
8 78
Finally 8 is compared with 23and swapped We thencontinue this process back with56 hellip
8 23
1 Pass of the Bubble Sort
Quick Sort In the bubble sort consecutive items
are compared and possibly exchanged on each pass through the list
This means that many exchanges may be needed to move an element to its correct position
Quick sort is more efficient than bubble sort because a typical exchange involves elements that are far apart so fewer exchanges are required to correctly position an element
Idea of Quick Sort1) Select pick an element
2) Divide rearrange elements so that x goes to its final position E
3) Recurse and Conquer recursively sort
Quick Sort Each iteration of the quick sort
selects an element known as the pivot and divides the list into 3 groups Elements whose keys are less than (or
equal to) the pivotrsquos key The pivot element Elements whose keys are greater than
(or equal to) the pivotrsquos key
Quick Sort The sorting then continues by
quick sorting the left partition followed by quick sorting the right partition
The basic algorithm is as follows
Quick Sort1) Partitioning Step Take an element in the
unsorted array and determine its final location in the sorted array This occurs when all values to the left of the element in the array are less than (or equal to) the element and all values to the right of the element are greater than (or equal to) the element We now have 1 element in its proper location and two unsorted subarrays
2) Recursive Step Perform step 1 on each unsorted subarray
Quick Sort Each time step 1 is performed on a
subarray another element is placed in its final location of the sorted array and two unsorted subarrays are created
When a subarray consists of one element that subarray is sorted
Therefore that element is in its final location
Quick Sort There are several partitioning strategies
used in practice (ie several ldquoversionsrdquo of quick sort) but the one we are about to describe is known to work well
For simplicity we will choose the last element to be the pivot element
We could also chose a different pivot element and swap it with the last element in the array
Quick Sort Below is the array we would like to
sort1 4 8 9 0 11 5 10 7 6
Quick Sort The index left starts at the first element
and right starts at the next-to-last element
We want to move all the elements smaller than the pivot to the left part of the array and all the elements larger than the pivot to the right part
1 4 8 9 0 11 5 10 7 6left righ
t
Quick Sort We move left to the right skipping
over elements that are smaller than the pivot
1 4 8 9 0 11 5 10 7 6left righ
t
Quick Sort We then move right to the left skipping
over elements that are greater than the pivot
When left and right have stopped left is on an element greater than (or equal to) the pivot and right is on an element smaller than (or equal to) the pivot
1 4 8 9 0 11 5 10 7 6left right
Quick Sort If left is to the left of right (or if left
= right) those elements are swapped
1 4 8 9 0 11 5 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
Quick Sort The effect is to push a large
element to the right and a small element to the left
We then repeat the process until left and right cross
Quick Sort
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6left righ
t
Quick Sort
1 4 5 0 9 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6righ
tleft
Quick Sort
At this point left and right have crossed so no swap is performed
The final part of the partitioning is to swap the pivot element with left
1 4 5 0 9 11 8 10 7 6righ
tleft
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort Note that all elements to the left of
the pivot are less than (or equal to) the pivot and all elements to the right of the pivot are greater than (or equal to) the pivot
Hence the pivot element has been placed in its final sorted position
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort We now repeat the process using
the sub-arrays to the left and right of the pivot
1 4 5 0 6 11 8 10 7 9
1 4 5 0 11 8 10 7 9
Analysis of quicksortmdashbest case
Suppose each partition operation divides the array almost exactly in half
Then the depth of the recursion in log2n Because thatrsquos how many times we can
halve n However there are many recursions at
each level How can we figure this out We note that
Each partition is linear over its subarray All the partitions at one level cover the array
Partitioning at various levels
Best case II We cut the array size in half each time So the depth of the recursion in log2n At each level of the recursion all the
partitions at that level do work that is linear in n
O(log2n) O(n) = O(n log2n)
What about the worst case
Worst case In the worst case partitioning always
divides the size n array into these three parts A length one part containing the pivot itself A length zero part and A length n-1 part containing everything else
We donrsquot recur on the zero-length part Recurring on the length n-1 part requires
(in the worst case) recurring to depth n-1
Worst case partitioning
Worst case for quicksort In the worst case recursion may be
O(n) levels deep (for an array of size n)
But the partitioning work done at each level is still O(n)
O(n) O(n) = O(n2) So worst case for Quicksort is O(n2) When does this happen
When the array is sorted to begin with
Typical case for quicksort If the array is sorted to begin with
Quicksort is terrible O(n2) It is possible to construct other bad
cases However Quicksort is usually O(n log2n) The constants are so good that Quicksort
is generally the fastest algorithm known Most real-world sorting is done by
Quicksort
Tweaking Quicksort Almost anything you can try to
ldquoimproverdquo Quicksort will actually slow it down
One good tweak is to switch to a different sorting method when the subarrays get small (say 10 or 12) Quicksort has too much overhead for small
array sizes For large arrays it might be a good
idea to check beforehand if the array is already sorted But there is a better tweak than this
Randomized Quick-Sort Select the pivot as a random element of the sequence The expected running time of randomized quick-sort on a
sequence of size n is O(n log n) The time spent at a level of the quick-sort tree is O(n) We show that the expected height of the quick-sort tree is
O(log n) good vs bad pivots
bull The probability of a good pivot is 12 thus we expect k2 good pivots out of k pivotsbull After a good pivot the size of each child sequence is at most 34 the size of the parent sequencebull After h pivots we expect (34)h2 n elementsbull The expected height h of the quick-sort tree is at most 2 log43n
Median of three Obviously it doesnrsquot make sense to sort the
array in order to find the median to use as a pivot
Instead compare just three elements of our (sub)arraymdashthe first the last and the middle Take the median (middle value) of these three as
pivot Itrsquos possible (but not easy) to construct cases
which will make this technique O(n2) Suppose we rearrange (sort) these three
numbers so that the smallest is in the first position the largest in the last position and the other in the middle This lets us simplify and speed up the partition
loop
Final comments Weissrsquos code shows some additional
optimizations on pp 246-247 Weiss chooses to stop both searches on
equality to pivot This design decision is debatable
Quicksort is the fastest known general sorting algorithm on average
For optimum speed the pivot must be chosen carefully
ldquoMedian of threerdquo is a good technique for choosing the pivot
There will be some cases where Quicksort runs in O(n2) time
Quick Sort A couple of notes about quick sort There are more optimal ways to
choose the pivot value (such as the median-of-three method)
Also when the subarrays get small it becomes more efficient to use the insertion sort as opposed to continued use of quick sort
Bubble Sort vs Quick Sort If we calculate the Big-O notation
we find that (in the average case) Bubble Sort O(n2) Quick Sort O(nlog2n)
(quick-sort runs in time O(n2) in the worst case)
Heap Sort Idea take the items that need to be
sorted and insert them into the heap By calling deleteHeap we remove the
smallest or largest element depending on whether or not we are working with a min- or max-heap respectively
Hence the elements are removed in ascending or descending order
Efficiency O(nlog2n)
Merge Sort Idea
Take the array you would like to sort and divide it in half to create 2 unsorted subarrays
Next sort each of the 2 subarrays Finally merge the 2 sorted subarrays
into 1 sorted array Efficiency O(nlog2n)
Merge Sort
Merge Sort Although the merge step produces
a sorted array we have overlooked a very important step
How did we sort the 2 halves before performing the merge step
We used merge sort
Merge Sort By continually calling the merge
sort algorithm we eventually get a subarray of size 1
Since an array with only 1 element is clearly sorted we can back out and merge 2 arrays of size 1
Merge Sort
Merge Sort The basic merging algorithm
consists of 2 input arrays (arrayA and arrayB) An ouput array (arrayC) 3 position holders (indexA indexB
indexC) which are initially set to the beginning of their respective arrays
Merge Sort The smaller of arrayA[indexA] and
arrayB[indexB] is copied into arrayC[indexC] and the appropriate position holders are advanced
When either input list is exhausted the remainder of the other list is copied into arrayC
Merge Sort
1 13 24 26index
A
indexC
arrayA
2 15 27 38index
B
arrayB
arrayC
We compare arrayA[indexA]with arrayB[indexB] Whichever value is smaller isplaced into arrayC[indexC]
1 lt 2 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1index
C
arrayA
2 15 27 38index
B
arrayB
arrayC
2 lt 13 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
13 lt 15 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
15 lt 24 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13 15indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
24 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
indexA
1 2 13 15 24indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
26 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
1 2 13 15 24 26indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
Since we have exhaustedone of the arrays arrayAwe simply copy the remaining items from theother array arrayB intoarrayC
Merge Sort
1 13 24
26
1 2 13 15 24 26 27 38
arrayA
2 15 27 38arrayB
arrayC
Efficiency Summary
Sort Worst Case Average Case
Insertion O(n2) O(n2)
Shell O(n15) O(n125)
Selection O(n2) O(n2)
Bubble O(n2) O(n2)
Quick O(n2) O(nlog2n)
Heap O(nlog2n) O(nlog2n)
Merge O(nlog2n) O(nlog2n)
More Sorting
radix sort bucket sort in-place sorting how fast can we sort
Radix Sort
Unlike other sorting methods radix sort considers the structure of the keys
Assume keys are represented in a base M number system (M = radix) ie if M = 2 the keys are represented in binary
Sorting is done by comparing bits in the same position
Extension to keys that are alphanumeric strings
Radix Exchange Sort
Examine bits from left to right1 Sort array with respect to leftmost bit
2 Partition array
3Recursionrecursively sort top subarray ignoring leftmost bitrecursively sort bottom subarray ignoring leftmost bit
Time to sort n b-bit numbers O(b n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Shell SortPass Notes
6 21 62 14 9 30 77 80 25 70 55 Swap7 14 62 21 9 30 77 80 25 70 55 Swap8 14 9 21 62 30 77 80 25 70 55 In order9 14 9 21 62 30 77 80 25 70 55 In order10 14 9 21 62 30 77 80 25 70 55 In order11 14 9 21 62 30 77 80 25 70 55 Swap
14 9 21 62 30 25 80 77 70 55 Swap14 9 21 25 30 62 80 77 70 55 In order
12 14 9 21 25 30 62 80 77 70 55 Swap14 9 21 25 30 62 70 77 80 55 In order
13 14 9 21 25 30 62 70 77 80 55 Swap14 9 21 25 30 62 70 55 80 77 Swap14 9 21 25 30 55 70 62 80 77 In order
List (K=2)
Here we have reduced K to 2 Just as in the insertion sort if we swap 2 values we have to go back and compare theprevious 2 values to make sure they are still in order
Shell Sort
All shell sorts will terminate by running an insertion sort(ie K=1) However using the larger values of K first has helped to sort our list so that the straight insertion sort will runfaster
Shell Sort There are many schools of thought
on what the increment should be in the shell sort
Also note that just because an increment is optimal on one list it might not be optimal for another list
Insertion Sort vs Shell Sort Comparing the Big-O notation (for the
average case) we find that Insertion O(n2) Shell O(n125) empirically determined
Although this doesnrsquot seem like much of a gain it makes a big difference as n gets large
Note that in the worst case the Shell sort has an efficiency of O(n2)
However using a special incrementing technique this worst case can be reduced to O(n15)
Insertion Sort vs Shell Sort(Average Case)
Straight Insertion Shell25 625 55100 10000 316500 250000 2364
1000 1000000 56232000 4000000 13374
Number of Loopsn
Selection Sort Imagine some data that you can examine
all at once To sort it you could select the smallest
element and put it in its place select the next smallest and put it in its place etc
For a card player this process is analogous to looking at an entire hand of cards and ordering them by selecting cards one at a time and placing them in their proper order
Selection Sort The selection sort follows this idea Given a list of data to be sorted
we simply select the smallest item and place it in a sorted list
We then repeat these steps until the list is sorted
Selection Sort In the selection sort the list at any moment
is divided into 2 sublists sorted and unsorted separated by a ldquoconceptualrdquo wall
We select the smallest element from the unsorted sublist and exchange it with the element at the beginning of the unsorted data
After each selection and exchange the wall between the 2 sublists moves ndash increasing the number of sorted elements and decreasing the number of unsorted elements
Selection Sort
We start with an unsorted list We search this list for thesmallest element We then exchange the smallest element (8)with the first element in the unsorted list (23)
Again we search the unsorted list for the smallest element We then exchange the smallest element (23) with the first element in the unsorted list (78)
This process continues until the list is fully sorted
Bubble Sort In the bubble sort the list at any
moment is divided into 2 sublists sorted and unsorted
The smallest element is ldquobubbledrdquo from the unsorted sublist to the sorted sublist
Bubble Sort
23 78 45 8 56 3232 56
We start with 32 and compareit with 56 Because 32 is lessthan 56 we swap the twoand step down one element
We then compare 32 and 8Because 32 is not less than 8we do not swap these elements
We step down one element andcompare 45 and 8 They areout of sequence so we swapthem and step down again
8 45
We step down again and compare 8 with 78 These two elements are swapped
8 78
Finally 8 is compared with 23and swapped We thencontinue this process back with56 hellip
8 23
1 Pass of the Bubble Sort
Quick Sort In the bubble sort consecutive items
are compared and possibly exchanged on each pass through the list
This means that many exchanges may be needed to move an element to its correct position
Quick sort is more efficient than bubble sort because a typical exchange involves elements that are far apart so fewer exchanges are required to correctly position an element
Idea of Quick Sort1) Select pick an element
2) Divide rearrange elements so that x goes to its final position E
3) Recurse and Conquer recursively sort
Quick Sort Each iteration of the quick sort
selects an element known as the pivot and divides the list into 3 groups Elements whose keys are less than (or
equal to) the pivotrsquos key The pivot element Elements whose keys are greater than
(or equal to) the pivotrsquos key
Quick Sort The sorting then continues by
quick sorting the left partition followed by quick sorting the right partition
The basic algorithm is as follows
Quick Sort1) Partitioning Step Take an element in the
unsorted array and determine its final location in the sorted array This occurs when all values to the left of the element in the array are less than (or equal to) the element and all values to the right of the element are greater than (or equal to) the element We now have 1 element in its proper location and two unsorted subarrays
2) Recursive Step Perform step 1 on each unsorted subarray
Quick Sort Each time step 1 is performed on a
subarray another element is placed in its final location of the sorted array and two unsorted subarrays are created
When a subarray consists of one element that subarray is sorted
Therefore that element is in its final location
Quick Sort There are several partitioning strategies
used in practice (ie several ldquoversionsrdquo of quick sort) but the one we are about to describe is known to work well
For simplicity we will choose the last element to be the pivot element
We could also chose a different pivot element and swap it with the last element in the array
Quick Sort Below is the array we would like to
sort1 4 8 9 0 11 5 10 7 6
Quick Sort The index left starts at the first element
and right starts at the next-to-last element
We want to move all the elements smaller than the pivot to the left part of the array and all the elements larger than the pivot to the right part
1 4 8 9 0 11 5 10 7 6left righ
t
Quick Sort We move left to the right skipping
over elements that are smaller than the pivot
1 4 8 9 0 11 5 10 7 6left righ
t
Quick Sort We then move right to the left skipping
over elements that are greater than the pivot
When left and right have stopped left is on an element greater than (or equal to) the pivot and right is on an element smaller than (or equal to) the pivot
1 4 8 9 0 11 5 10 7 6left right
Quick Sort If left is to the left of right (or if left
= right) those elements are swapped
1 4 8 9 0 11 5 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
Quick Sort The effect is to push a large
element to the right and a small element to the left
We then repeat the process until left and right cross
Quick Sort
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6left righ
t
Quick Sort
1 4 5 0 9 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6righ
tleft
Quick Sort
At this point left and right have crossed so no swap is performed
The final part of the partitioning is to swap the pivot element with left
1 4 5 0 9 11 8 10 7 6righ
tleft
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort Note that all elements to the left of
the pivot are less than (or equal to) the pivot and all elements to the right of the pivot are greater than (or equal to) the pivot
Hence the pivot element has been placed in its final sorted position
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort We now repeat the process using
the sub-arrays to the left and right of the pivot
1 4 5 0 6 11 8 10 7 9
1 4 5 0 11 8 10 7 9
Analysis of quicksortmdashbest case
Suppose each partition operation divides the array almost exactly in half
Then the depth of the recursion in log2n Because thatrsquos how many times we can
halve n However there are many recursions at
each level How can we figure this out We note that
Each partition is linear over its subarray All the partitions at one level cover the array
Partitioning at various levels
Best case II We cut the array size in half each time So the depth of the recursion in log2n At each level of the recursion all the
partitions at that level do work that is linear in n
O(log2n) O(n) = O(n log2n)
What about the worst case
Worst case In the worst case partitioning always
divides the size n array into these three parts A length one part containing the pivot itself A length zero part and A length n-1 part containing everything else
We donrsquot recur on the zero-length part Recurring on the length n-1 part requires
(in the worst case) recurring to depth n-1
Worst case partitioning
Worst case for quicksort In the worst case recursion may be
O(n) levels deep (for an array of size n)
But the partitioning work done at each level is still O(n)
O(n) O(n) = O(n2) So worst case for Quicksort is O(n2) When does this happen
When the array is sorted to begin with
Typical case for quicksort If the array is sorted to begin with
Quicksort is terrible O(n2) It is possible to construct other bad
cases However Quicksort is usually O(n log2n) The constants are so good that Quicksort
is generally the fastest algorithm known Most real-world sorting is done by
Quicksort
Tweaking Quicksort Almost anything you can try to
ldquoimproverdquo Quicksort will actually slow it down
One good tweak is to switch to a different sorting method when the subarrays get small (say 10 or 12) Quicksort has too much overhead for small
array sizes For large arrays it might be a good
idea to check beforehand if the array is already sorted But there is a better tweak than this
Randomized Quick-Sort Select the pivot as a random element of the sequence The expected running time of randomized quick-sort on a
sequence of size n is O(n log n) The time spent at a level of the quick-sort tree is O(n) We show that the expected height of the quick-sort tree is
O(log n) good vs bad pivots
bull The probability of a good pivot is 12 thus we expect k2 good pivots out of k pivotsbull After a good pivot the size of each child sequence is at most 34 the size of the parent sequencebull After h pivots we expect (34)h2 n elementsbull The expected height h of the quick-sort tree is at most 2 log43n
Median of three Obviously it doesnrsquot make sense to sort the
array in order to find the median to use as a pivot
Instead compare just three elements of our (sub)arraymdashthe first the last and the middle Take the median (middle value) of these three as
pivot Itrsquos possible (but not easy) to construct cases
which will make this technique O(n2) Suppose we rearrange (sort) these three
numbers so that the smallest is in the first position the largest in the last position and the other in the middle This lets us simplify and speed up the partition
loop
Final comments Weissrsquos code shows some additional
optimizations on pp 246-247 Weiss chooses to stop both searches on
equality to pivot This design decision is debatable
Quicksort is the fastest known general sorting algorithm on average
For optimum speed the pivot must be chosen carefully
ldquoMedian of threerdquo is a good technique for choosing the pivot
There will be some cases where Quicksort runs in O(n2) time
Quick Sort A couple of notes about quick sort There are more optimal ways to
choose the pivot value (such as the median-of-three method)
Also when the subarrays get small it becomes more efficient to use the insertion sort as opposed to continued use of quick sort
Bubble Sort vs Quick Sort If we calculate the Big-O notation
we find that (in the average case) Bubble Sort O(n2) Quick Sort O(nlog2n)
(quick-sort runs in time O(n2) in the worst case)
Heap Sort Idea take the items that need to be
sorted and insert them into the heap By calling deleteHeap we remove the
smallest or largest element depending on whether or not we are working with a min- or max-heap respectively
Hence the elements are removed in ascending or descending order
Efficiency O(nlog2n)
Merge Sort Idea
Take the array you would like to sort and divide it in half to create 2 unsorted subarrays
Next sort each of the 2 subarrays Finally merge the 2 sorted subarrays
into 1 sorted array Efficiency O(nlog2n)
Merge Sort
Merge Sort Although the merge step produces
a sorted array we have overlooked a very important step
How did we sort the 2 halves before performing the merge step
We used merge sort
Merge Sort By continually calling the merge
sort algorithm we eventually get a subarray of size 1
Since an array with only 1 element is clearly sorted we can back out and merge 2 arrays of size 1
Merge Sort
Merge Sort The basic merging algorithm
consists of 2 input arrays (arrayA and arrayB) An ouput array (arrayC) 3 position holders (indexA indexB
indexC) which are initially set to the beginning of their respective arrays
Merge Sort The smaller of arrayA[indexA] and
arrayB[indexB] is copied into arrayC[indexC] and the appropriate position holders are advanced
When either input list is exhausted the remainder of the other list is copied into arrayC
Merge Sort
1 13 24 26index
A
indexC
arrayA
2 15 27 38index
B
arrayB
arrayC
We compare arrayA[indexA]with arrayB[indexB] Whichever value is smaller isplaced into arrayC[indexC]
1 lt 2 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1index
C
arrayA
2 15 27 38index
B
arrayB
arrayC
2 lt 13 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
13 lt 15 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
15 lt 24 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13 15indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
24 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
indexA
1 2 13 15 24indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
26 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
1 2 13 15 24 26indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
Since we have exhaustedone of the arrays arrayAwe simply copy the remaining items from theother array arrayB intoarrayC
Merge Sort
1 13 24
26
1 2 13 15 24 26 27 38
arrayA
2 15 27 38arrayB
arrayC
Efficiency Summary
Sort Worst Case Average Case
Insertion O(n2) O(n2)
Shell O(n15) O(n125)
Selection O(n2) O(n2)
Bubble O(n2) O(n2)
Quick O(n2) O(nlog2n)
Heap O(nlog2n) O(nlog2n)
Merge O(nlog2n) O(nlog2n)
More Sorting
radix sort bucket sort in-place sorting how fast can we sort
Radix Sort
Unlike other sorting methods radix sort considers the structure of the keys
Assume keys are represented in a base M number system (M = radix) ie if M = 2 the keys are represented in binary
Sorting is done by comparing bits in the same position
Extension to keys that are alphanumeric strings
Radix Exchange Sort
Examine bits from left to right1 Sort array with respect to leftmost bit
2 Partition array
3Recursionrecursively sort top subarray ignoring leftmost bitrecursively sort bottom subarray ignoring leftmost bit
Time to sort n b-bit numbers O(b n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Shell Sort
All shell sorts will terminate by running an insertion sort(ie K=1) However using the larger values of K first has helped to sort our list so that the straight insertion sort will runfaster
Shell Sort There are many schools of thought
on what the increment should be in the shell sort
Also note that just because an increment is optimal on one list it might not be optimal for another list
Insertion Sort vs Shell Sort Comparing the Big-O notation (for the
average case) we find that Insertion O(n2) Shell O(n125) empirically determined
Although this doesnrsquot seem like much of a gain it makes a big difference as n gets large
Note that in the worst case the Shell sort has an efficiency of O(n2)
However using a special incrementing technique this worst case can be reduced to O(n15)
Insertion Sort vs Shell Sort(Average Case)
Straight Insertion Shell25 625 55100 10000 316500 250000 2364
1000 1000000 56232000 4000000 13374
Number of Loopsn
Selection Sort Imagine some data that you can examine
all at once To sort it you could select the smallest
element and put it in its place select the next smallest and put it in its place etc
For a card player this process is analogous to looking at an entire hand of cards and ordering them by selecting cards one at a time and placing them in their proper order
Selection Sort The selection sort follows this idea Given a list of data to be sorted
we simply select the smallest item and place it in a sorted list
We then repeat these steps until the list is sorted
Selection Sort In the selection sort the list at any moment
is divided into 2 sublists sorted and unsorted separated by a ldquoconceptualrdquo wall
We select the smallest element from the unsorted sublist and exchange it with the element at the beginning of the unsorted data
After each selection and exchange the wall between the 2 sublists moves ndash increasing the number of sorted elements and decreasing the number of unsorted elements
Selection Sort
We start with an unsorted list We search this list for thesmallest element We then exchange the smallest element (8)with the first element in the unsorted list (23)
Again we search the unsorted list for the smallest element We then exchange the smallest element (23) with the first element in the unsorted list (78)
This process continues until the list is fully sorted
Bubble Sort In the bubble sort the list at any
moment is divided into 2 sublists sorted and unsorted
The smallest element is ldquobubbledrdquo from the unsorted sublist to the sorted sublist
Bubble Sort
23 78 45 8 56 3232 56
We start with 32 and compareit with 56 Because 32 is lessthan 56 we swap the twoand step down one element
We then compare 32 and 8Because 32 is not less than 8we do not swap these elements
We step down one element andcompare 45 and 8 They areout of sequence so we swapthem and step down again
8 45
We step down again and compare 8 with 78 These two elements are swapped
8 78
Finally 8 is compared with 23and swapped We thencontinue this process back with56 hellip
8 23
1 Pass of the Bubble Sort
Quick Sort In the bubble sort consecutive items
are compared and possibly exchanged on each pass through the list
This means that many exchanges may be needed to move an element to its correct position
Quick sort is more efficient than bubble sort because a typical exchange involves elements that are far apart so fewer exchanges are required to correctly position an element
Idea of Quick Sort1) Select pick an element
2) Divide rearrange elements so that x goes to its final position E
3) Recurse and Conquer recursively sort
Quick Sort Each iteration of the quick sort
selects an element known as the pivot and divides the list into 3 groups Elements whose keys are less than (or
equal to) the pivotrsquos key The pivot element Elements whose keys are greater than
(or equal to) the pivotrsquos key
Quick Sort The sorting then continues by
quick sorting the left partition followed by quick sorting the right partition
The basic algorithm is as follows
Quick Sort1) Partitioning Step Take an element in the
unsorted array and determine its final location in the sorted array This occurs when all values to the left of the element in the array are less than (or equal to) the element and all values to the right of the element are greater than (or equal to) the element We now have 1 element in its proper location and two unsorted subarrays
2) Recursive Step Perform step 1 on each unsorted subarray
Quick Sort Each time step 1 is performed on a
subarray another element is placed in its final location of the sorted array and two unsorted subarrays are created
When a subarray consists of one element that subarray is sorted
Therefore that element is in its final location
Quick Sort There are several partitioning strategies
used in practice (ie several ldquoversionsrdquo of quick sort) but the one we are about to describe is known to work well
For simplicity we will choose the last element to be the pivot element
We could also chose a different pivot element and swap it with the last element in the array
Quick Sort Below is the array we would like to
sort1 4 8 9 0 11 5 10 7 6
Quick Sort The index left starts at the first element
and right starts at the next-to-last element
We want to move all the elements smaller than the pivot to the left part of the array and all the elements larger than the pivot to the right part
1 4 8 9 0 11 5 10 7 6left righ
t
Quick Sort We move left to the right skipping
over elements that are smaller than the pivot
1 4 8 9 0 11 5 10 7 6left righ
t
Quick Sort We then move right to the left skipping
over elements that are greater than the pivot
When left and right have stopped left is on an element greater than (or equal to) the pivot and right is on an element smaller than (or equal to) the pivot
1 4 8 9 0 11 5 10 7 6left right
Quick Sort If left is to the left of right (or if left
= right) those elements are swapped
1 4 8 9 0 11 5 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
Quick Sort The effect is to push a large
element to the right and a small element to the left
We then repeat the process until left and right cross
Quick Sort
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6left righ
t
Quick Sort
1 4 5 0 9 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6righ
tleft
Quick Sort
At this point left and right have crossed so no swap is performed
The final part of the partitioning is to swap the pivot element with left
1 4 5 0 9 11 8 10 7 6righ
tleft
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort Note that all elements to the left of
the pivot are less than (or equal to) the pivot and all elements to the right of the pivot are greater than (or equal to) the pivot
Hence the pivot element has been placed in its final sorted position
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort We now repeat the process using
the sub-arrays to the left and right of the pivot
1 4 5 0 6 11 8 10 7 9
1 4 5 0 11 8 10 7 9
Analysis of quicksortmdashbest case
Suppose each partition operation divides the array almost exactly in half
Then the depth of the recursion in log2n Because thatrsquos how many times we can
halve n However there are many recursions at
each level How can we figure this out We note that
Each partition is linear over its subarray All the partitions at one level cover the array
Partitioning at various levels
Best case II We cut the array size in half each time So the depth of the recursion in log2n At each level of the recursion all the
partitions at that level do work that is linear in n
O(log2n) O(n) = O(n log2n)
What about the worst case
Worst case In the worst case partitioning always
divides the size n array into these three parts A length one part containing the pivot itself A length zero part and A length n-1 part containing everything else
We donrsquot recur on the zero-length part Recurring on the length n-1 part requires
(in the worst case) recurring to depth n-1
Worst case partitioning
Worst case for quicksort In the worst case recursion may be
O(n) levels deep (for an array of size n)
But the partitioning work done at each level is still O(n)
O(n) O(n) = O(n2) So worst case for Quicksort is O(n2) When does this happen
When the array is sorted to begin with
Typical case for quicksort If the array is sorted to begin with
Quicksort is terrible O(n2) It is possible to construct other bad
cases However Quicksort is usually O(n log2n) The constants are so good that Quicksort
is generally the fastest algorithm known Most real-world sorting is done by
Quicksort
Tweaking Quicksort Almost anything you can try to
ldquoimproverdquo Quicksort will actually slow it down
One good tweak is to switch to a different sorting method when the subarrays get small (say 10 or 12) Quicksort has too much overhead for small
array sizes For large arrays it might be a good
idea to check beforehand if the array is already sorted But there is a better tweak than this
Randomized Quick-Sort Select the pivot as a random element of the sequence The expected running time of randomized quick-sort on a
sequence of size n is O(n log n) The time spent at a level of the quick-sort tree is O(n) We show that the expected height of the quick-sort tree is
O(log n) good vs bad pivots
bull The probability of a good pivot is 12 thus we expect k2 good pivots out of k pivotsbull After a good pivot the size of each child sequence is at most 34 the size of the parent sequencebull After h pivots we expect (34)h2 n elementsbull The expected height h of the quick-sort tree is at most 2 log43n
Median of three Obviously it doesnrsquot make sense to sort the
array in order to find the median to use as a pivot
Instead compare just three elements of our (sub)arraymdashthe first the last and the middle Take the median (middle value) of these three as
pivot Itrsquos possible (but not easy) to construct cases
which will make this technique O(n2) Suppose we rearrange (sort) these three
numbers so that the smallest is in the first position the largest in the last position and the other in the middle This lets us simplify and speed up the partition
loop
Final comments Weissrsquos code shows some additional
optimizations on pp 246-247 Weiss chooses to stop both searches on
equality to pivot This design decision is debatable
Quicksort is the fastest known general sorting algorithm on average
For optimum speed the pivot must be chosen carefully
ldquoMedian of threerdquo is a good technique for choosing the pivot
There will be some cases where Quicksort runs in O(n2) time
Quick Sort A couple of notes about quick sort There are more optimal ways to
choose the pivot value (such as the median-of-three method)
Also when the subarrays get small it becomes more efficient to use the insertion sort as opposed to continued use of quick sort
Bubble Sort vs Quick Sort If we calculate the Big-O notation
we find that (in the average case) Bubble Sort O(n2) Quick Sort O(nlog2n)
(quick-sort runs in time O(n2) in the worst case)
Heap Sort Idea take the items that need to be
sorted and insert them into the heap By calling deleteHeap we remove the
smallest or largest element depending on whether or not we are working with a min- or max-heap respectively
Hence the elements are removed in ascending or descending order
Efficiency O(nlog2n)
Merge Sort Idea
Take the array you would like to sort and divide it in half to create 2 unsorted subarrays
Next sort each of the 2 subarrays Finally merge the 2 sorted subarrays
into 1 sorted array Efficiency O(nlog2n)
Merge Sort
Merge Sort Although the merge step produces
a sorted array we have overlooked a very important step
How did we sort the 2 halves before performing the merge step
We used merge sort
Merge Sort By continually calling the merge
sort algorithm we eventually get a subarray of size 1
Since an array with only 1 element is clearly sorted we can back out and merge 2 arrays of size 1
Merge Sort
Merge Sort The basic merging algorithm
consists of 2 input arrays (arrayA and arrayB) An ouput array (arrayC) 3 position holders (indexA indexB
indexC) which are initially set to the beginning of their respective arrays
Merge Sort The smaller of arrayA[indexA] and
arrayB[indexB] is copied into arrayC[indexC] and the appropriate position holders are advanced
When either input list is exhausted the remainder of the other list is copied into arrayC
Merge Sort
1 13 24 26index
A
indexC
arrayA
2 15 27 38index
B
arrayB
arrayC
We compare arrayA[indexA]with arrayB[indexB] Whichever value is smaller isplaced into arrayC[indexC]
1 lt 2 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1index
C
arrayA
2 15 27 38index
B
arrayB
arrayC
2 lt 13 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
13 lt 15 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
15 lt 24 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13 15indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
24 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
indexA
1 2 13 15 24indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
26 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
1 2 13 15 24 26indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
Since we have exhaustedone of the arrays arrayAwe simply copy the remaining items from theother array arrayB intoarrayC
Merge Sort
1 13 24
26
1 2 13 15 24 26 27 38
arrayA
2 15 27 38arrayB
arrayC
Efficiency Summary
Sort Worst Case Average Case
Insertion O(n2) O(n2)
Shell O(n15) O(n125)
Selection O(n2) O(n2)
Bubble O(n2) O(n2)
Quick O(n2) O(nlog2n)
Heap O(nlog2n) O(nlog2n)
Merge O(nlog2n) O(nlog2n)
More Sorting
radix sort bucket sort in-place sorting how fast can we sort
Radix Sort
Unlike other sorting methods radix sort considers the structure of the keys
Assume keys are represented in a base M number system (M = radix) ie if M = 2 the keys are represented in binary
Sorting is done by comparing bits in the same position
Extension to keys that are alphanumeric strings
Radix Exchange Sort
Examine bits from left to right1 Sort array with respect to leftmost bit
2 Partition array
3Recursionrecursively sort top subarray ignoring leftmost bitrecursively sort bottom subarray ignoring leftmost bit
Time to sort n b-bit numbers O(b n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Shell Sort There are many schools of thought
on what the increment should be in the shell sort
Also note that just because an increment is optimal on one list it might not be optimal for another list
Insertion Sort vs Shell Sort Comparing the Big-O notation (for the
average case) we find that Insertion O(n2) Shell O(n125) empirically determined
Although this doesnrsquot seem like much of a gain it makes a big difference as n gets large
Note that in the worst case the Shell sort has an efficiency of O(n2)
However using a special incrementing technique this worst case can be reduced to O(n15)
Insertion Sort vs Shell Sort(Average Case)
Straight Insertion Shell25 625 55100 10000 316500 250000 2364
1000 1000000 56232000 4000000 13374
Number of Loopsn
Selection Sort Imagine some data that you can examine
all at once To sort it you could select the smallest
element and put it in its place select the next smallest and put it in its place etc
For a card player this process is analogous to looking at an entire hand of cards and ordering them by selecting cards one at a time and placing them in their proper order
Selection Sort The selection sort follows this idea Given a list of data to be sorted
we simply select the smallest item and place it in a sorted list
We then repeat these steps until the list is sorted
Selection Sort In the selection sort the list at any moment
is divided into 2 sublists sorted and unsorted separated by a ldquoconceptualrdquo wall
We select the smallest element from the unsorted sublist and exchange it with the element at the beginning of the unsorted data
After each selection and exchange the wall between the 2 sublists moves ndash increasing the number of sorted elements and decreasing the number of unsorted elements
Selection Sort
We start with an unsorted list We search this list for thesmallest element We then exchange the smallest element (8)with the first element in the unsorted list (23)
Again we search the unsorted list for the smallest element We then exchange the smallest element (23) with the first element in the unsorted list (78)
This process continues until the list is fully sorted
Bubble Sort In the bubble sort the list at any
moment is divided into 2 sublists sorted and unsorted
The smallest element is ldquobubbledrdquo from the unsorted sublist to the sorted sublist
Bubble Sort
23 78 45 8 56 3232 56
We start with 32 and compareit with 56 Because 32 is lessthan 56 we swap the twoand step down one element
We then compare 32 and 8Because 32 is not less than 8we do not swap these elements
We step down one element andcompare 45 and 8 They areout of sequence so we swapthem and step down again
8 45
We step down again and compare 8 with 78 These two elements are swapped
8 78
Finally 8 is compared with 23and swapped We thencontinue this process back with56 hellip
8 23
1 Pass of the Bubble Sort
Quick Sort In the bubble sort consecutive items
are compared and possibly exchanged on each pass through the list
This means that many exchanges may be needed to move an element to its correct position
Quick sort is more efficient than bubble sort because a typical exchange involves elements that are far apart so fewer exchanges are required to correctly position an element
Idea of Quick Sort1) Select pick an element
2) Divide rearrange elements so that x goes to its final position E
3) Recurse and Conquer recursively sort
Quick Sort Each iteration of the quick sort
selects an element known as the pivot and divides the list into 3 groups Elements whose keys are less than (or
equal to) the pivotrsquos key The pivot element Elements whose keys are greater than
(or equal to) the pivotrsquos key
Quick Sort The sorting then continues by
quick sorting the left partition followed by quick sorting the right partition
The basic algorithm is as follows
Quick Sort1) Partitioning Step Take an element in the
unsorted array and determine its final location in the sorted array This occurs when all values to the left of the element in the array are less than (or equal to) the element and all values to the right of the element are greater than (or equal to) the element We now have 1 element in its proper location and two unsorted subarrays
2) Recursive Step Perform step 1 on each unsorted subarray
Quick Sort Each time step 1 is performed on a
subarray another element is placed in its final location of the sorted array and two unsorted subarrays are created
When a subarray consists of one element that subarray is sorted
Therefore that element is in its final location
Quick Sort There are several partitioning strategies
used in practice (ie several ldquoversionsrdquo of quick sort) but the one we are about to describe is known to work well
For simplicity we will choose the last element to be the pivot element
We could also chose a different pivot element and swap it with the last element in the array
Quick Sort Below is the array we would like to
sort1 4 8 9 0 11 5 10 7 6
Quick Sort The index left starts at the first element
and right starts at the next-to-last element
We want to move all the elements smaller than the pivot to the left part of the array and all the elements larger than the pivot to the right part
1 4 8 9 0 11 5 10 7 6left righ
t
Quick Sort We move left to the right skipping
over elements that are smaller than the pivot
1 4 8 9 0 11 5 10 7 6left righ
t
Quick Sort We then move right to the left skipping
over elements that are greater than the pivot
When left and right have stopped left is on an element greater than (or equal to) the pivot and right is on an element smaller than (or equal to) the pivot
1 4 8 9 0 11 5 10 7 6left right
Quick Sort If left is to the left of right (or if left
= right) those elements are swapped
1 4 8 9 0 11 5 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
Quick Sort The effect is to push a large
element to the right and a small element to the left
We then repeat the process until left and right cross
Quick Sort
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6left righ
t
Quick Sort
1 4 5 0 9 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6righ
tleft
Quick Sort
At this point left and right have crossed so no swap is performed
The final part of the partitioning is to swap the pivot element with left
1 4 5 0 9 11 8 10 7 6righ
tleft
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort Note that all elements to the left of
the pivot are less than (or equal to) the pivot and all elements to the right of the pivot are greater than (or equal to) the pivot
Hence the pivot element has been placed in its final sorted position
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort We now repeat the process using
the sub-arrays to the left and right of the pivot
1 4 5 0 6 11 8 10 7 9
1 4 5 0 11 8 10 7 9
Analysis of quicksortmdashbest case
Suppose each partition operation divides the array almost exactly in half
Then the depth of the recursion in log2n Because thatrsquos how many times we can
halve n However there are many recursions at
each level How can we figure this out We note that
Each partition is linear over its subarray All the partitions at one level cover the array
Partitioning at various levels
Best case II We cut the array size in half each time So the depth of the recursion in log2n At each level of the recursion all the
partitions at that level do work that is linear in n
O(log2n) O(n) = O(n log2n)
What about the worst case
Worst case In the worst case partitioning always
divides the size n array into these three parts A length one part containing the pivot itself A length zero part and A length n-1 part containing everything else
We donrsquot recur on the zero-length part Recurring on the length n-1 part requires
(in the worst case) recurring to depth n-1
Worst case partitioning
Worst case for quicksort In the worst case recursion may be
O(n) levels deep (for an array of size n)
But the partitioning work done at each level is still O(n)
O(n) O(n) = O(n2) So worst case for Quicksort is O(n2) When does this happen
When the array is sorted to begin with
Typical case for quicksort If the array is sorted to begin with
Quicksort is terrible O(n2) It is possible to construct other bad
cases However Quicksort is usually O(n log2n) The constants are so good that Quicksort
is generally the fastest algorithm known Most real-world sorting is done by
Quicksort
Tweaking Quicksort Almost anything you can try to
ldquoimproverdquo Quicksort will actually slow it down
One good tweak is to switch to a different sorting method when the subarrays get small (say 10 or 12) Quicksort has too much overhead for small
array sizes For large arrays it might be a good
idea to check beforehand if the array is already sorted But there is a better tweak than this
Randomized Quick-Sort Select the pivot as a random element of the sequence The expected running time of randomized quick-sort on a
sequence of size n is O(n log n) The time spent at a level of the quick-sort tree is O(n) We show that the expected height of the quick-sort tree is
O(log n) good vs bad pivots
bull The probability of a good pivot is 12 thus we expect k2 good pivots out of k pivotsbull After a good pivot the size of each child sequence is at most 34 the size of the parent sequencebull After h pivots we expect (34)h2 n elementsbull The expected height h of the quick-sort tree is at most 2 log43n
Median of three Obviously it doesnrsquot make sense to sort the
array in order to find the median to use as a pivot
Instead compare just three elements of our (sub)arraymdashthe first the last and the middle Take the median (middle value) of these three as
pivot Itrsquos possible (but not easy) to construct cases
which will make this technique O(n2) Suppose we rearrange (sort) these three
numbers so that the smallest is in the first position the largest in the last position and the other in the middle This lets us simplify and speed up the partition
loop
Final comments Weissrsquos code shows some additional
optimizations on pp 246-247 Weiss chooses to stop both searches on
equality to pivot This design decision is debatable
Quicksort is the fastest known general sorting algorithm on average
For optimum speed the pivot must be chosen carefully
ldquoMedian of threerdquo is a good technique for choosing the pivot
There will be some cases where Quicksort runs in O(n2) time
Quick Sort A couple of notes about quick sort There are more optimal ways to
choose the pivot value (such as the median-of-three method)
Also when the subarrays get small it becomes more efficient to use the insertion sort as opposed to continued use of quick sort
Bubble Sort vs Quick Sort If we calculate the Big-O notation
we find that (in the average case) Bubble Sort O(n2) Quick Sort O(nlog2n)
(quick-sort runs in time O(n2) in the worst case)
Heap Sort Idea take the items that need to be
sorted and insert them into the heap By calling deleteHeap we remove the
smallest or largest element depending on whether or not we are working with a min- or max-heap respectively
Hence the elements are removed in ascending or descending order
Efficiency O(nlog2n)
Merge Sort Idea
Take the array you would like to sort and divide it in half to create 2 unsorted subarrays
Next sort each of the 2 subarrays Finally merge the 2 sorted subarrays
into 1 sorted array Efficiency O(nlog2n)
Merge Sort
Merge Sort Although the merge step produces
a sorted array we have overlooked a very important step
How did we sort the 2 halves before performing the merge step
We used merge sort
Merge Sort By continually calling the merge
sort algorithm we eventually get a subarray of size 1
Since an array with only 1 element is clearly sorted we can back out and merge 2 arrays of size 1
Merge Sort
Merge Sort The basic merging algorithm
consists of 2 input arrays (arrayA and arrayB) An ouput array (arrayC) 3 position holders (indexA indexB
indexC) which are initially set to the beginning of their respective arrays
Merge Sort The smaller of arrayA[indexA] and
arrayB[indexB] is copied into arrayC[indexC] and the appropriate position holders are advanced
When either input list is exhausted the remainder of the other list is copied into arrayC
Merge Sort
1 13 24 26index
A
indexC
arrayA
2 15 27 38index
B
arrayB
arrayC
We compare arrayA[indexA]with arrayB[indexB] Whichever value is smaller isplaced into arrayC[indexC]
1 lt 2 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1index
C
arrayA
2 15 27 38index
B
arrayB
arrayC
2 lt 13 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
13 lt 15 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
15 lt 24 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13 15indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
24 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
indexA
1 2 13 15 24indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
26 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
1 2 13 15 24 26indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
Since we have exhaustedone of the arrays arrayAwe simply copy the remaining items from theother array arrayB intoarrayC
Merge Sort
1 13 24
26
1 2 13 15 24 26 27 38
arrayA
2 15 27 38arrayB
arrayC
Efficiency Summary
Sort Worst Case Average Case
Insertion O(n2) O(n2)
Shell O(n15) O(n125)
Selection O(n2) O(n2)
Bubble O(n2) O(n2)
Quick O(n2) O(nlog2n)
Heap O(nlog2n) O(nlog2n)
Merge O(nlog2n) O(nlog2n)
More Sorting
radix sort bucket sort in-place sorting how fast can we sort
Radix Sort
Unlike other sorting methods radix sort considers the structure of the keys
Assume keys are represented in a base M number system (M = radix) ie if M = 2 the keys are represented in binary
Sorting is done by comparing bits in the same position
Extension to keys that are alphanumeric strings
Radix Exchange Sort
Examine bits from left to right1 Sort array with respect to leftmost bit
2 Partition array
3Recursionrecursively sort top subarray ignoring leftmost bitrecursively sort bottom subarray ignoring leftmost bit
Time to sort n b-bit numbers O(b n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Insertion Sort vs Shell Sort Comparing the Big-O notation (for the
average case) we find that Insertion O(n2) Shell O(n125) empirically determined
Although this doesnrsquot seem like much of a gain it makes a big difference as n gets large
Note that in the worst case the Shell sort has an efficiency of O(n2)
However using a special incrementing technique this worst case can be reduced to O(n15)
Insertion Sort vs Shell Sort(Average Case)
Straight Insertion Shell25 625 55100 10000 316500 250000 2364
1000 1000000 56232000 4000000 13374
Number of Loopsn
Selection Sort Imagine some data that you can examine
all at once To sort it you could select the smallest
element and put it in its place select the next smallest and put it in its place etc
For a card player this process is analogous to looking at an entire hand of cards and ordering them by selecting cards one at a time and placing them in their proper order
Selection Sort The selection sort follows this idea Given a list of data to be sorted
we simply select the smallest item and place it in a sorted list
We then repeat these steps until the list is sorted
Selection Sort In the selection sort the list at any moment
is divided into 2 sublists sorted and unsorted separated by a ldquoconceptualrdquo wall
We select the smallest element from the unsorted sublist and exchange it with the element at the beginning of the unsorted data
After each selection and exchange the wall between the 2 sublists moves ndash increasing the number of sorted elements and decreasing the number of unsorted elements
Selection Sort
We start with an unsorted list We search this list for thesmallest element We then exchange the smallest element (8)with the first element in the unsorted list (23)
Again we search the unsorted list for the smallest element We then exchange the smallest element (23) with the first element in the unsorted list (78)
This process continues until the list is fully sorted
Bubble Sort In the bubble sort the list at any
moment is divided into 2 sublists sorted and unsorted
The smallest element is ldquobubbledrdquo from the unsorted sublist to the sorted sublist
Bubble Sort
23 78 45 8 56 3232 56
We start with 32 and compareit with 56 Because 32 is lessthan 56 we swap the twoand step down one element
We then compare 32 and 8Because 32 is not less than 8we do not swap these elements
We step down one element andcompare 45 and 8 They areout of sequence so we swapthem and step down again
8 45
We step down again and compare 8 with 78 These two elements are swapped
8 78
Finally 8 is compared with 23and swapped We thencontinue this process back with56 hellip
8 23
1 Pass of the Bubble Sort
Quick Sort In the bubble sort consecutive items
are compared and possibly exchanged on each pass through the list
This means that many exchanges may be needed to move an element to its correct position
Quick sort is more efficient than bubble sort because a typical exchange involves elements that are far apart so fewer exchanges are required to correctly position an element
Idea of Quick Sort1) Select pick an element
2) Divide rearrange elements so that x goes to its final position E
3) Recurse and Conquer recursively sort
Quick Sort Each iteration of the quick sort
selects an element known as the pivot and divides the list into 3 groups Elements whose keys are less than (or
equal to) the pivotrsquos key The pivot element Elements whose keys are greater than
(or equal to) the pivotrsquos key
Quick Sort The sorting then continues by
quick sorting the left partition followed by quick sorting the right partition
The basic algorithm is as follows
Quick Sort1) Partitioning Step Take an element in the
unsorted array and determine its final location in the sorted array This occurs when all values to the left of the element in the array are less than (or equal to) the element and all values to the right of the element are greater than (or equal to) the element We now have 1 element in its proper location and two unsorted subarrays
2) Recursive Step Perform step 1 on each unsorted subarray
Quick Sort Each time step 1 is performed on a
subarray another element is placed in its final location of the sorted array and two unsorted subarrays are created
When a subarray consists of one element that subarray is sorted
Therefore that element is in its final location
Quick Sort There are several partitioning strategies
used in practice (ie several ldquoversionsrdquo of quick sort) but the one we are about to describe is known to work well
For simplicity we will choose the last element to be the pivot element
We could also chose a different pivot element and swap it with the last element in the array
Quick Sort Below is the array we would like to
sort1 4 8 9 0 11 5 10 7 6
Quick Sort The index left starts at the first element
and right starts at the next-to-last element
We want to move all the elements smaller than the pivot to the left part of the array and all the elements larger than the pivot to the right part
1 4 8 9 0 11 5 10 7 6left righ
t
Quick Sort We move left to the right skipping
over elements that are smaller than the pivot
1 4 8 9 0 11 5 10 7 6left righ
t
Quick Sort We then move right to the left skipping
over elements that are greater than the pivot
When left and right have stopped left is on an element greater than (or equal to) the pivot and right is on an element smaller than (or equal to) the pivot
1 4 8 9 0 11 5 10 7 6left right
Quick Sort If left is to the left of right (or if left
= right) those elements are swapped
1 4 8 9 0 11 5 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
Quick Sort The effect is to push a large
element to the right and a small element to the left
We then repeat the process until left and right cross
Quick Sort
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6left righ
t
Quick Sort
1 4 5 0 9 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6righ
tleft
Quick Sort
At this point left and right have crossed so no swap is performed
The final part of the partitioning is to swap the pivot element with left
1 4 5 0 9 11 8 10 7 6righ
tleft
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort Note that all elements to the left of
the pivot are less than (or equal to) the pivot and all elements to the right of the pivot are greater than (or equal to) the pivot
Hence the pivot element has been placed in its final sorted position
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort We now repeat the process using
the sub-arrays to the left and right of the pivot
1 4 5 0 6 11 8 10 7 9
1 4 5 0 11 8 10 7 9
Analysis of quicksortmdashbest case
Suppose each partition operation divides the array almost exactly in half
Then the depth of the recursion in log2n Because thatrsquos how many times we can
halve n However there are many recursions at
each level How can we figure this out We note that
Each partition is linear over its subarray All the partitions at one level cover the array
Partitioning at various levels
Best case II We cut the array size in half each time So the depth of the recursion in log2n At each level of the recursion all the
partitions at that level do work that is linear in n
O(log2n) O(n) = O(n log2n)
What about the worst case
Worst case In the worst case partitioning always
divides the size n array into these three parts A length one part containing the pivot itself A length zero part and A length n-1 part containing everything else
We donrsquot recur on the zero-length part Recurring on the length n-1 part requires
(in the worst case) recurring to depth n-1
Worst case partitioning
Worst case for quicksort In the worst case recursion may be
O(n) levels deep (for an array of size n)
But the partitioning work done at each level is still O(n)
O(n) O(n) = O(n2) So worst case for Quicksort is O(n2) When does this happen
When the array is sorted to begin with
Typical case for quicksort If the array is sorted to begin with
Quicksort is terrible O(n2) It is possible to construct other bad
cases However Quicksort is usually O(n log2n) The constants are so good that Quicksort
is generally the fastest algorithm known Most real-world sorting is done by
Quicksort
Tweaking Quicksort Almost anything you can try to
ldquoimproverdquo Quicksort will actually slow it down
One good tweak is to switch to a different sorting method when the subarrays get small (say 10 or 12) Quicksort has too much overhead for small
array sizes For large arrays it might be a good
idea to check beforehand if the array is already sorted But there is a better tweak than this
Randomized Quick-Sort Select the pivot as a random element of the sequence The expected running time of randomized quick-sort on a
sequence of size n is O(n log n) The time spent at a level of the quick-sort tree is O(n) We show that the expected height of the quick-sort tree is
O(log n) good vs bad pivots
bull The probability of a good pivot is 12 thus we expect k2 good pivots out of k pivotsbull After a good pivot the size of each child sequence is at most 34 the size of the parent sequencebull After h pivots we expect (34)h2 n elementsbull The expected height h of the quick-sort tree is at most 2 log43n
Median of three Obviously it doesnrsquot make sense to sort the
array in order to find the median to use as a pivot
Instead compare just three elements of our (sub)arraymdashthe first the last and the middle Take the median (middle value) of these three as
pivot Itrsquos possible (but not easy) to construct cases
which will make this technique O(n2) Suppose we rearrange (sort) these three
numbers so that the smallest is in the first position the largest in the last position and the other in the middle This lets us simplify and speed up the partition
loop
Final comments Weissrsquos code shows some additional
optimizations on pp 246-247 Weiss chooses to stop both searches on
equality to pivot This design decision is debatable
Quicksort is the fastest known general sorting algorithm on average
For optimum speed the pivot must be chosen carefully
ldquoMedian of threerdquo is a good technique for choosing the pivot
There will be some cases where Quicksort runs in O(n2) time
Quick Sort A couple of notes about quick sort There are more optimal ways to
choose the pivot value (such as the median-of-three method)
Also when the subarrays get small it becomes more efficient to use the insertion sort as opposed to continued use of quick sort
Bubble Sort vs Quick Sort If we calculate the Big-O notation
we find that (in the average case) Bubble Sort O(n2) Quick Sort O(nlog2n)
(quick-sort runs in time O(n2) in the worst case)
Heap Sort Idea take the items that need to be
sorted and insert them into the heap By calling deleteHeap we remove the
smallest or largest element depending on whether or not we are working with a min- or max-heap respectively
Hence the elements are removed in ascending or descending order
Efficiency O(nlog2n)
Merge Sort Idea
Take the array you would like to sort and divide it in half to create 2 unsorted subarrays
Next sort each of the 2 subarrays Finally merge the 2 sorted subarrays
into 1 sorted array Efficiency O(nlog2n)
Merge Sort
Merge Sort Although the merge step produces
a sorted array we have overlooked a very important step
How did we sort the 2 halves before performing the merge step
We used merge sort
Merge Sort By continually calling the merge
sort algorithm we eventually get a subarray of size 1
Since an array with only 1 element is clearly sorted we can back out and merge 2 arrays of size 1
Merge Sort
Merge Sort The basic merging algorithm
consists of 2 input arrays (arrayA and arrayB) An ouput array (arrayC) 3 position holders (indexA indexB
indexC) which are initially set to the beginning of their respective arrays
Merge Sort The smaller of arrayA[indexA] and
arrayB[indexB] is copied into arrayC[indexC] and the appropriate position holders are advanced
When either input list is exhausted the remainder of the other list is copied into arrayC
Merge Sort
1 13 24 26index
A
indexC
arrayA
2 15 27 38index
B
arrayB
arrayC
We compare arrayA[indexA]with arrayB[indexB] Whichever value is smaller isplaced into arrayC[indexC]
1 lt 2 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1index
C
arrayA
2 15 27 38index
B
arrayB
arrayC
2 lt 13 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
13 lt 15 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
15 lt 24 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13 15indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
24 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
indexA
1 2 13 15 24indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
26 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
1 2 13 15 24 26indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
Since we have exhaustedone of the arrays arrayAwe simply copy the remaining items from theother array arrayB intoarrayC
Merge Sort
1 13 24
26
1 2 13 15 24 26 27 38
arrayA
2 15 27 38arrayB
arrayC
Efficiency Summary
Sort Worst Case Average Case
Insertion O(n2) O(n2)
Shell O(n15) O(n125)
Selection O(n2) O(n2)
Bubble O(n2) O(n2)
Quick O(n2) O(nlog2n)
Heap O(nlog2n) O(nlog2n)
Merge O(nlog2n) O(nlog2n)
More Sorting
radix sort bucket sort in-place sorting how fast can we sort
Radix Sort
Unlike other sorting methods radix sort considers the structure of the keys
Assume keys are represented in a base M number system (M = radix) ie if M = 2 the keys are represented in binary
Sorting is done by comparing bits in the same position
Extension to keys that are alphanumeric strings
Radix Exchange Sort
Examine bits from left to right1 Sort array with respect to leftmost bit
2 Partition array
3Recursionrecursively sort top subarray ignoring leftmost bitrecursively sort bottom subarray ignoring leftmost bit
Time to sort n b-bit numbers O(b n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Insertion Sort vs Shell Sort(Average Case)
Straight Insertion Shell25 625 55100 10000 316500 250000 2364
1000 1000000 56232000 4000000 13374
Number of Loopsn
Selection Sort Imagine some data that you can examine
all at once To sort it you could select the smallest
element and put it in its place select the next smallest and put it in its place etc
For a card player this process is analogous to looking at an entire hand of cards and ordering them by selecting cards one at a time and placing them in their proper order
Selection Sort The selection sort follows this idea Given a list of data to be sorted
we simply select the smallest item and place it in a sorted list
We then repeat these steps until the list is sorted
Selection Sort In the selection sort the list at any moment
is divided into 2 sublists sorted and unsorted separated by a ldquoconceptualrdquo wall
We select the smallest element from the unsorted sublist and exchange it with the element at the beginning of the unsorted data
After each selection and exchange the wall between the 2 sublists moves ndash increasing the number of sorted elements and decreasing the number of unsorted elements
Selection Sort
We start with an unsorted list We search this list for thesmallest element We then exchange the smallest element (8)with the first element in the unsorted list (23)
Again we search the unsorted list for the smallest element We then exchange the smallest element (23) with the first element in the unsorted list (78)
This process continues until the list is fully sorted
Bubble Sort In the bubble sort the list at any
moment is divided into 2 sublists sorted and unsorted
The smallest element is ldquobubbledrdquo from the unsorted sublist to the sorted sublist
Bubble Sort
23 78 45 8 56 3232 56
We start with 32 and compareit with 56 Because 32 is lessthan 56 we swap the twoand step down one element
We then compare 32 and 8Because 32 is not less than 8we do not swap these elements
We step down one element andcompare 45 and 8 They areout of sequence so we swapthem and step down again
8 45
We step down again and compare 8 with 78 These two elements are swapped
8 78
Finally 8 is compared with 23and swapped We thencontinue this process back with56 hellip
8 23
1 Pass of the Bubble Sort
Quick Sort In the bubble sort consecutive items
are compared and possibly exchanged on each pass through the list
This means that many exchanges may be needed to move an element to its correct position
Quick sort is more efficient than bubble sort because a typical exchange involves elements that are far apart so fewer exchanges are required to correctly position an element
Idea of Quick Sort1) Select pick an element
2) Divide rearrange elements so that x goes to its final position E
3) Recurse and Conquer recursively sort
Quick Sort Each iteration of the quick sort
selects an element known as the pivot and divides the list into 3 groups Elements whose keys are less than (or
equal to) the pivotrsquos key The pivot element Elements whose keys are greater than
(or equal to) the pivotrsquos key
Quick Sort The sorting then continues by
quick sorting the left partition followed by quick sorting the right partition
The basic algorithm is as follows
Quick Sort1) Partitioning Step Take an element in the
unsorted array and determine its final location in the sorted array This occurs when all values to the left of the element in the array are less than (or equal to) the element and all values to the right of the element are greater than (or equal to) the element We now have 1 element in its proper location and two unsorted subarrays
2) Recursive Step Perform step 1 on each unsorted subarray
Quick Sort Each time step 1 is performed on a
subarray another element is placed in its final location of the sorted array and two unsorted subarrays are created
When a subarray consists of one element that subarray is sorted
Therefore that element is in its final location
Quick Sort There are several partitioning strategies
used in practice (ie several ldquoversionsrdquo of quick sort) but the one we are about to describe is known to work well
For simplicity we will choose the last element to be the pivot element
We could also chose a different pivot element and swap it with the last element in the array
Quick Sort Below is the array we would like to
sort1 4 8 9 0 11 5 10 7 6
Quick Sort The index left starts at the first element
and right starts at the next-to-last element
We want to move all the elements smaller than the pivot to the left part of the array and all the elements larger than the pivot to the right part
1 4 8 9 0 11 5 10 7 6left righ
t
Quick Sort We move left to the right skipping
over elements that are smaller than the pivot
1 4 8 9 0 11 5 10 7 6left righ
t
Quick Sort We then move right to the left skipping
over elements that are greater than the pivot
When left and right have stopped left is on an element greater than (or equal to) the pivot and right is on an element smaller than (or equal to) the pivot
1 4 8 9 0 11 5 10 7 6left right
Quick Sort If left is to the left of right (or if left
= right) those elements are swapped
1 4 8 9 0 11 5 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
Quick Sort The effect is to push a large
element to the right and a small element to the left
We then repeat the process until left and right cross
Quick Sort
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6left righ
t
Quick Sort
1 4 5 0 9 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6righ
tleft
Quick Sort
At this point left and right have crossed so no swap is performed
The final part of the partitioning is to swap the pivot element with left
1 4 5 0 9 11 8 10 7 6righ
tleft
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort Note that all elements to the left of
the pivot are less than (or equal to) the pivot and all elements to the right of the pivot are greater than (or equal to) the pivot
Hence the pivot element has been placed in its final sorted position
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort We now repeat the process using
the sub-arrays to the left and right of the pivot
1 4 5 0 6 11 8 10 7 9
1 4 5 0 11 8 10 7 9
Analysis of quicksortmdashbest case
Suppose each partition operation divides the array almost exactly in half
Then the depth of the recursion in log2n Because thatrsquos how many times we can
halve n However there are many recursions at
each level How can we figure this out We note that
Each partition is linear over its subarray All the partitions at one level cover the array
Partitioning at various levels
Best case II We cut the array size in half each time So the depth of the recursion in log2n At each level of the recursion all the
partitions at that level do work that is linear in n
O(log2n) O(n) = O(n log2n)
What about the worst case
Worst case In the worst case partitioning always
divides the size n array into these three parts A length one part containing the pivot itself A length zero part and A length n-1 part containing everything else
We donrsquot recur on the zero-length part Recurring on the length n-1 part requires
(in the worst case) recurring to depth n-1
Worst case partitioning
Worst case for quicksort In the worst case recursion may be
O(n) levels deep (for an array of size n)
But the partitioning work done at each level is still O(n)
O(n) O(n) = O(n2) So worst case for Quicksort is O(n2) When does this happen
When the array is sorted to begin with
Typical case for quicksort If the array is sorted to begin with
Quicksort is terrible O(n2) It is possible to construct other bad
cases However Quicksort is usually O(n log2n) The constants are so good that Quicksort
is generally the fastest algorithm known Most real-world sorting is done by
Quicksort
Tweaking Quicksort Almost anything you can try to
ldquoimproverdquo Quicksort will actually slow it down
One good tweak is to switch to a different sorting method when the subarrays get small (say 10 or 12) Quicksort has too much overhead for small
array sizes For large arrays it might be a good
idea to check beforehand if the array is already sorted But there is a better tweak than this
Randomized Quick-Sort Select the pivot as a random element of the sequence The expected running time of randomized quick-sort on a
sequence of size n is O(n log n) The time spent at a level of the quick-sort tree is O(n) We show that the expected height of the quick-sort tree is
O(log n) good vs bad pivots
bull The probability of a good pivot is 12 thus we expect k2 good pivots out of k pivotsbull After a good pivot the size of each child sequence is at most 34 the size of the parent sequencebull After h pivots we expect (34)h2 n elementsbull The expected height h of the quick-sort tree is at most 2 log43n
Median of three Obviously it doesnrsquot make sense to sort the
array in order to find the median to use as a pivot
Instead compare just three elements of our (sub)arraymdashthe first the last and the middle Take the median (middle value) of these three as
pivot Itrsquos possible (but not easy) to construct cases
which will make this technique O(n2) Suppose we rearrange (sort) these three
numbers so that the smallest is in the first position the largest in the last position and the other in the middle This lets us simplify and speed up the partition
loop
Final comments Weissrsquos code shows some additional
optimizations on pp 246-247 Weiss chooses to stop both searches on
equality to pivot This design decision is debatable
Quicksort is the fastest known general sorting algorithm on average
For optimum speed the pivot must be chosen carefully
ldquoMedian of threerdquo is a good technique for choosing the pivot
There will be some cases where Quicksort runs in O(n2) time
Quick Sort A couple of notes about quick sort There are more optimal ways to
choose the pivot value (such as the median-of-three method)
Also when the subarrays get small it becomes more efficient to use the insertion sort as opposed to continued use of quick sort
Bubble Sort vs Quick Sort If we calculate the Big-O notation
we find that (in the average case) Bubble Sort O(n2) Quick Sort O(nlog2n)
(quick-sort runs in time O(n2) in the worst case)
Heap Sort Idea take the items that need to be
sorted and insert them into the heap By calling deleteHeap we remove the
smallest or largest element depending on whether or not we are working with a min- or max-heap respectively
Hence the elements are removed in ascending or descending order
Efficiency O(nlog2n)
Merge Sort Idea
Take the array you would like to sort and divide it in half to create 2 unsorted subarrays
Next sort each of the 2 subarrays Finally merge the 2 sorted subarrays
into 1 sorted array Efficiency O(nlog2n)
Merge Sort
Merge Sort Although the merge step produces
a sorted array we have overlooked a very important step
How did we sort the 2 halves before performing the merge step
We used merge sort
Merge Sort By continually calling the merge
sort algorithm we eventually get a subarray of size 1
Since an array with only 1 element is clearly sorted we can back out and merge 2 arrays of size 1
Merge Sort
Merge Sort The basic merging algorithm
consists of 2 input arrays (arrayA and arrayB) An ouput array (arrayC) 3 position holders (indexA indexB
indexC) which are initially set to the beginning of their respective arrays
Merge Sort The smaller of arrayA[indexA] and
arrayB[indexB] is copied into arrayC[indexC] and the appropriate position holders are advanced
When either input list is exhausted the remainder of the other list is copied into arrayC
Merge Sort
1 13 24 26index
A
indexC
arrayA
2 15 27 38index
B
arrayB
arrayC
We compare arrayA[indexA]with arrayB[indexB] Whichever value is smaller isplaced into arrayC[indexC]
1 lt 2 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1index
C
arrayA
2 15 27 38index
B
arrayB
arrayC
2 lt 13 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
13 lt 15 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
15 lt 24 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13 15indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
24 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
indexA
1 2 13 15 24indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
26 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
1 2 13 15 24 26indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
Since we have exhaustedone of the arrays arrayAwe simply copy the remaining items from theother array arrayB intoarrayC
Merge Sort
1 13 24
26
1 2 13 15 24 26 27 38
arrayA
2 15 27 38arrayB
arrayC
Efficiency Summary
Sort Worst Case Average Case
Insertion O(n2) O(n2)
Shell O(n15) O(n125)
Selection O(n2) O(n2)
Bubble O(n2) O(n2)
Quick O(n2) O(nlog2n)
Heap O(nlog2n) O(nlog2n)
Merge O(nlog2n) O(nlog2n)
More Sorting
radix sort bucket sort in-place sorting how fast can we sort
Radix Sort
Unlike other sorting methods radix sort considers the structure of the keys
Assume keys are represented in a base M number system (M = radix) ie if M = 2 the keys are represented in binary
Sorting is done by comparing bits in the same position
Extension to keys that are alphanumeric strings
Radix Exchange Sort
Examine bits from left to right1 Sort array with respect to leftmost bit
2 Partition array
3Recursionrecursively sort top subarray ignoring leftmost bitrecursively sort bottom subarray ignoring leftmost bit
Time to sort n b-bit numbers O(b n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Selection Sort Imagine some data that you can examine
all at once To sort it you could select the smallest
element and put it in its place select the next smallest and put it in its place etc
For a card player this process is analogous to looking at an entire hand of cards and ordering them by selecting cards one at a time and placing them in their proper order
Selection Sort The selection sort follows this idea Given a list of data to be sorted
we simply select the smallest item and place it in a sorted list
We then repeat these steps until the list is sorted
Selection Sort In the selection sort the list at any moment
is divided into 2 sublists sorted and unsorted separated by a ldquoconceptualrdquo wall
We select the smallest element from the unsorted sublist and exchange it with the element at the beginning of the unsorted data
After each selection and exchange the wall between the 2 sublists moves ndash increasing the number of sorted elements and decreasing the number of unsorted elements
Selection Sort
We start with an unsorted list We search this list for thesmallest element We then exchange the smallest element (8)with the first element in the unsorted list (23)
Again we search the unsorted list for the smallest element We then exchange the smallest element (23) with the first element in the unsorted list (78)
This process continues until the list is fully sorted
Bubble Sort In the bubble sort the list at any
moment is divided into 2 sublists sorted and unsorted
The smallest element is ldquobubbledrdquo from the unsorted sublist to the sorted sublist
Bubble Sort
23 78 45 8 56 3232 56
We start with 32 and compareit with 56 Because 32 is lessthan 56 we swap the twoand step down one element
We then compare 32 and 8Because 32 is not less than 8we do not swap these elements
We step down one element andcompare 45 and 8 They areout of sequence so we swapthem and step down again
8 45
We step down again and compare 8 with 78 These two elements are swapped
8 78
Finally 8 is compared with 23and swapped We thencontinue this process back with56 hellip
8 23
1 Pass of the Bubble Sort
Quick Sort In the bubble sort consecutive items
are compared and possibly exchanged on each pass through the list
This means that many exchanges may be needed to move an element to its correct position
Quick sort is more efficient than bubble sort because a typical exchange involves elements that are far apart so fewer exchanges are required to correctly position an element
Idea of Quick Sort1) Select pick an element
2) Divide rearrange elements so that x goes to its final position E
3) Recurse and Conquer recursively sort
Quick Sort Each iteration of the quick sort
selects an element known as the pivot and divides the list into 3 groups Elements whose keys are less than (or
equal to) the pivotrsquos key The pivot element Elements whose keys are greater than
(or equal to) the pivotrsquos key
Quick Sort The sorting then continues by
quick sorting the left partition followed by quick sorting the right partition
The basic algorithm is as follows
Quick Sort1) Partitioning Step Take an element in the
unsorted array and determine its final location in the sorted array This occurs when all values to the left of the element in the array are less than (or equal to) the element and all values to the right of the element are greater than (or equal to) the element We now have 1 element in its proper location and two unsorted subarrays
2) Recursive Step Perform step 1 on each unsorted subarray
Quick Sort Each time step 1 is performed on a
subarray another element is placed in its final location of the sorted array and two unsorted subarrays are created
When a subarray consists of one element that subarray is sorted
Therefore that element is in its final location
Quick Sort There are several partitioning strategies
used in practice (ie several ldquoversionsrdquo of quick sort) but the one we are about to describe is known to work well
For simplicity we will choose the last element to be the pivot element
We could also chose a different pivot element and swap it with the last element in the array
Quick Sort Below is the array we would like to
sort1 4 8 9 0 11 5 10 7 6
Quick Sort The index left starts at the first element
and right starts at the next-to-last element
We want to move all the elements smaller than the pivot to the left part of the array and all the elements larger than the pivot to the right part
1 4 8 9 0 11 5 10 7 6left righ
t
Quick Sort We move left to the right skipping
over elements that are smaller than the pivot
1 4 8 9 0 11 5 10 7 6left righ
t
Quick Sort We then move right to the left skipping
over elements that are greater than the pivot
When left and right have stopped left is on an element greater than (or equal to) the pivot and right is on an element smaller than (or equal to) the pivot
1 4 8 9 0 11 5 10 7 6left right
Quick Sort If left is to the left of right (or if left
= right) those elements are swapped
1 4 8 9 0 11 5 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
Quick Sort The effect is to push a large
element to the right and a small element to the left
We then repeat the process until left and right cross
Quick Sort
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6left righ
t
Quick Sort
1 4 5 0 9 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6righ
tleft
Quick Sort
At this point left and right have crossed so no swap is performed
The final part of the partitioning is to swap the pivot element with left
1 4 5 0 9 11 8 10 7 6righ
tleft
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort Note that all elements to the left of
the pivot are less than (or equal to) the pivot and all elements to the right of the pivot are greater than (or equal to) the pivot
Hence the pivot element has been placed in its final sorted position
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort We now repeat the process using
the sub-arrays to the left and right of the pivot
1 4 5 0 6 11 8 10 7 9
1 4 5 0 11 8 10 7 9
Analysis of quicksortmdashbest case
Suppose each partition operation divides the array almost exactly in half
Then the depth of the recursion in log2n Because thatrsquos how many times we can
halve n However there are many recursions at
each level How can we figure this out We note that
Each partition is linear over its subarray All the partitions at one level cover the array
Partitioning at various levels
Best case II We cut the array size in half each time So the depth of the recursion in log2n At each level of the recursion all the
partitions at that level do work that is linear in n
O(log2n) O(n) = O(n log2n)
What about the worst case
Worst case In the worst case partitioning always
divides the size n array into these three parts A length one part containing the pivot itself A length zero part and A length n-1 part containing everything else
We donrsquot recur on the zero-length part Recurring on the length n-1 part requires
(in the worst case) recurring to depth n-1
Worst case partitioning
Worst case for quicksort In the worst case recursion may be
O(n) levels deep (for an array of size n)
But the partitioning work done at each level is still O(n)
O(n) O(n) = O(n2) So worst case for Quicksort is O(n2) When does this happen
When the array is sorted to begin with
Typical case for quicksort If the array is sorted to begin with
Quicksort is terrible O(n2) It is possible to construct other bad
cases However Quicksort is usually O(n log2n) The constants are so good that Quicksort
is generally the fastest algorithm known Most real-world sorting is done by
Quicksort
Tweaking Quicksort Almost anything you can try to
ldquoimproverdquo Quicksort will actually slow it down
One good tweak is to switch to a different sorting method when the subarrays get small (say 10 or 12) Quicksort has too much overhead for small
array sizes For large arrays it might be a good
idea to check beforehand if the array is already sorted But there is a better tweak than this
Randomized Quick-Sort Select the pivot as a random element of the sequence The expected running time of randomized quick-sort on a
sequence of size n is O(n log n) The time spent at a level of the quick-sort tree is O(n) We show that the expected height of the quick-sort tree is
O(log n) good vs bad pivots
bull The probability of a good pivot is 12 thus we expect k2 good pivots out of k pivotsbull After a good pivot the size of each child sequence is at most 34 the size of the parent sequencebull After h pivots we expect (34)h2 n elementsbull The expected height h of the quick-sort tree is at most 2 log43n
Median of three Obviously it doesnrsquot make sense to sort the
array in order to find the median to use as a pivot
Instead compare just three elements of our (sub)arraymdashthe first the last and the middle Take the median (middle value) of these three as
pivot Itrsquos possible (but not easy) to construct cases
which will make this technique O(n2) Suppose we rearrange (sort) these three
numbers so that the smallest is in the first position the largest in the last position and the other in the middle This lets us simplify and speed up the partition
loop
Final comments Weissrsquos code shows some additional
optimizations on pp 246-247 Weiss chooses to stop both searches on
equality to pivot This design decision is debatable
Quicksort is the fastest known general sorting algorithm on average
For optimum speed the pivot must be chosen carefully
ldquoMedian of threerdquo is a good technique for choosing the pivot
There will be some cases where Quicksort runs in O(n2) time
Quick Sort A couple of notes about quick sort There are more optimal ways to
choose the pivot value (such as the median-of-three method)
Also when the subarrays get small it becomes more efficient to use the insertion sort as opposed to continued use of quick sort
Bubble Sort vs Quick Sort If we calculate the Big-O notation
we find that (in the average case) Bubble Sort O(n2) Quick Sort O(nlog2n)
(quick-sort runs in time O(n2) in the worst case)
Heap Sort Idea take the items that need to be
sorted and insert them into the heap By calling deleteHeap we remove the
smallest or largest element depending on whether or not we are working with a min- or max-heap respectively
Hence the elements are removed in ascending or descending order
Efficiency O(nlog2n)
Merge Sort Idea
Take the array you would like to sort and divide it in half to create 2 unsorted subarrays
Next sort each of the 2 subarrays Finally merge the 2 sorted subarrays
into 1 sorted array Efficiency O(nlog2n)
Merge Sort
Merge Sort Although the merge step produces
a sorted array we have overlooked a very important step
How did we sort the 2 halves before performing the merge step
We used merge sort
Merge Sort By continually calling the merge
sort algorithm we eventually get a subarray of size 1
Since an array with only 1 element is clearly sorted we can back out and merge 2 arrays of size 1
Merge Sort
Merge Sort The basic merging algorithm
consists of 2 input arrays (arrayA and arrayB) An ouput array (arrayC) 3 position holders (indexA indexB
indexC) which are initially set to the beginning of their respective arrays
Merge Sort The smaller of arrayA[indexA] and
arrayB[indexB] is copied into arrayC[indexC] and the appropriate position holders are advanced
When either input list is exhausted the remainder of the other list is copied into arrayC
Merge Sort
1 13 24 26index
A
indexC
arrayA
2 15 27 38index
B
arrayB
arrayC
We compare arrayA[indexA]with arrayB[indexB] Whichever value is smaller isplaced into arrayC[indexC]
1 lt 2 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1index
C
arrayA
2 15 27 38index
B
arrayB
arrayC
2 lt 13 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
13 lt 15 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
15 lt 24 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13 15indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
24 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
indexA
1 2 13 15 24indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
26 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
1 2 13 15 24 26indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
Since we have exhaustedone of the arrays arrayAwe simply copy the remaining items from theother array arrayB intoarrayC
Merge Sort
1 13 24
26
1 2 13 15 24 26 27 38
arrayA
2 15 27 38arrayB
arrayC
Efficiency Summary
Sort Worst Case Average Case
Insertion O(n2) O(n2)
Shell O(n15) O(n125)
Selection O(n2) O(n2)
Bubble O(n2) O(n2)
Quick O(n2) O(nlog2n)
Heap O(nlog2n) O(nlog2n)
Merge O(nlog2n) O(nlog2n)
More Sorting
radix sort bucket sort in-place sorting how fast can we sort
Radix Sort
Unlike other sorting methods radix sort considers the structure of the keys
Assume keys are represented in a base M number system (M = radix) ie if M = 2 the keys are represented in binary
Sorting is done by comparing bits in the same position
Extension to keys that are alphanumeric strings
Radix Exchange Sort
Examine bits from left to right1 Sort array with respect to leftmost bit
2 Partition array
3Recursionrecursively sort top subarray ignoring leftmost bitrecursively sort bottom subarray ignoring leftmost bit
Time to sort n b-bit numbers O(b n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Selection Sort The selection sort follows this idea Given a list of data to be sorted
we simply select the smallest item and place it in a sorted list
We then repeat these steps until the list is sorted
Selection Sort In the selection sort the list at any moment
is divided into 2 sublists sorted and unsorted separated by a ldquoconceptualrdquo wall
We select the smallest element from the unsorted sublist and exchange it with the element at the beginning of the unsorted data
After each selection and exchange the wall between the 2 sublists moves ndash increasing the number of sorted elements and decreasing the number of unsorted elements
Selection Sort
We start with an unsorted list We search this list for thesmallest element We then exchange the smallest element (8)with the first element in the unsorted list (23)
Again we search the unsorted list for the smallest element We then exchange the smallest element (23) with the first element in the unsorted list (78)
This process continues until the list is fully sorted
Bubble Sort In the bubble sort the list at any
moment is divided into 2 sublists sorted and unsorted
The smallest element is ldquobubbledrdquo from the unsorted sublist to the sorted sublist
Bubble Sort
23 78 45 8 56 3232 56
We start with 32 and compareit with 56 Because 32 is lessthan 56 we swap the twoand step down one element
We then compare 32 and 8Because 32 is not less than 8we do not swap these elements
We step down one element andcompare 45 and 8 They areout of sequence so we swapthem and step down again
8 45
We step down again and compare 8 with 78 These two elements are swapped
8 78
Finally 8 is compared with 23and swapped We thencontinue this process back with56 hellip
8 23
1 Pass of the Bubble Sort
Quick Sort In the bubble sort consecutive items
are compared and possibly exchanged on each pass through the list
This means that many exchanges may be needed to move an element to its correct position
Quick sort is more efficient than bubble sort because a typical exchange involves elements that are far apart so fewer exchanges are required to correctly position an element
Idea of Quick Sort1) Select pick an element
2) Divide rearrange elements so that x goes to its final position E
3) Recurse and Conquer recursively sort
Quick Sort Each iteration of the quick sort
selects an element known as the pivot and divides the list into 3 groups Elements whose keys are less than (or
equal to) the pivotrsquos key The pivot element Elements whose keys are greater than
(or equal to) the pivotrsquos key
Quick Sort The sorting then continues by
quick sorting the left partition followed by quick sorting the right partition
The basic algorithm is as follows
Quick Sort1) Partitioning Step Take an element in the
unsorted array and determine its final location in the sorted array This occurs when all values to the left of the element in the array are less than (or equal to) the element and all values to the right of the element are greater than (or equal to) the element We now have 1 element in its proper location and two unsorted subarrays
2) Recursive Step Perform step 1 on each unsorted subarray
Quick Sort Each time step 1 is performed on a
subarray another element is placed in its final location of the sorted array and two unsorted subarrays are created
When a subarray consists of one element that subarray is sorted
Therefore that element is in its final location
Quick Sort There are several partitioning strategies
used in practice (ie several ldquoversionsrdquo of quick sort) but the one we are about to describe is known to work well
For simplicity we will choose the last element to be the pivot element
We could also chose a different pivot element and swap it with the last element in the array
Quick Sort Below is the array we would like to
sort1 4 8 9 0 11 5 10 7 6
Quick Sort The index left starts at the first element
and right starts at the next-to-last element
We want to move all the elements smaller than the pivot to the left part of the array and all the elements larger than the pivot to the right part
1 4 8 9 0 11 5 10 7 6left righ
t
Quick Sort We move left to the right skipping
over elements that are smaller than the pivot
1 4 8 9 0 11 5 10 7 6left righ
t
Quick Sort We then move right to the left skipping
over elements that are greater than the pivot
When left and right have stopped left is on an element greater than (or equal to) the pivot and right is on an element smaller than (or equal to) the pivot
1 4 8 9 0 11 5 10 7 6left right
Quick Sort If left is to the left of right (or if left
= right) those elements are swapped
1 4 8 9 0 11 5 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
Quick Sort The effect is to push a large
element to the right and a small element to the left
We then repeat the process until left and right cross
Quick Sort
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6left righ
t
Quick Sort
1 4 5 0 9 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6righ
tleft
Quick Sort
At this point left and right have crossed so no swap is performed
The final part of the partitioning is to swap the pivot element with left
1 4 5 0 9 11 8 10 7 6righ
tleft
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort Note that all elements to the left of
the pivot are less than (or equal to) the pivot and all elements to the right of the pivot are greater than (or equal to) the pivot
Hence the pivot element has been placed in its final sorted position
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort We now repeat the process using
the sub-arrays to the left and right of the pivot
1 4 5 0 6 11 8 10 7 9
1 4 5 0 11 8 10 7 9
Analysis of quicksortmdashbest case
Suppose each partition operation divides the array almost exactly in half
Then the depth of the recursion in log2n Because thatrsquos how many times we can
halve n However there are many recursions at
each level How can we figure this out We note that
Each partition is linear over its subarray All the partitions at one level cover the array
Partitioning at various levels
Best case II We cut the array size in half each time So the depth of the recursion in log2n At each level of the recursion all the
partitions at that level do work that is linear in n
O(log2n) O(n) = O(n log2n)
What about the worst case
Worst case In the worst case partitioning always
divides the size n array into these three parts A length one part containing the pivot itself A length zero part and A length n-1 part containing everything else
We donrsquot recur on the zero-length part Recurring on the length n-1 part requires
(in the worst case) recurring to depth n-1
Worst case partitioning
Worst case for quicksort In the worst case recursion may be
O(n) levels deep (for an array of size n)
But the partitioning work done at each level is still O(n)
O(n) O(n) = O(n2) So worst case for Quicksort is O(n2) When does this happen
When the array is sorted to begin with
Typical case for quicksort If the array is sorted to begin with
Quicksort is terrible O(n2) It is possible to construct other bad
cases However Quicksort is usually O(n log2n) The constants are so good that Quicksort
is generally the fastest algorithm known Most real-world sorting is done by
Quicksort
Tweaking Quicksort Almost anything you can try to
ldquoimproverdquo Quicksort will actually slow it down
One good tweak is to switch to a different sorting method when the subarrays get small (say 10 or 12) Quicksort has too much overhead for small
array sizes For large arrays it might be a good
idea to check beforehand if the array is already sorted But there is a better tweak than this
Randomized Quick-Sort Select the pivot as a random element of the sequence The expected running time of randomized quick-sort on a
sequence of size n is O(n log n) The time spent at a level of the quick-sort tree is O(n) We show that the expected height of the quick-sort tree is
O(log n) good vs bad pivots
bull The probability of a good pivot is 12 thus we expect k2 good pivots out of k pivotsbull After a good pivot the size of each child sequence is at most 34 the size of the parent sequencebull After h pivots we expect (34)h2 n elementsbull The expected height h of the quick-sort tree is at most 2 log43n
Median of three Obviously it doesnrsquot make sense to sort the
array in order to find the median to use as a pivot
Instead compare just three elements of our (sub)arraymdashthe first the last and the middle Take the median (middle value) of these three as
pivot Itrsquos possible (but not easy) to construct cases
which will make this technique O(n2) Suppose we rearrange (sort) these three
numbers so that the smallest is in the first position the largest in the last position and the other in the middle This lets us simplify and speed up the partition
loop
Final comments Weissrsquos code shows some additional
optimizations on pp 246-247 Weiss chooses to stop both searches on
equality to pivot This design decision is debatable
Quicksort is the fastest known general sorting algorithm on average
For optimum speed the pivot must be chosen carefully
ldquoMedian of threerdquo is a good technique for choosing the pivot
There will be some cases where Quicksort runs in O(n2) time
Quick Sort A couple of notes about quick sort There are more optimal ways to
choose the pivot value (such as the median-of-three method)
Also when the subarrays get small it becomes more efficient to use the insertion sort as opposed to continued use of quick sort
Bubble Sort vs Quick Sort If we calculate the Big-O notation
we find that (in the average case) Bubble Sort O(n2) Quick Sort O(nlog2n)
(quick-sort runs in time O(n2) in the worst case)
Heap Sort Idea take the items that need to be
sorted and insert them into the heap By calling deleteHeap we remove the
smallest or largest element depending on whether or not we are working with a min- or max-heap respectively
Hence the elements are removed in ascending or descending order
Efficiency O(nlog2n)
Merge Sort Idea
Take the array you would like to sort and divide it in half to create 2 unsorted subarrays
Next sort each of the 2 subarrays Finally merge the 2 sorted subarrays
into 1 sorted array Efficiency O(nlog2n)
Merge Sort
Merge Sort Although the merge step produces
a sorted array we have overlooked a very important step
How did we sort the 2 halves before performing the merge step
We used merge sort
Merge Sort By continually calling the merge
sort algorithm we eventually get a subarray of size 1
Since an array with only 1 element is clearly sorted we can back out and merge 2 arrays of size 1
Merge Sort
Merge Sort The basic merging algorithm
consists of 2 input arrays (arrayA and arrayB) An ouput array (arrayC) 3 position holders (indexA indexB
indexC) which are initially set to the beginning of their respective arrays
Merge Sort The smaller of arrayA[indexA] and
arrayB[indexB] is copied into arrayC[indexC] and the appropriate position holders are advanced
When either input list is exhausted the remainder of the other list is copied into arrayC
Merge Sort
1 13 24 26index
A
indexC
arrayA
2 15 27 38index
B
arrayB
arrayC
We compare arrayA[indexA]with arrayB[indexB] Whichever value is smaller isplaced into arrayC[indexC]
1 lt 2 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1index
C
arrayA
2 15 27 38index
B
arrayB
arrayC
2 lt 13 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
13 lt 15 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
15 lt 24 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13 15indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
24 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
indexA
1 2 13 15 24indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
26 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
1 2 13 15 24 26indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
Since we have exhaustedone of the arrays arrayAwe simply copy the remaining items from theother array arrayB intoarrayC
Merge Sort
1 13 24
26
1 2 13 15 24 26 27 38
arrayA
2 15 27 38arrayB
arrayC
Efficiency Summary
Sort Worst Case Average Case
Insertion O(n2) O(n2)
Shell O(n15) O(n125)
Selection O(n2) O(n2)
Bubble O(n2) O(n2)
Quick O(n2) O(nlog2n)
Heap O(nlog2n) O(nlog2n)
Merge O(nlog2n) O(nlog2n)
More Sorting
radix sort bucket sort in-place sorting how fast can we sort
Radix Sort
Unlike other sorting methods radix sort considers the structure of the keys
Assume keys are represented in a base M number system (M = radix) ie if M = 2 the keys are represented in binary
Sorting is done by comparing bits in the same position
Extension to keys that are alphanumeric strings
Radix Exchange Sort
Examine bits from left to right1 Sort array with respect to leftmost bit
2 Partition array
3Recursionrecursively sort top subarray ignoring leftmost bitrecursively sort bottom subarray ignoring leftmost bit
Time to sort n b-bit numbers O(b n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Selection Sort In the selection sort the list at any moment
is divided into 2 sublists sorted and unsorted separated by a ldquoconceptualrdquo wall
We select the smallest element from the unsorted sublist and exchange it with the element at the beginning of the unsorted data
After each selection and exchange the wall between the 2 sublists moves ndash increasing the number of sorted elements and decreasing the number of unsorted elements
Selection Sort
We start with an unsorted list We search this list for thesmallest element We then exchange the smallest element (8)with the first element in the unsorted list (23)
Again we search the unsorted list for the smallest element We then exchange the smallest element (23) with the first element in the unsorted list (78)
This process continues until the list is fully sorted
Bubble Sort In the bubble sort the list at any
moment is divided into 2 sublists sorted and unsorted
The smallest element is ldquobubbledrdquo from the unsorted sublist to the sorted sublist
Bubble Sort
23 78 45 8 56 3232 56
We start with 32 and compareit with 56 Because 32 is lessthan 56 we swap the twoand step down one element
We then compare 32 and 8Because 32 is not less than 8we do not swap these elements
We step down one element andcompare 45 and 8 They areout of sequence so we swapthem and step down again
8 45
We step down again and compare 8 with 78 These two elements are swapped
8 78
Finally 8 is compared with 23and swapped We thencontinue this process back with56 hellip
8 23
1 Pass of the Bubble Sort
Quick Sort In the bubble sort consecutive items
are compared and possibly exchanged on each pass through the list
This means that many exchanges may be needed to move an element to its correct position
Quick sort is more efficient than bubble sort because a typical exchange involves elements that are far apart so fewer exchanges are required to correctly position an element
Idea of Quick Sort1) Select pick an element
2) Divide rearrange elements so that x goes to its final position E
3) Recurse and Conquer recursively sort
Quick Sort Each iteration of the quick sort
selects an element known as the pivot and divides the list into 3 groups Elements whose keys are less than (or
equal to) the pivotrsquos key The pivot element Elements whose keys are greater than
(or equal to) the pivotrsquos key
Quick Sort The sorting then continues by
quick sorting the left partition followed by quick sorting the right partition
The basic algorithm is as follows
Quick Sort1) Partitioning Step Take an element in the
unsorted array and determine its final location in the sorted array This occurs when all values to the left of the element in the array are less than (or equal to) the element and all values to the right of the element are greater than (or equal to) the element We now have 1 element in its proper location and two unsorted subarrays
2) Recursive Step Perform step 1 on each unsorted subarray
Quick Sort Each time step 1 is performed on a
subarray another element is placed in its final location of the sorted array and two unsorted subarrays are created
When a subarray consists of one element that subarray is sorted
Therefore that element is in its final location
Quick Sort There are several partitioning strategies
used in practice (ie several ldquoversionsrdquo of quick sort) but the one we are about to describe is known to work well
For simplicity we will choose the last element to be the pivot element
We could also chose a different pivot element and swap it with the last element in the array
Quick Sort Below is the array we would like to
sort1 4 8 9 0 11 5 10 7 6
Quick Sort The index left starts at the first element
and right starts at the next-to-last element
We want to move all the elements smaller than the pivot to the left part of the array and all the elements larger than the pivot to the right part
1 4 8 9 0 11 5 10 7 6left righ
t
Quick Sort We move left to the right skipping
over elements that are smaller than the pivot
1 4 8 9 0 11 5 10 7 6left righ
t
Quick Sort We then move right to the left skipping
over elements that are greater than the pivot
When left and right have stopped left is on an element greater than (or equal to) the pivot and right is on an element smaller than (or equal to) the pivot
1 4 8 9 0 11 5 10 7 6left right
Quick Sort If left is to the left of right (or if left
= right) those elements are swapped
1 4 8 9 0 11 5 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
Quick Sort The effect is to push a large
element to the right and a small element to the left
We then repeat the process until left and right cross
Quick Sort
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6left righ
t
Quick Sort
1 4 5 0 9 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6righ
tleft
Quick Sort
At this point left and right have crossed so no swap is performed
The final part of the partitioning is to swap the pivot element with left
1 4 5 0 9 11 8 10 7 6righ
tleft
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort Note that all elements to the left of
the pivot are less than (or equal to) the pivot and all elements to the right of the pivot are greater than (or equal to) the pivot
Hence the pivot element has been placed in its final sorted position
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort We now repeat the process using
the sub-arrays to the left and right of the pivot
1 4 5 0 6 11 8 10 7 9
1 4 5 0 11 8 10 7 9
Analysis of quicksortmdashbest case
Suppose each partition operation divides the array almost exactly in half
Then the depth of the recursion in log2n Because thatrsquos how many times we can
halve n However there are many recursions at
each level How can we figure this out We note that
Each partition is linear over its subarray All the partitions at one level cover the array
Partitioning at various levels
Best case II We cut the array size in half each time So the depth of the recursion in log2n At each level of the recursion all the
partitions at that level do work that is linear in n
O(log2n) O(n) = O(n log2n)
What about the worst case
Worst case In the worst case partitioning always
divides the size n array into these three parts A length one part containing the pivot itself A length zero part and A length n-1 part containing everything else
We donrsquot recur on the zero-length part Recurring on the length n-1 part requires
(in the worst case) recurring to depth n-1
Worst case partitioning
Worst case for quicksort In the worst case recursion may be
O(n) levels deep (for an array of size n)
But the partitioning work done at each level is still O(n)
O(n) O(n) = O(n2) So worst case for Quicksort is O(n2) When does this happen
When the array is sorted to begin with
Typical case for quicksort If the array is sorted to begin with
Quicksort is terrible O(n2) It is possible to construct other bad
cases However Quicksort is usually O(n log2n) The constants are so good that Quicksort
is generally the fastest algorithm known Most real-world sorting is done by
Quicksort
Tweaking Quicksort Almost anything you can try to
ldquoimproverdquo Quicksort will actually slow it down
One good tweak is to switch to a different sorting method when the subarrays get small (say 10 or 12) Quicksort has too much overhead for small
array sizes For large arrays it might be a good
idea to check beforehand if the array is already sorted But there is a better tweak than this
Randomized Quick-Sort Select the pivot as a random element of the sequence The expected running time of randomized quick-sort on a
sequence of size n is O(n log n) The time spent at a level of the quick-sort tree is O(n) We show that the expected height of the quick-sort tree is
O(log n) good vs bad pivots
bull The probability of a good pivot is 12 thus we expect k2 good pivots out of k pivotsbull After a good pivot the size of each child sequence is at most 34 the size of the parent sequencebull After h pivots we expect (34)h2 n elementsbull The expected height h of the quick-sort tree is at most 2 log43n
Median of three Obviously it doesnrsquot make sense to sort the
array in order to find the median to use as a pivot
Instead compare just three elements of our (sub)arraymdashthe first the last and the middle Take the median (middle value) of these three as
pivot Itrsquos possible (but not easy) to construct cases
which will make this technique O(n2) Suppose we rearrange (sort) these three
numbers so that the smallest is in the first position the largest in the last position and the other in the middle This lets us simplify and speed up the partition
loop
Final comments Weissrsquos code shows some additional
optimizations on pp 246-247 Weiss chooses to stop both searches on
equality to pivot This design decision is debatable
Quicksort is the fastest known general sorting algorithm on average
For optimum speed the pivot must be chosen carefully
ldquoMedian of threerdquo is a good technique for choosing the pivot
There will be some cases where Quicksort runs in O(n2) time
Quick Sort A couple of notes about quick sort There are more optimal ways to
choose the pivot value (such as the median-of-three method)
Also when the subarrays get small it becomes more efficient to use the insertion sort as opposed to continued use of quick sort
Bubble Sort vs Quick Sort If we calculate the Big-O notation
we find that (in the average case) Bubble Sort O(n2) Quick Sort O(nlog2n)
(quick-sort runs in time O(n2) in the worst case)
Heap Sort Idea take the items that need to be
sorted and insert them into the heap By calling deleteHeap we remove the
smallest or largest element depending on whether or not we are working with a min- or max-heap respectively
Hence the elements are removed in ascending or descending order
Efficiency O(nlog2n)
Merge Sort Idea
Take the array you would like to sort and divide it in half to create 2 unsorted subarrays
Next sort each of the 2 subarrays Finally merge the 2 sorted subarrays
into 1 sorted array Efficiency O(nlog2n)
Merge Sort
Merge Sort Although the merge step produces
a sorted array we have overlooked a very important step
How did we sort the 2 halves before performing the merge step
We used merge sort
Merge Sort By continually calling the merge
sort algorithm we eventually get a subarray of size 1
Since an array with only 1 element is clearly sorted we can back out and merge 2 arrays of size 1
Merge Sort
Merge Sort The basic merging algorithm
consists of 2 input arrays (arrayA and arrayB) An ouput array (arrayC) 3 position holders (indexA indexB
indexC) which are initially set to the beginning of their respective arrays
Merge Sort The smaller of arrayA[indexA] and
arrayB[indexB] is copied into arrayC[indexC] and the appropriate position holders are advanced
When either input list is exhausted the remainder of the other list is copied into arrayC
Merge Sort
1 13 24 26index
A
indexC
arrayA
2 15 27 38index
B
arrayB
arrayC
We compare arrayA[indexA]with arrayB[indexB] Whichever value is smaller isplaced into arrayC[indexC]
1 lt 2 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1index
C
arrayA
2 15 27 38index
B
arrayB
arrayC
2 lt 13 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
13 lt 15 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
15 lt 24 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13 15indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
24 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
indexA
1 2 13 15 24indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
26 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
1 2 13 15 24 26indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
Since we have exhaustedone of the arrays arrayAwe simply copy the remaining items from theother array arrayB intoarrayC
Merge Sort
1 13 24
26
1 2 13 15 24 26 27 38
arrayA
2 15 27 38arrayB
arrayC
Efficiency Summary
Sort Worst Case Average Case
Insertion O(n2) O(n2)
Shell O(n15) O(n125)
Selection O(n2) O(n2)
Bubble O(n2) O(n2)
Quick O(n2) O(nlog2n)
Heap O(nlog2n) O(nlog2n)
Merge O(nlog2n) O(nlog2n)
More Sorting
radix sort bucket sort in-place sorting how fast can we sort
Radix Sort
Unlike other sorting methods radix sort considers the structure of the keys
Assume keys are represented in a base M number system (M = radix) ie if M = 2 the keys are represented in binary
Sorting is done by comparing bits in the same position
Extension to keys that are alphanumeric strings
Radix Exchange Sort
Examine bits from left to right1 Sort array with respect to leftmost bit
2 Partition array
3Recursionrecursively sort top subarray ignoring leftmost bitrecursively sort bottom subarray ignoring leftmost bit
Time to sort n b-bit numbers O(b n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Selection Sort
We start with an unsorted list We search this list for thesmallest element We then exchange the smallest element (8)with the first element in the unsorted list (23)
Again we search the unsorted list for the smallest element We then exchange the smallest element (23) with the first element in the unsorted list (78)
This process continues until the list is fully sorted
Bubble Sort In the bubble sort the list at any
moment is divided into 2 sublists sorted and unsorted
The smallest element is ldquobubbledrdquo from the unsorted sublist to the sorted sublist
Bubble Sort
23 78 45 8 56 3232 56
We start with 32 and compareit with 56 Because 32 is lessthan 56 we swap the twoand step down one element
We then compare 32 and 8Because 32 is not less than 8we do not swap these elements
We step down one element andcompare 45 and 8 They areout of sequence so we swapthem and step down again
8 45
We step down again and compare 8 with 78 These two elements are swapped
8 78
Finally 8 is compared with 23and swapped We thencontinue this process back with56 hellip
8 23
1 Pass of the Bubble Sort
Quick Sort In the bubble sort consecutive items
are compared and possibly exchanged on each pass through the list
This means that many exchanges may be needed to move an element to its correct position
Quick sort is more efficient than bubble sort because a typical exchange involves elements that are far apart so fewer exchanges are required to correctly position an element
Idea of Quick Sort1) Select pick an element
2) Divide rearrange elements so that x goes to its final position E
3) Recurse and Conquer recursively sort
Quick Sort Each iteration of the quick sort
selects an element known as the pivot and divides the list into 3 groups Elements whose keys are less than (or
equal to) the pivotrsquos key The pivot element Elements whose keys are greater than
(or equal to) the pivotrsquos key
Quick Sort The sorting then continues by
quick sorting the left partition followed by quick sorting the right partition
The basic algorithm is as follows
Quick Sort1) Partitioning Step Take an element in the
unsorted array and determine its final location in the sorted array This occurs when all values to the left of the element in the array are less than (or equal to) the element and all values to the right of the element are greater than (or equal to) the element We now have 1 element in its proper location and two unsorted subarrays
2) Recursive Step Perform step 1 on each unsorted subarray
Quick Sort Each time step 1 is performed on a
subarray another element is placed in its final location of the sorted array and two unsorted subarrays are created
When a subarray consists of one element that subarray is sorted
Therefore that element is in its final location
Quick Sort There are several partitioning strategies
used in practice (ie several ldquoversionsrdquo of quick sort) but the one we are about to describe is known to work well
For simplicity we will choose the last element to be the pivot element
We could also chose a different pivot element and swap it with the last element in the array
Quick Sort Below is the array we would like to
sort1 4 8 9 0 11 5 10 7 6
Quick Sort The index left starts at the first element
and right starts at the next-to-last element
We want to move all the elements smaller than the pivot to the left part of the array and all the elements larger than the pivot to the right part
1 4 8 9 0 11 5 10 7 6left righ
t
Quick Sort We move left to the right skipping
over elements that are smaller than the pivot
1 4 8 9 0 11 5 10 7 6left righ
t
Quick Sort We then move right to the left skipping
over elements that are greater than the pivot
When left and right have stopped left is on an element greater than (or equal to) the pivot and right is on an element smaller than (or equal to) the pivot
1 4 8 9 0 11 5 10 7 6left right
Quick Sort If left is to the left of right (or if left
= right) those elements are swapped
1 4 8 9 0 11 5 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
Quick Sort The effect is to push a large
element to the right and a small element to the left
We then repeat the process until left and right cross
Quick Sort
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6left righ
t
Quick Sort
1 4 5 0 9 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6righ
tleft
Quick Sort
At this point left and right have crossed so no swap is performed
The final part of the partitioning is to swap the pivot element with left
1 4 5 0 9 11 8 10 7 6righ
tleft
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort Note that all elements to the left of
the pivot are less than (or equal to) the pivot and all elements to the right of the pivot are greater than (or equal to) the pivot
Hence the pivot element has been placed in its final sorted position
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort We now repeat the process using
the sub-arrays to the left and right of the pivot
1 4 5 0 6 11 8 10 7 9
1 4 5 0 11 8 10 7 9
Analysis of quicksortmdashbest case
Suppose each partition operation divides the array almost exactly in half
Then the depth of the recursion in log2n Because thatrsquos how many times we can
halve n However there are many recursions at
each level How can we figure this out We note that
Each partition is linear over its subarray All the partitions at one level cover the array
Partitioning at various levels
Best case II We cut the array size in half each time So the depth of the recursion in log2n At each level of the recursion all the
partitions at that level do work that is linear in n
O(log2n) O(n) = O(n log2n)
What about the worst case
Worst case In the worst case partitioning always
divides the size n array into these three parts A length one part containing the pivot itself A length zero part and A length n-1 part containing everything else
We donrsquot recur on the zero-length part Recurring on the length n-1 part requires
(in the worst case) recurring to depth n-1
Worst case partitioning
Worst case for quicksort In the worst case recursion may be
O(n) levels deep (for an array of size n)
But the partitioning work done at each level is still O(n)
O(n) O(n) = O(n2) So worst case for Quicksort is O(n2) When does this happen
When the array is sorted to begin with
Typical case for quicksort If the array is sorted to begin with
Quicksort is terrible O(n2) It is possible to construct other bad
cases However Quicksort is usually O(n log2n) The constants are so good that Quicksort
is generally the fastest algorithm known Most real-world sorting is done by
Quicksort
Tweaking Quicksort Almost anything you can try to
ldquoimproverdquo Quicksort will actually slow it down
One good tweak is to switch to a different sorting method when the subarrays get small (say 10 or 12) Quicksort has too much overhead for small
array sizes For large arrays it might be a good
idea to check beforehand if the array is already sorted But there is a better tweak than this
Randomized Quick-Sort Select the pivot as a random element of the sequence The expected running time of randomized quick-sort on a
sequence of size n is O(n log n) The time spent at a level of the quick-sort tree is O(n) We show that the expected height of the quick-sort tree is
O(log n) good vs bad pivots
bull The probability of a good pivot is 12 thus we expect k2 good pivots out of k pivotsbull After a good pivot the size of each child sequence is at most 34 the size of the parent sequencebull After h pivots we expect (34)h2 n elementsbull The expected height h of the quick-sort tree is at most 2 log43n
Median of three Obviously it doesnrsquot make sense to sort the
array in order to find the median to use as a pivot
Instead compare just three elements of our (sub)arraymdashthe first the last and the middle Take the median (middle value) of these three as
pivot Itrsquos possible (but not easy) to construct cases
which will make this technique O(n2) Suppose we rearrange (sort) these three
numbers so that the smallest is in the first position the largest in the last position and the other in the middle This lets us simplify and speed up the partition
loop
Final comments Weissrsquos code shows some additional
optimizations on pp 246-247 Weiss chooses to stop both searches on
equality to pivot This design decision is debatable
Quicksort is the fastest known general sorting algorithm on average
For optimum speed the pivot must be chosen carefully
ldquoMedian of threerdquo is a good technique for choosing the pivot
There will be some cases where Quicksort runs in O(n2) time
Quick Sort A couple of notes about quick sort There are more optimal ways to
choose the pivot value (such as the median-of-three method)
Also when the subarrays get small it becomes more efficient to use the insertion sort as opposed to continued use of quick sort
Bubble Sort vs Quick Sort If we calculate the Big-O notation
we find that (in the average case) Bubble Sort O(n2) Quick Sort O(nlog2n)
(quick-sort runs in time O(n2) in the worst case)
Heap Sort Idea take the items that need to be
sorted and insert them into the heap By calling deleteHeap we remove the
smallest or largest element depending on whether or not we are working with a min- or max-heap respectively
Hence the elements are removed in ascending or descending order
Efficiency O(nlog2n)
Merge Sort Idea
Take the array you would like to sort and divide it in half to create 2 unsorted subarrays
Next sort each of the 2 subarrays Finally merge the 2 sorted subarrays
into 1 sorted array Efficiency O(nlog2n)
Merge Sort
Merge Sort Although the merge step produces
a sorted array we have overlooked a very important step
How did we sort the 2 halves before performing the merge step
We used merge sort
Merge Sort By continually calling the merge
sort algorithm we eventually get a subarray of size 1
Since an array with only 1 element is clearly sorted we can back out and merge 2 arrays of size 1
Merge Sort
Merge Sort The basic merging algorithm
consists of 2 input arrays (arrayA and arrayB) An ouput array (arrayC) 3 position holders (indexA indexB
indexC) which are initially set to the beginning of their respective arrays
Merge Sort The smaller of arrayA[indexA] and
arrayB[indexB] is copied into arrayC[indexC] and the appropriate position holders are advanced
When either input list is exhausted the remainder of the other list is copied into arrayC
Merge Sort
1 13 24 26index
A
indexC
arrayA
2 15 27 38index
B
arrayB
arrayC
We compare arrayA[indexA]with arrayB[indexB] Whichever value is smaller isplaced into arrayC[indexC]
1 lt 2 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1index
C
arrayA
2 15 27 38index
B
arrayB
arrayC
2 lt 13 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
13 lt 15 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
15 lt 24 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13 15indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
24 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
indexA
1 2 13 15 24indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
26 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
1 2 13 15 24 26indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
Since we have exhaustedone of the arrays arrayAwe simply copy the remaining items from theother array arrayB intoarrayC
Merge Sort
1 13 24
26
1 2 13 15 24 26 27 38
arrayA
2 15 27 38arrayB
arrayC
Efficiency Summary
Sort Worst Case Average Case
Insertion O(n2) O(n2)
Shell O(n15) O(n125)
Selection O(n2) O(n2)
Bubble O(n2) O(n2)
Quick O(n2) O(nlog2n)
Heap O(nlog2n) O(nlog2n)
Merge O(nlog2n) O(nlog2n)
More Sorting
radix sort bucket sort in-place sorting how fast can we sort
Radix Sort
Unlike other sorting methods radix sort considers the structure of the keys
Assume keys are represented in a base M number system (M = radix) ie if M = 2 the keys are represented in binary
Sorting is done by comparing bits in the same position
Extension to keys that are alphanumeric strings
Radix Exchange Sort
Examine bits from left to right1 Sort array with respect to leftmost bit
2 Partition array
3Recursionrecursively sort top subarray ignoring leftmost bitrecursively sort bottom subarray ignoring leftmost bit
Time to sort n b-bit numbers O(b n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Bubble Sort In the bubble sort the list at any
moment is divided into 2 sublists sorted and unsorted
The smallest element is ldquobubbledrdquo from the unsorted sublist to the sorted sublist
Bubble Sort
23 78 45 8 56 3232 56
We start with 32 and compareit with 56 Because 32 is lessthan 56 we swap the twoand step down one element
We then compare 32 and 8Because 32 is not less than 8we do not swap these elements
We step down one element andcompare 45 and 8 They areout of sequence so we swapthem and step down again
8 45
We step down again and compare 8 with 78 These two elements are swapped
8 78
Finally 8 is compared with 23and swapped We thencontinue this process back with56 hellip
8 23
1 Pass of the Bubble Sort
Quick Sort In the bubble sort consecutive items
are compared and possibly exchanged on each pass through the list
This means that many exchanges may be needed to move an element to its correct position
Quick sort is more efficient than bubble sort because a typical exchange involves elements that are far apart so fewer exchanges are required to correctly position an element
Idea of Quick Sort1) Select pick an element
2) Divide rearrange elements so that x goes to its final position E
3) Recurse and Conquer recursively sort
Quick Sort Each iteration of the quick sort
selects an element known as the pivot and divides the list into 3 groups Elements whose keys are less than (or
equal to) the pivotrsquos key The pivot element Elements whose keys are greater than
(or equal to) the pivotrsquos key
Quick Sort The sorting then continues by
quick sorting the left partition followed by quick sorting the right partition
The basic algorithm is as follows
Quick Sort1) Partitioning Step Take an element in the
unsorted array and determine its final location in the sorted array This occurs when all values to the left of the element in the array are less than (or equal to) the element and all values to the right of the element are greater than (or equal to) the element We now have 1 element in its proper location and two unsorted subarrays
2) Recursive Step Perform step 1 on each unsorted subarray
Quick Sort Each time step 1 is performed on a
subarray another element is placed in its final location of the sorted array and two unsorted subarrays are created
When a subarray consists of one element that subarray is sorted
Therefore that element is in its final location
Quick Sort There are several partitioning strategies
used in practice (ie several ldquoversionsrdquo of quick sort) but the one we are about to describe is known to work well
For simplicity we will choose the last element to be the pivot element
We could also chose a different pivot element and swap it with the last element in the array
Quick Sort Below is the array we would like to
sort1 4 8 9 0 11 5 10 7 6
Quick Sort The index left starts at the first element
and right starts at the next-to-last element
We want to move all the elements smaller than the pivot to the left part of the array and all the elements larger than the pivot to the right part
1 4 8 9 0 11 5 10 7 6left righ
t
Quick Sort We move left to the right skipping
over elements that are smaller than the pivot
1 4 8 9 0 11 5 10 7 6left righ
t
Quick Sort We then move right to the left skipping
over elements that are greater than the pivot
When left and right have stopped left is on an element greater than (or equal to) the pivot and right is on an element smaller than (or equal to) the pivot
1 4 8 9 0 11 5 10 7 6left right
Quick Sort If left is to the left of right (or if left
= right) those elements are swapped
1 4 8 9 0 11 5 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
Quick Sort The effect is to push a large
element to the right and a small element to the left
We then repeat the process until left and right cross
Quick Sort
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6left righ
t
Quick Sort
1 4 5 0 9 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6righ
tleft
Quick Sort
At this point left and right have crossed so no swap is performed
The final part of the partitioning is to swap the pivot element with left
1 4 5 0 9 11 8 10 7 6righ
tleft
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort Note that all elements to the left of
the pivot are less than (or equal to) the pivot and all elements to the right of the pivot are greater than (or equal to) the pivot
Hence the pivot element has been placed in its final sorted position
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort We now repeat the process using
the sub-arrays to the left and right of the pivot
1 4 5 0 6 11 8 10 7 9
1 4 5 0 11 8 10 7 9
Analysis of quicksortmdashbest case
Suppose each partition operation divides the array almost exactly in half
Then the depth of the recursion in log2n Because thatrsquos how many times we can
halve n However there are many recursions at
each level How can we figure this out We note that
Each partition is linear over its subarray All the partitions at one level cover the array
Partitioning at various levels
Best case II We cut the array size in half each time So the depth of the recursion in log2n At each level of the recursion all the
partitions at that level do work that is linear in n
O(log2n) O(n) = O(n log2n)
What about the worst case
Worst case In the worst case partitioning always
divides the size n array into these three parts A length one part containing the pivot itself A length zero part and A length n-1 part containing everything else
We donrsquot recur on the zero-length part Recurring on the length n-1 part requires
(in the worst case) recurring to depth n-1
Worst case partitioning
Worst case for quicksort In the worst case recursion may be
O(n) levels deep (for an array of size n)
But the partitioning work done at each level is still O(n)
O(n) O(n) = O(n2) So worst case for Quicksort is O(n2) When does this happen
When the array is sorted to begin with
Typical case for quicksort If the array is sorted to begin with
Quicksort is terrible O(n2) It is possible to construct other bad
cases However Quicksort is usually O(n log2n) The constants are so good that Quicksort
is generally the fastest algorithm known Most real-world sorting is done by
Quicksort
Tweaking Quicksort Almost anything you can try to
ldquoimproverdquo Quicksort will actually slow it down
One good tweak is to switch to a different sorting method when the subarrays get small (say 10 or 12) Quicksort has too much overhead for small
array sizes For large arrays it might be a good
idea to check beforehand if the array is already sorted But there is a better tweak than this
Randomized Quick-Sort Select the pivot as a random element of the sequence The expected running time of randomized quick-sort on a
sequence of size n is O(n log n) The time spent at a level of the quick-sort tree is O(n) We show that the expected height of the quick-sort tree is
O(log n) good vs bad pivots
bull The probability of a good pivot is 12 thus we expect k2 good pivots out of k pivotsbull After a good pivot the size of each child sequence is at most 34 the size of the parent sequencebull After h pivots we expect (34)h2 n elementsbull The expected height h of the quick-sort tree is at most 2 log43n
Median of three Obviously it doesnrsquot make sense to sort the
array in order to find the median to use as a pivot
Instead compare just three elements of our (sub)arraymdashthe first the last and the middle Take the median (middle value) of these three as
pivot Itrsquos possible (but not easy) to construct cases
which will make this technique O(n2) Suppose we rearrange (sort) these three
numbers so that the smallest is in the first position the largest in the last position and the other in the middle This lets us simplify and speed up the partition
loop
Final comments Weissrsquos code shows some additional
optimizations on pp 246-247 Weiss chooses to stop both searches on
equality to pivot This design decision is debatable
Quicksort is the fastest known general sorting algorithm on average
For optimum speed the pivot must be chosen carefully
ldquoMedian of threerdquo is a good technique for choosing the pivot
There will be some cases where Quicksort runs in O(n2) time
Quick Sort A couple of notes about quick sort There are more optimal ways to
choose the pivot value (such as the median-of-three method)
Also when the subarrays get small it becomes more efficient to use the insertion sort as opposed to continued use of quick sort
Bubble Sort vs Quick Sort If we calculate the Big-O notation
we find that (in the average case) Bubble Sort O(n2) Quick Sort O(nlog2n)
(quick-sort runs in time O(n2) in the worst case)
Heap Sort Idea take the items that need to be
sorted and insert them into the heap By calling deleteHeap we remove the
smallest or largest element depending on whether or not we are working with a min- or max-heap respectively
Hence the elements are removed in ascending or descending order
Efficiency O(nlog2n)
Merge Sort Idea
Take the array you would like to sort and divide it in half to create 2 unsorted subarrays
Next sort each of the 2 subarrays Finally merge the 2 sorted subarrays
into 1 sorted array Efficiency O(nlog2n)
Merge Sort
Merge Sort Although the merge step produces
a sorted array we have overlooked a very important step
How did we sort the 2 halves before performing the merge step
We used merge sort
Merge Sort By continually calling the merge
sort algorithm we eventually get a subarray of size 1
Since an array with only 1 element is clearly sorted we can back out and merge 2 arrays of size 1
Merge Sort
Merge Sort The basic merging algorithm
consists of 2 input arrays (arrayA and arrayB) An ouput array (arrayC) 3 position holders (indexA indexB
indexC) which are initially set to the beginning of their respective arrays
Merge Sort The smaller of arrayA[indexA] and
arrayB[indexB] is copied into arrayC[indexC] and the appropriate position holders are advanced
When either input list is exhausted the remainder of the other list is copied into arrayC
Merge Sort
1 13 24 26index
A
indexC
arrayA
2 15 27 38index
B
arrayB
arrayC
We compare arrayA[indexA]with arrayB[indexB] Whichever value is smaller isplaced into arrayC[indexC]
1 lt 2 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1index
C
arrayA
2 15 27 38index
B
arrayB
arrayC
2 lt 13 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
13 lt 15 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
15 lt 24 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13 15indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
24 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
indexA
1 2 13 15 24indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
26 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
1 2 13 15 24 26indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
Since we have exhaustedone of the arrays arrayAwe simply copy the remaining items from theother array arrayB intoarrayC
Merge Sort
1 13 24
26
1 2 13 15 24 26 27 38
arrayA
2 15 27 38arrayB
arrayC
Efficiency Summary
Sort Worst Case Average Case
Insertion O(n2) O(n2)
Shell O(n15) O(n125)
Selection O(n2) O(n2)
Bubble O(n2) O(n2)
Quick O(n2) O(nlog2n)
Heap O(nlog2n) O(nlog2n)
Merge O(nlog2n) O(nlog2n)
More Sorting
radix sort bucket sort in-place sorting how fast can we sort
Radix Sort
Unlike other sorting methods radix sort considers the structure of the keys
Assume keys are represented in a base M number system (M = radix) ie if M = 2 the keys are represented in binary
Sorting is done by comparing bits in the same position
Extension to keys that are alphanumeric strings
Radix Exchange Sort
Examine bits from left to right1 Sort array with respect to leftmost bit
2 Partition array
3Recursionrecursively sort top subarray ignoring leftmost bitrecursively sort bottom subarray ignoring leftmost bit
Time to sort n b-bit numbers O(b n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Bubble Sort
23 78 45 8 56 3232 56
We start with 32 and compareit with 56 Because 32 is lessthan 56 we swap the twoand step down one element
We then compare 32 and 8Because 32 is not less than 8we do not swap these elements
We step down one element andcompare 45 and 8 They areout of sequence so we swapthem and step down again
8 45
We step down again and compare 8 with 78 These two elements are swapped
8 78
Finally 8 is compared with 23and swapped We thencontinue this process back with56 hellip
8 23
1 Pass of the Bubble Sort
Quick Sort In the bubble sort consecutive items
are compared and possibly exchanged on each pass through the list
This means that many exchanges may be needed to move an element to its correct position
Quick sort is more efficient than bubble sort because a typical exchange involves elements that are far apart so fewer exchanges are required to correctly position an element
Idea of Quick Sort1) Select pick an element
2) Divide rearrange elements so that x goes to its final position E
3) Recurse and Conquer recursively sort
Quick Sort Each iteration of the quick sort
selects an element known as the pivot and divides the list into 3 groups Elements whose keys are less than (or
equal to) the pivotrsquos key The pivot element Elements whose keys are greater than
(or equal to) the pivotrsquos key
Quick Sort The sorting then continues by
quick sorting the left partition followed by quick sorting the right partition
The basic algorithm is as follows
Quick Sort1) Partitioning Step Take an element in the
unsorted array and determine its final location in the sorted array This occurs when all values to the left of the element in the array are less than (or equal to) the element and all values to the right of the element are greater than (or equal to) the element We now have 1 element in its proper location and two unsorted subarrays
2) Recursive Step Perform step 1 on each unsorted subarray
Quick Sort Each time step 1 is performed on a
subarray another element is placed in its final location of the sorted array and two unsorted subarrays are created
When a subarray consists of one element that subarray is sorted
Therefore that element is in its final location
Quick Sort There are several partitioning strategies
used in practice (ie several ldquoversionsrdquo of quick sort) but the one we are about to describe is known to work well
For simplicity we will choose the last element to be the pivot element
We could also chose a different pivot element and swap it with the last element in the array
Quick Sort Below is the array we would like to
sort1 4 8 9 0 11 5 10 7 6
Quick Sort The index left starts at the first element
and right starts at the next-to-last element
We want to move all the elements smaller than the pivot to the left part of the array and all the elements larger than the pivot to the right part
1 4 8 9 0 11 5 10 7 6left righ
t
Quick Sort We move left to the right skipping
over elements that are smaller than the pivot
1 4 8 9 0 11 5 10 7 6left righ
t
Quick Sort We then move right to the left skipping
over elements that are greater than the pivot
When left and right have stopped left is on an element greater than (or equal to) the pivot and right is on an element smaller than (or equal to) the pivot
1 4 8 9 0 11 5 10 7 6left right
Quick Sort If left is to the left of right (or if left
= right) those elements are swapped
1 4 8 9 0 11 5 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
Quick Sort The effect is to push a large
element to the right and a small element to the left
We then repeat the process until left and right cross
Quick Sort
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6left righ
t
Quick Sort
1 4 5 0 9 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6righ
tleft
Quick Sort
At this point left and right have crossed so no swap is performed
The final part of the partitioning is to swap the pivot element with left
1 4 5 0 9 11 8 10 7 6righ
tleft
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort Note that all elements to the left of
the pivot are less than (or equal to) the pivot and all elements to the right of the pivot are greater than (or equal to) the pivot
Hence the pivot element has been placed in its final sorted position
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort We now repeat the process using
the sub-arrays to the left and right of the pivot
1 4 5 0 6 11 8 10 7 9
1 4 5 0 11 8 10 7 9
Analysis of quicksortmdashbest case
Suppose each partition operation divides the array almost exactly in half
Then the depth of the recursion in log2n Because thatrsquos how many times we can
halve n However there are many recursions at
each level How can we figure this out We note that
Each partition is linear over its subarray All the partitions at one level cover the array
Partitioning at various levels
Best case II We cut the array size in half each time So the depth of the recursion in log2n At each level of the recursion all the
partitions at that level do work that is linear in n
O(log2n) O(n) = O(n log2n)
What about the worst case
Worst case In the worst case partitioning always
divides the size n array into these three parts A length one part containing the pivot itself A length zero part and A length n-1 part containing everything else
We donrsquot recur on the zero-length part Recurring on the length n-1 part requires
(in the worst case) recurring to depth n-1
Worst case partitioning
Worst case for quicksort In the worst case recursion may be
O(n) levels deep (for an array of size n)
But the partitioning work done at each level is still O(n)
O(n) O(n) = O(n2) So worst case for Quicksort is O(n2) When does this happen
When the array is sorted to begin with
Typical case for quicksort If the array is sorted to begin with
Quicksort is terrible O(n2) It is possible to construct other bad
cases However Quicksort is usually O(n log2n) The constants are so good that Quicksort
is generally the fastest algorithm known Most real-world sorting is done by
Quicksort
Tweaking Quicksort Almost anything you can try to
ldquoimproverdquo Quicksort will actually slow it down
One good tweak is to switch to a different sorting method when the subarrays get small (say 10 or 12) Quicksort has too much overhead for small
array sizes For large arrays it might be a good
idea to check beforehand if the array is already sorted But there is a better tweak than this
Randomized Quick-Sort Select the pivot as a random element of the sequence The expected running time of randomized quick-sort on a
sequence of size n is O(n log n) The time spent at a level of the quick-sort tree is O(n) We show that the expected height of the quick-sort tree is
O(log n) good vs bad pivots
bull The probability of a good pivot is 12 thus we expect k2 good pivots out of k pivotsbull After a good pivot the size of each child sequence is at most 34 the size of the parent sequencebull After h pivots we expect (34)h2 n elementsbull The expected height h of the quick-sort tree is at most 2 log43n
Median of three Obviously it doesnrsquot make sense to sort the
array in order to find the median to use as a pivot
Instead compare just three elements of our (sub)arraymdashthe first the last and the middle Take the median (middle value) of these three as
pivot Itrsquos possible (but not easy) to construct cases
which will make this technique O(n2) Suppose we rearrange (sort) these three
numbers so that the smallest is in the first position the largest in the last position and the other in the middle This lets us simplify and speed up the partition
loop
Final comments Weissrsquos code shows some additional
optimizations on pp 246-247 Weiss chooses to stop both searches on
equality to pivot This design decision is debatable
Quicksort is the fastest known general sorting algorithm on average
For optimum speed the pivot must be chosen carefully
ldquoMedian of threerdquo is a good technique for choosing the pivot
There will be some cases where Quicksort runs in O(n2) time
Quick Sort A couple of notes about quick sort There are more optimal ways to
choose the pivot value (such as the median-of-three method)
Also when the subarrays get small it becomes more efficient to use the insertion sort as opposed to continued use of quick sort
Bubble Sort vs Quick Sort If we calculate the Big-O notation
we find that (in the average case) Bubble Sort O(n2) Quick Sort O(nlog2n)
(quick-sort runs in time O(n2) in the worst case)
Heap Sort Idea take the items that need to be
sorted and insert them into the heap By calling deleteHeap we remove the
smallest or largest element depending on whether or not we are working with a min- or max-heap respectively
Hence the elements are removed in ascending or descending order
Efficiency O(nlog2n)
Merge Sort Idea
Take the array you would like to sort and divide it in half to create 2 unsorted subarrays
Next sort each of the 2 subarrays Finally merge the 2 sorted subarrays
into 1 sorted array Efficiency O(nlog2n)
Merge Sort
Merge Sort Although the merge step produces
a sorted array we have overlooked a very important step
How did we sort the 2 halves before performing the merge step
We used merge sort
Merge Sort By continually calling the merge
sort algorithm we eventually get a subarray of size 1
Since an array with only 1 element is clearly sorted we can back out and merge 2 arrays of size 1
Merge Sort
Merge Sort The basic merging algorithm
consists of 2 input arrays (arrayA and arrayB) An ouput array (arrayC) 3 position holders (indexA indexB
indexC) which are initially set to the beginning of their respective arrays
Merge Sort The smaller of arrayA[indexA] and
arrayB[indexB] is copied into arrayC[indexC] and the appropriate position holders are advanced
When either input list is exhausted the remainder of the other list is copied into arrayC
Merge Sort
1 13 24 26index
A
indexC
arrayA
2 15 27 38index
B
arrayB
arrayC
We compare arrayA[indexA]with arrayB[indexB] Whichever value is smaller isplaced into arrayC[indexC]
1 lt 2 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1index
C
arrayA
2 15 27 38index
B
arrayB
arrayC
2 lt 13 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
13 lt 15 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
15 lt 24 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13 15indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
24 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
indexA
1 2 13 15 24indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
26 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
1 2 13 15 24 26indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
Since we have exhaustedone of the arrays arrayAwe simply copy the remaining items from theother array arrayB intoarrayC
Merge Sort
1 13 24
26
1 2 13 15 24 26 27 38
arrayA
2 15 27 38arrayB
arrayC
Efficiency Summary
Sort Worst Case Average Case
Insertion O(n2) O(n2)
Shell O(n15) O(n125)
Selection O(n2) O(n2)
Bubble O(n2) O(n2)
Quick O(n2) O(nlog2n)
Heap O(nlog2n) O(nlog2n)
Merge O(nlog2n) O(nlog2n)
More Sorting
radix sort bucket sort in-place sorting how fast can we sort
Radix Sort
Unlike other sorting methods radix sort considers the structure of the keys
Assume keys are represented in a base M number system (M = radix) ie if M = 2 the keys are represented in binary
Sorting is done by comparing bits in the same position
Extension to keys that are alphanumeric strings
Radix Exchange Sort
Examine bits from left to right1 Sort array with respect to leftmost bit
2 Partition array
3Recursionrecursively sort top subarray ignoring leftmost bitrecursively sort bottom subarray ignoring leftmost bit
Time to sort n b-bit numbers O(b n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Quick Sort In the bubble sort consecutive items
are compared and possibly exchanged on each pass through the list
This means that many exchanges may be needed to move an element to its correct position
Quick sort is more efficient than bubble sort because a typical exchange involves elements that are far apart so fewer exchanges are required to correctly position an element
Idea of Quick Sort1) Select pick an element
2) Divide rearrange elements so that x goes to its final position E
3) Recurse and Conquer recursively sort
Quick Sort Each iteration of the quick sort
selects an element known as the pivot and divides the list into 3 groups Elements whose keys are less than (or
equal to) the pivotrsquos key The pivot element Elements whose keys are greater than
(or equal to) the pivotrsquos key
Quick Sort The sorting then continues by
quick sorting the left partition followed by quick sorting the right partition
The basic algorithm is as follows
Quick Sort1) Partitioning Step Take an element in the
unsorted array and determine its final location in the sorted array This occurs when all values to the left of the element in the array are less than (or equal to) the element and all values to the right of the element are greater than (or equal to) the element We now have 1 element in its proper location and two unsorted subarrays
2) Recursive Step Perform step 1 on each unsorted subarray
Quick Sort Each time step 1 is performed on a
subarray another element is placed in its final location of the sorted array and two unsorted subarrays are created
When a subarray consists of one element that subarray is sorted
Therefore that element is in its final location
Quick Sort There are several partitioning strategies
used in practice (ie several ldquoversionsrdquo of quick sort) but the one we are about to describe is known to work well
For simplicity we will choose the last element to be the pivot element
We could also chose a different pivot element and swap it with the last element in the array
Quick Sort Below is the array we would like to
sort1 4 8 9 0 11 5 10 7 6
Quick Sort The index left starts at the first element
and right starts at the next-to-last element
We want to move all the elements smaller than the pivot to the left part of the array and all the elements larger than the pivot to the right part
1 4 8 9 0 11 5 10 7 6left righ
t
Quick Sort We move left to the right skipping
over elements that are smaller than the pivot
1 4 8 9 0 11 5 10 7 6left righ
t
Quick Sort We then move right to the left skipping
over elements that are greater than the pivot
When left and right have stopped left is on an element greater than (or equal to) the pivot and right is on an element smaller than (or equal to) the pivot
1 4 8 9 0 11 5 10 7 6left right
Quick Sort If left is to the left of right (or if left
= right) those elements are swapped
1 4 8 9 0 11 5 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
Quick Sort The effect is to push a large
element to the right and a small element to the left
We then repeat the process until left and right cross
Quick Sort
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6left righ
t
Quick Sort
1 4 5 0 9 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6righ
tleft
Quick Sort
At this point left and right have crossed so no swap is performed
The final part of the partitioning is to swap the pivot element with left
1 4 5 0 9 11 8 10 7 6righ
tleft
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort Note that all elements to the left of
the pivot are less than (or equal to) the pivot and all elements to the right of the pivot are greater than (or equal to) the pivot
Hence the pivot element has been placed in its final sorted position
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort We now repeat the process using
the sub-arrays to the left and right of the pivot
1 4 5 0 6 11 8 10 7 9
1 4 5 0 11 8 10 7 9
Analysis of quicksortmdashbest case
Suppose each partition operation divides the array almost exactly in half
Then the depth of the recursion in log2n Because thatrsquos how many times we can
halve n However there are many recursions at
each level How can we figure this out We note that
Each partition is linear over its subarray All the partitions at one level cover the array
Partitioning at various levels
Best case II We cut the array size in half each time So the depth of the recursion in log2n At each level of the recursion all the
partitions at that level do work that is linear in n
O(log2n) O(n) = O(n log2n)
What about the worst case
Worst case In the worst case partitioning always
divides the size n array into these three parts A length one part containing the pivot itself A length zero part and A length n-1 part containing everything else
We donrsquot recur on the zero-length part Recurring on the length n-1 part requires
(in the worst case) recurring to depth n-1
Worst case partitioning
Worst case for quicksort In the worst case recursion may be
O(n) levels deep (for an array of size n)
But the partitioning work done at each level is still O(n)
O(n) O(n) = O(n2) So worst case for Quicksort is O(n2) When does this happen
When the array is sorted to begin with
Typical case for quicksort If the array is sorted to begin with
Quicksort is terrible O(n2) It is possible to construct other bad
cases However Quicksort is usually O(n log2n) The constants are so good that Quicksort
is generally the fastest algorithm known Most real-world sorting is done by
Quicksort
Tweaking Quicksort Almost anything you can try to
ldquoimproverdquo Quicksort will actually slow it down
One good tweak is to switch to a different sorting method when the subarrays get small (say 10 or 12) Quicksort has too much overhead for small
array sizes For large arrays it might be a good
idea to check beforehand if the array is already sorted But there is a better tweak than this
Randomized Quick-Sort Select the pivot as a random element of the sequence The expected running time of randomized quick-sort on a
sequence of size n is O(n log n) The time spent at a level of the quick-sort tree is O(n) We show that the expected height of the quick-sort tree is
O(log n) good vs bad pivots
bull The probability of a good pivot is 12 thus we expect k2 good pivots out of k pivotsbull After a good pivot the size of each child sequence is at most 34 the size of the parent sequencebull After h pivots we expect (34)h2 n elementsbull The expected height h of the quick-sort tree is at most 2 log43n
Median of three Obviously it doesnrsquot make sense to sort the
array in order to find the median to use as a pivot
Instead compare just three elements of our (sub)arraymdashthe first the last and the middle Take the median (middle value) of these three as
pivot Itrsquos possible (but not easy) to construct cases
which will make this technique O(n2) Suppose we rearrange (sort) these three
numbers so that the smallest is in the first position the largest in the last position and the other in the middle This lets us simplify and speed up the partition
loop
Final comments Weissrsquos code shows some additional
optimizations on pp 246-247 Weiss chooses to stop both searches on
equality to pivot This design decision is debatable
Quicksort is the fastest known general sorting algorithm on average
For optimum speed the pivot must be chosen carefully
ldquoMedian of threerdquo is a good technique for choosing the pivot
There will be some cases where Quicksort runs in O(n2) time
Quick Sort A couple of notes about quick sort There are more optimal ways to
choose the pivot value (such as the median-of-three method)
Also when the subarrays get small it becomes more efficient to use the insertion sort as opposed to continued use of quick sort
Bubble Sort vs Quick Sort If we calculate the Big-O notation
we find that (in the average case) Bubble Sort O(n2) Quick Sort O(nlog2n)
(quick-sort runs in time O(n2) in the worst case)
Heap Sort Idea take the items that need to be
sorted and insert them into the heap By calling deleteHeap we remove the
smallest or largest element depending on whether or not we are working with a min- or max-heap respectively
Hence the elements are removed in ascending or descending order
Efficiency O(nlog2n)
Merge Sort Idea
Take the array you would like to sort and divide it in half to create 2 unsorted subarrays
Next sort each of the 2 subarrays Finally merge the 2 sorted subarrays
into 1 sorted array Efficiency O(nlog2n)
Merge Sort
Merge Sort Although the merge step produces
a sorted array we have overlooked a very important step
How did we sort the 2 halves before performing the merge step
We used merge sort
Merge Sort By continually calling the merge
sort algorithm we eventually get a subarray of size 1
Since an array with only 1 element is clearly sorted we can back out and merge 2 arrays of size 1
Merge Sort
Merge Sort The basic merging algorithm
consists of 2 input arrays (arrayA and arrayB) An ouput array (arrayC) 3 position holders (indexA indexB
indexC) which are initially set to the beginning of their respective arrays
Merge Sort The smaller of arrayA[indexA] and
arrayB[indexB] is copied into arrayC[indexC] and the appropriate position holders are advanced
When either input list is exhausted the remainder of the other list is copied into arrayC
Merge Sort
1 13 24 26index
A
indexC
arrayA
2 15 27 38index
B
arrayB
arrayC
We compare arrayA[indexA]with arrayB[indexB] Whichever value is smaller isplaced into arrayC[indexC]
1 lt 2 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1index
C
arrayA
2 15 27 38index
B
arrayB
arrayC
2 lt 13 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
13 lt 15 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
15 lt 24 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13 15indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
24 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
indexA
1 2 13 15 24indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
26 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
1 2 13 15 24 26indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
Since we have exhaustedone of the arrays arrayAwe simply copy the remaining items from theother array arrayB intoarrayC
Merge Sort
1 13 24
26
1 2 13 15 24 26 27 38
arrayA
2 15 27 38arrayB
arrayC
Efficiency Summary
Sort Worst Case Average Case
Insertion O(n2) O(n2)
Shell O(n15) O(n125)
Selection O(n2) O(n2)
Bubble O(n2) O(n2)
Quick O(n2) O(nlog2n)
Heap O(nlog2n) O(nlog2n)
Merge O(nlog2n) O(nlog2n)
More Sorting
radix sort bucket sort in-place sorting how fast can we sort
Radix Sort
Unlike other sorting methods radix sort considers the structure of the keys
Assume keys are represented in a base M number system (M = radix) ie if M = 2 the keys are represented in binary
Sorting is done by comparing bits in the same position
Extension to keys that are alphanumeric strings
Radix Exchange Sort
Examine bits from left to right1 Sort array with respect to leftmost bit
2 Partition array
3Recursionrecursively sort top subarray ignoring leftmost bitrecursively sort bottom subarray ignoring leftmost bit
Time to sort n b-bit numbers O(b n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Idea of Quick Sort1) Select pick an element
2) Divide rearrange elements so that x goes to its final position E
3) Recurse and Conquer recursively sort
Quick Sort Each iteration of the quick sort
selects an element known as the pivot and divides the list into 3 groups Elements whose keys are less than (or
equal to) the pivotrsquos key The pivot element Elements whose keys are greater than
(or equal to) the pivotrsquos key
Quick Sort The sorting then continues by
quick sorting the left partition followed by quick sorting the right partition
The basic algorithm is as follows
Quick Sort1) Partitioning Step Take an element in the
unsorted array and determine its final location in the sorted array This occurs when all values to the left of the element in the array are less than (or equal to) the element and all values to the right of the element are greater than (or equal to) the element We now have 1 element in its proper location and two unsorted subarrays
2) Recursive Step Perform step 1 on each unsorted subarray
Quick Sort Each time step 1 is performed on a
subarray another element is placed in its final location of the sorted array and two unsorted subarrays are created
When a subarray consists of one element that subarray is sorted
Therefore that element is in its final location
Quick Sort There are several partitioning strategies
used in practice (ie several ldquoversionsrdquo of quick sort) but the one we are about to describe is known to work well
For simplicity we will choose the last element to be the pivot element
We could also chose a different pivot element and swap it with the last element in the array
Quick Sort Below is the array we would like to
sort1 4 8 9 0 11 5 10 7 6
Quick Sort The index left starts at the first element
and right starts at the next-to-last element
We want to move all the elements smaller than the pivot to the left part of the array and all the elements larger than the pivot to the right part
1 4 8 9 0 11 5 10 7 6left righ
t
Quick Sort We move left to the right skipping
over elements that are smaller than the pivot
1 4 8 9 0 11 5 10 7 6left righ
t
Quick Sort We then move right to the left skipping
over elements that are greater than the pivot
When left and right have stopped left is on an element greater than (or equal to) the pivot and right is on an element smaller than (or equal to) the pivot
1 4 8 9 0 11 5 10 7 6left right
Quick Sort If left is to the left of right (or if left
= right) those elements are swapped
1 4 8 9 0 11 5 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
Quick Sort The effect is to push a large
element to the right and a small element to the left
We then repeat the process until left and right cross
Quick Sort
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6left righ
t
Quick Sort
1 4 5 0 9 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6righ
tleft
Quick Sort
At this point left and right have crossed so no swap is performed
The final part of the partitioning is to swap the pivot element with left
1 4 5 0 9 11 8 10 7 6righ
tleft
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort Note that all elements to the left of
the pivot are less than (or equal to) the pivot and all elements to the right of the pivot are greater than (or equal to) the pivot
Hence the pivot element has been placed in its final sorted position
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort We now repeat the process using
the sub-arrays to the left and right of the pivot
1 4 5 0 6 11 8 10 7 9
1 4 5 0 11 8 10 7 9
Analysis of quicksortmdashbest case
Suppose each partition operation divides the array almost exactly in half
Then the depth of the recursion in log2n Because thatrsquos how many times we can
halve n However there are many recursions at
each level How can we figure this out We note that
Each partition is linear over its subarray All the partitions at one level cover the array
Partitioning at various levels
Best case II We cut the array size in half each time So the depth of the recursion in log2n At each level of the recursion all the
partitions at that level do work that is linear in n
O(log2n) O(n) = O(n log2n)
What about the worst case
Worst case In the worst case partitioning always
divides the size n array into these three parts A length one part containing the pivot itself A length zero part and A length n-1 part containing everything else
We donrsquot recur on the zero-length part Recurring on the length n-1 part requires
(in the worst case) recurring to depth n-1
Worst case partitioning
Worst case for quicksort In the worst case recursion may be
O(n) levels deep (for an array of size n)
But the partitioning work done at each level is still O(n)
O(n) O(n) = O(n2) So worst case for Quicksort is O(n2) When does this happen
When the array is sorted to begin with
Typical case for quicksort If the array is sorted to begin with
Quicksort is terrible O(n2) It is possible to construct other bad
cases However Quicksort is usually O(n log2n) The constants are so good that Quicksort
is generally the fastest algorithm known Most real-world sorting is done by
Quicksort
Tweaking Quicksort Almost anything you can try to
ldquoimproverdquo Quicksort will actually slow it down
One good tweak is to switch to a different sorting method when the subarrays get small (say 10 or 12) Quicksort has too much overhead for small
array sizes For large arrays it might be a good
idea to check beforehand if the array is already sorted But there is a better tweak than this
Randomized Quick-Sort Select the pivot as a random element of the sequence The expected running time of randomized quick-sort on a
sequence of size n is O(n log n) The time spent at a level of the quick-sort tree is O(n) We show that the expected height of the quick-sort tree is
O(log n) good vs bad pivots
bull The probability of a good pivot is 12 thus we expect k2 good pivots out of k pivotsbull After a good pivot the size of each child sequence is at most 34 the size of the parent sequencebull After h pivots we expect (34)h2 n elementsbull The expected height h of the quick-sort tree is at most 2 log43n
Median of three Obviously it doesnrsquot make sense to sort the
array in order to find the median to use as a pivot
Instead compare just three elements of our (sub)arraymdashthe first the last and the middle Take the median (middle value) of these three as
pivot Itrsquos possible (but not easy) to construct cases
which will make this technique O(n2) Suppose we rearrange (sort) these three
numbers so that the smallest is in the first position the largest in the last position and the other in the middle This lets us simplify and speed up the partition
loop
Final comments Weissrsquos code shows some additional
optimizations on pp 246-247 Weiss chooses to stop both searches on
equality to pivot This design decision is debatable
Quicksort is the fastest known general sorting algorithm on average
For optimum speed the pivot must be chosen carefully
ldquoMedian of threerdquo is a good technique for choosing the pivot
There will be some cases where Quicksort runs in O(n2) time
Quick Sort A couple of notes about quick sort There are more optimal ways to
choose the pivot value (such as the median-of-three method)
Also when the subarrays get small it becomes more efficient to use the insertion sort as opposed to continued use of quick sort
Bubble Sort vs Quick Sort If we calculate the Big-O notation
we find that (in the average case) Bubble Sort O(n2) Quick Sort O(nlog2n)
(quick-sort runs in time O(n2) in the worst case)
Heap Sort Idea take the items that need to be
sorted and insert them into the heap By calling deleteHeap we remove the
smallest or largest element depending on whether or not we are working with a min- or max-heap respectively
Hence the elements are removed in ascending or descending order
Efficiency O(nlog2n)
Merge Sort Idea
Take the array you would like to sort and divide it in half to create 2 unsorted subarrays
Next sort each of the 2 subarrays Finally merge the 2 sorted subarrays
into 1 sorted array Efficiency O(nlog2n)
Merge Sort
Merge Sort Although the merge step produces
a sorted array we have overlooked a very important step
How did we sort the 2 halves before performing the merge step
We used merge sort
Merge Sort By continually calling the merge
sort algorithm we eventually get a subarray of size 1
Since an array with only 1 element is clearly sorted we can back out and merge 2 arrays of size 1
Merge Sort
Merge Sort The basic merging algorithm
consists of 2 input arrays (arrayA and arrayB) An ouput array (arrayC) 3 position holders (indexA indexB
indexC) which are initially set to the beginning of their respective arrays
Merge Sort The smaller of arrayA[indexA] and
arrayB[indexB] is copied into arrayC[indexC] and the appropriate position holders are advanced
When either input list is exhausted the remainder of the other list is copied into arrayC
Merge Sort
1 13 24 26index
A
indexC
arrayA
2 15 27 38index
B
arrayB
arrayC
We compare arrayA[indexA]with arrayB[indexB] Whichever value is smaller isplaced into arrayC[indexC]
1 lt 2 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1index
C
arrayA
2 15 27 38index
B
arrayB
arrayC
2 lt 13 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
13 lt 15 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
15 lt 24 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13 15indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
24 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
indexA
1 2 13 15 24indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
26 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
1 2 13 15 24 26indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
Since we have exhaustedone of the arrays arrayAwe simply copy the remaining items from theother array arrayB intoarrayC
Merge Sort
1 13 24
26
1 2 13 15 24 26 27 38
arrayA
2 15 27 38arrayB
arrayC
Efficiency Summary
Sort Worst Case Average Case
Insertion O(n2) O(n2)
Shell O(n15) O(n125)
Selection O(n2) O(n2)
Bubble O(n2) O(n2)
Quick O(n2) O(nlog2n)
Heap O(nlog2n) O(nlog2n)
Merge O(nlog2n) O(nlog2n)
More Sorting
radix sort bucket sort in-place sorting how fast can we sort
Radix Sort
Unlike other sorting methods radix sort considers the structure of the keys
Assume keys are represented in a base M number system (M = radix) ie if M = 2 the keys are represented in binary
Sorting is done by comparing bits in the same position
Extension to keys that are alphanumeric strings
Radix Exchange Sort
Examine bits from left to right1 Sort array with respect to leftmost bit
2 Partition array
3Recursionrecursively sort top subarray ignoring leftmost bitrecursively sort bottom subarray ignoring leftmost bit
Time to sort n b-bit numbers O(b n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Quick Sort Each iteration of the quick sort
selects an element known as the pivot and divides the list into 3 groups Elements whose keys are less than (or
equal to) the pivotrsquos key The pivot element Elements whose keys are greater than
(or equal to) the pivotrsquos key
Quick Sort The sorting then continues by
quick sorting the left partition followed by quick sorting the right partition
The basic algorithm is as follows
Quick Sort1) Partitioning Step Take an element in the
unsorted array and determine its final location in the sorted array This occurs when all values to the left of the element in the array are less than (or equal to) the element and all values to the right of the element are greater than (or equal to) the element We now have 1 element in its proper location and two unsorted subarrays
2) Recursive Step Perform step 1 on each unsorted subarray
Quick Sort Each time step 1 is performed on a
subarray another element is placed in its final location of the sorted array and two unsorted subarrays are created
When a subarray consists of one element that subarray is sorted
Therefore that element is in its final location
Quick Sort There are several partitioning strategies
used in practice (ie several ldquoversionsrdquo of quick sort) but the one we are about to describe is known to work well
For simplicity we will choose the last element to be the pivot element
We could also chose a different pivot element and swap it with the last element in the array
Quick Sort Below is the array we would like to
sort1 4 8 9 0 11 5 10 7 6
Quick Sort The index left starts at the first element
and right starts at the next-to-last element
We want to move all the elements smaller than the pivot to the left part of the array and all the elements larger than the pivot to the right part
1 4 8 9 0 11 5 10 7 6left righ
t
Quick Sort We move left to the right skipping
over elements that are smaller than the pivot
1 4 8 9 0 11 5 10 7 6left righ
t
Quick Sort We then move right to the left skipping
over elements that are greater than the pivot
When left and right have stopped left is on an element greater than (or equal to) the pivot and right is on an element smaller than (or equal to) the pivot
1 4 8 9 0 11 5 10 7 6left right
Quick Sort If left is to the left of right (or if left
= right) those elements are swapped
1 4 8 9 0 11 5 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
Quick Sort The effect is to push a large
element to the right and a small element to the left
We then repeat the process until left and right cross
Quick Sort
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6left righ
t
Quick Sort
1 4 5 0 9 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6righ
tleft
Quick Sort
At this point left and right have crossed so no swap is performed
The final part of the partitioning is to swap the pivot element with left
1 4 5 0 9 11 8 10 7 6righ
tleft
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort Note that all elements to the left of
the pivot are less than (or equal to) the pivot and all elements to the right of the pivot are greater than (or equal to) the pivot
Hence the pivot element has been placed in its final sorted position
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort We now repeat the process using
the sub-arrays to the left and right of the pivot
1 4 5 0 6 11 8 10 7 9
1 4 5 0 11 8 10 7 9
Analysis of quicksortmdashbest case
Suppose each partition operation divides the array almost exactly in half
Then the depth of the recursion in log2n Because thatrsquos how many times we can
halve n However there are many recursions at
each level How can we figure this out We note that
Each partition is linear over its subarray All the partitions at one level cover the array
Partitioning at various levels
Best case II We cut the array size in half each time So the depth of the recursion in log2n At each level of the recursion all the
partitions at that level do work that is linear in n
O(log2n) O(n) = O(n log2n)
What about the worst case
Worst case In the worst case partitioning always
divides the size n array into these three parts A length one part containing the pivot itself A length zero part and A length n-1 part containing everything else
We donrsquot recur on the zero-length part Recurring on the length n-1 part requires
(in the worst case) recurring to depth n-1
Worst case partitioning
Worst case for quicksort In the worst case recursion may be
O(n) levels deep (for an array of size n)
But the partitioning work done at each level is still O(n)
O(n) O(n) = O(n2) So worst case for Quicksort is O(n2) When does this happen
When the array is sorted to begin with
Typical case for quicksort If the array is sorted to begin with
Quicksort is terrible O(n2) It is possible to construct other bad
cases However Quicksort is usually O(n log2n) The constants are so good that Quicksort
is generally the fastest algorithm known Most real-world sorting is done by
Quicksort
Tweaking Quicksort Almost anything you can try to
ldquoimproverdquo Quicksort will actually slow it down
One good tweak is to switch to a different sorting method when the subarrays get small (say 10 or 12) Quicksort has too much overhead for small
array sizes For large arrays it might be a good
idea to check beforehand if the array is already sorted But there is a better tweak than this
Randomized Quick-Sort Select the pivot as a random element of the sequence The expected running time of randomized quick-sort on a
sequence of size n is O(n log n) The time spent at a level of the quick-sort tree is O(n) We show that the expected height of the quick-sort tree is
O(log n) good vs bad pivots
bull The probability of a good pivot is 12 thus we expect k2 good pivots out of k pivotsbull After a good pivot the size of each child sequence is at most 34 the size of the parent sequencebull After h pivots we expect (34)h2 n elementsbull The expected height h of the quick-sort tree is at most 2 log43n
Median of three Obviously it doesnrsquot make sense to sort the
array in order to find the median to use as a pivot
Instead compare just three elements of our (sub)arraymdashthe first the last and the middle Take the median (middle value) of these three as
pivot Itrsquos possible (but not easy) to construct cases
which will make this technique O(n2) Suppose we rearrange (sort) these three
numbers so that the smallest is in the first position the largest in the last position and the other in the middle This lets us simplify and speed up the partition
loop
Final comments Weissrsquos code shows some additional
optimizations on pp 246-247 Weiss chooses to stop both searches on
equality to pivot This design decision is debatable
Quicksort is the fastest known general sorting algorithm on average
For optimum speed the pivot must be chosen carefully
ldquoMedian of threerdquo is a good technique for choosing the pivot
There will be some cases where Quicksort runs in O(n2) time
Quick Sort A couple of notes about quick sort There are more optimal ways to
choose the pivot value (such as the median-of-three method)
Also when the subarrays get small it becomes more efficient to use the insertion sort as opposed to continued use of quick sort
Bubble Sort vs Quick Sort If we calculate the Big-O notation
we find that (in the average case) Bubble Sort O(n2) Quick Sort O(nlog2n)
(quick-sort runs in time O(n2) in the worst case)
Heap Sort Idea take the items that need to be
sorted and insert them into the heap By calling deleteHeap we remove the
smallest or largest element depending on whether or not we are working with a min- or max-heap respectively
Hence the elements are removed in ascending or descending order
Efficiency O(nlog2n)
Merge Sort Idea
Take the array you would like to sort and divide it in half to create 2 unsorted subarrays
Next sort each of the 2 subarrays Finally merge the 2 sorted subarrays
into 1 sorted array Efficiency O(nlog2n)
Merge Sort
Merge Sort Although the merge step produces
a sorted array we have overlooked a very important step
How did we sort the 2 halves before performing the merge step
We used merge sort
Merge Sort By continually calling the merge
sort algorithm we eventually get a subarray of size 1
Since an array with only 1 element is clearly sorted we can back out and merge 2 arrays of size 1
Merge Sort
Merge Sort The basic merging algorithm
consists of 2 input arrays (arrayA and arrayB) An ouput array (arrayC) 3 position holders (indexA indexB
indexC) which are initially set to the beginning of their respective arrays
Merge Sort The smaller of arrayA[indexA] and
arrayB[indexB] is copied into arrayC[indexC] and the appropriate position holders are advanced
When either input list is exhausted the remainder of the other list is copied into arrayC
Merge Sort
1 13 24 26index
A
indexC
arrayA
2 15 27 38index
B
arrayB
arrayC
We compare arrayA[indexA]with arrayB[indexB] Whichever value is smaller isplaced into arrayC[indexC]
1 lt 2 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1index
C
arrayA
2 15 27 38index
B
arrayB
arrayC
2 lt 13 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
13 lt 15 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
15 lt 24 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13 15indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
24 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
indexA
1 2 13 15 24indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
26 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
1 2 13 15 24 26indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
Since we have exhaustedone of the arrays arrayAwe simply copy the remaining items from theother array arrayB intoarrayC
Merge Sort
1 13 24
26
1 2 13 15 24 26 27 38
arrayA
2 15 27 38arrayB
arrayC
Efficiency Summary
Sort Worst Case Average Case
Insertion O(n2) O(n2)
Shell O(n15) O(n125)
Selection O(n2) O(n2)
Bubble O(n2) O(n2)
Quick O(n2) O(nlog2n)
Heap O(nlog2n) O(nlog2n)
Merge O(nlog2n) O(nlog2n)
More Sorting
radix sort bucket sort in-place sorting how fast can we sort
Radix Sort
Unlike other sorting methods radix sort considers the structure of the keys
Assume keys are represented in a base M number system (M = radix) ie if M = 2 the keys are represented in binary
Sorting is done by comparing bits in the same position
Extension to keys that are alphanumeric strings
Radix Exchange Sort
Examine bits from left to right1 Sort array with respect to leftmost bit
2 Partition array
3Recursionrecursively sort top subarray ignoring leftmost bitrecursively sort bottom subarray ignoring leftmost bit
Time to sort n b-bit numbers O(b n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Quick Sort The sorting then continues by
quick sorting the left partition followed by quick sorting the right partition
The basic algorithm is as follows
Quick Sort1) Partitioning Step Take an element in the
unsorted array and determine its final location in the sorted array This occurs when all values to the left of the element in the array are less than (or equal to) the element and all values to the right of the element are greater than (or equal to) the element We now have 1 element in its proper location and two unsorted subarrays
2) Recursive Step Perform step 1 on each unsorted subarray
Quick Sort Each time step 1 is performed on a
subarray another element is placed in its final location of the sorted array and two unsorted subarrays are created
When a subarray consists of one element that subarray is sorted
Therefore that element is in its final location
Quick Sort There are several partitioning strategies
used in practice (ie several ldquoversionsrdquo of quick sort) but the one we are about to describe is known to work well
For simplicity we will choose the last element to be the pivot element
We could also chose a different pivot element and swap it with the last element in the array
Quick Sort Below is the array we would like to
sort1 4 8 9 0 11 5 10 7 6
Quick Sort The index left starts at the first element
and right starts at the next-to-last element
We want to move all the elements smaller than the pivot to the left part of the array and all the elements larger than the pivot to the right part
1 4 8 9 0 11 5 10 7 6left righ
t
Quick Sort We move left to the right skipping
over elements that are smaller than the pivot
1 4 8 9 0 11 5 10 7 6left righ
t
Quick Sort We then move right to the left skipping
over elements that are greater than the pivot
When left and right have stopped left is on an element greater than (or equal to) the pivot and right is on an element smaller than (or equal to) the pivot
1 4 8 9 0 11 5 10 7 6left right
Quick Sort If left is to the left of right (or if left
= right) those elements are swapped
1 4 8 9 0 11 5 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
Quick Sort The effect is to push a large
element to the right and a small element to the left
We then repeat the process until left and right cross
Quick Sort
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6left righ
t
Quick Sort
1 4 5 0 9 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6righ
tleft
Quick Sort
At this point left and right have crossed so no swap is performed
The final part of the partitioning is to swap the pivot element with left
1 4 5 0 9 11 8 10 7 6righ
tleft
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort Note that all elements to the left of
the pivot are less than (or equal to) the pivot and all elements to the right of the pivot are greater than (or equal to) the pivot
Hence the pivot element has been placed in its final sorted position
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort We now repeat the process using
the sub-arrays to the left and right of the pivot
1 4 5 0 6 11 8 10 7 9
1 4 5 0 11 8 10 7 9
Analysis of quicksortmdashbest case
Suppose each partition operation divides the array almost exactly in half
Then the depth of the recursion in log2n Because thatrsquos how many times we can
halve n However there are many recursions at
each level How can we figure this out We note that
Each partition is linear over its subarray All the partitions at one level cover the array
Partitioning at various levels
Best case II We cut the array size in half each time So the depth of the recursion in log2n At each level of the recursion all the
partitions at that level do work that is linear in n
O(log2n) O(n) = O(n log2n)
What about the worst case
Worst case In the worst case partitioning always
divides the size n array into these three parts A length one part containing the pivot itself A length zero part and A length n-1 part containing everything else
We donrsquot recur on the zero-length part Recurring on the length n-1 part requires
(in the worst case) recurring to depth n-1
Worst case partitioning
Worst case for quicksort In the worst case recursion may be
O(n) levels deep (for an array of size n)
But the partitioning work done at each level is still O(n)
O(n) O(n) = O(n2) So worst case for Quicksort is O(n2) When does this happen
When the array is sorted to begin with
Typical case for quicksort If the array is sorted to begin with
Quicksort is terrible O(n2) It is possible to construct other bad
cases However Quicksort is usually O(n log2n) The constants are so good that Quicksort
is generally the fastest algorithm known Most real-world sorting is done by
Quicksort
Tweaking Quicksort Almost anything you can try to
ldquoimproverdquo Quicksort will actually slow it down
One good tweak is to switch to a different sorting method when the subarrays get small (say 10 or 12) Quicksort has too much overhead for small
array sizes For large arrays it might be a good
idea to check beforehand if the array is already sorted But there is a better tweak than this
Randomized Quick-Sort Select the pivot as a random element of the sequence The expected running time of randomized quick-sort on a
sequence of size n is O(n log n) The time spent at a level of the quick-sort tree is O(n) We show that the expected height of the quick-sort tree is
O(log n) good vs bad pivots
bull The probability of a good pivot is 12 thus we expect k2 good pivots out of k pivotsbull After a good pivot the size of each child sequence is at most 34 the size of the parent sequencebull After h pivots we expect (34)h2 n elementsbull The expected height h of the quick-sort tree is at most 2 log43n
Median of three Obviously it doesnrsquot make sense to sort the
array in order to find the median to use as a pivot
Instead compare just three elements of our (sub)arraymdashthe first the last and the middle Take the median (middle value) of these three as
pivot Itrsquos possible (but not easy) to construct cases
which will make this technique O(n2) Suppose we rearrange (sort) these three
numbers so that the smallest is in the first position the largest in the last position and the other in the middle This lets us simplify and speed up the partition
loop
Final comments Weissrsquos code shows some additional
optimizations on pp 246-247 Weiss chooses to stop both searches on
equality to pivot This design decision is debatable
Quicksort is the fastest known general sorting algorithm on average
For optimum speed the pivot must be chosen carefully
ldquoMedian of threerdquo is a good technique for choosing the pivot
There will be some cases where Quicksort runs in O(n2) time
Quick Sort A couple of notes about quick sort There are more optimal ways to
choose the pivot value (such as the median-of-three method)
Also when the subarrays get small it becomes more efficient to use the insertion sort as opposed to continued use of quick sort
Bubble Sort vs Quick Sort If we calculate the Big-O notation
we find that (in the average case) Bubble Sort O(n2) Quick Sort O(nlog2n)
(quick-sort runs in time O(n2) in the worst case)
Heap Sort Idea take the items that need to be
sorted and insert them into the heap By calling deleteHeap we remove the
smallest or largest element depending on whether or not we are working with a min- or max-heap respectively
Hence the elements are removed in ascending or descending order
Efficiency O(nlog2n)
Merge Sort Idea
Take the array you would like to sort and divide it in half to create 2 unsorted subarrays
Next sort each of the 2 subarrays Finally merge the 2 sorted subarrays
into 1 sorted array Efficiency O(nlog2n)
Merge Sort
Merge Sort Although the merge step produces
a sorted array we have overlooked a very important step
How did we sort the 2 halves before performing the merge step
We used merge sort
Merge Sort By continually calling the merge
sort algorithm we eventually get a subarray of size 1
Since an array with only 1 element is clearly sorted we can back out and merge 2 arrays of size 1
Merge Sort
Merge Sort The basic merging algorithm
consists of 2 input arrays (arrayA and arrayB) An ouput array (arrayC) 3 position holders (indexA indexB
indexC) which are initially set to the beginning of their respective arrays
Merge Sort The smaller of arrayA[indexA] and
arrayB[indexB] is copied into arrayC[indexC] and the appropriate position holders are advanced
When either input list is exhausted the remainder of the other list is copied into arrayC
Merge Sort
1 13 24 26index
A
indexC
arrayA
2 15 27 38index
B
arrayB
arrayC
We compare arrayA[indexA]with arrayB[indexB] Whichever value is smaller isplaced into arrayC[indexC]
1 lt 2 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1index
C
arrayA
2 15 27 38index
B
arrayB
arrayC
2 lt 13 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
13 lt 15 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
15 lt 24 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13 15indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
24 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
indexA
1 2 13 15 24indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
26 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
1 2 13 15 24 26indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
Since we have exhaustedone of the arrays arrayAwe simply copy the remaining items from theother array arrayB intoarrayC
Merge Sort
1 13 24
26
1 2 13 15 24 26 27 38
arrayA
2 15 27 38arrayB
arrayC
Efficiency Summary
Sort Worst Case Average Case
Insertion O(n2) O(n2)
Shell O(n15) O(n125)
Selection O(n2) O(n2)
Bubble O(n2) O(n2)
Quick O(n2) O(nlog2n)
Heap O(nlog2n) O(nlog2n)
Merge O(nlog2n) O(nlog2n)
More Sorting
radix sort bucket sort in-place sorting how fast can we sort
Radix Sort
Unlike other sorting methods radix sort considers the structure of the keys
Assume keys are represented in a base M number system (M = radix) ie if M = 2 the keys are represented in binary
Sorting is done by comparing bits in the same position
Extension to keys that are alphanumeric strings
Radix Exchange Sort
Examine bits from left to right1 Sort array with respect to leftmost bit
2 Partition array
3Recursionrecursively sort top subarray ignoring leftmost bitrecursively sort bottom subarray ignoring leftmost bit
Time to sort n b-bit numbers O(b n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Quick Sort1) Partitioning Step Take an element in the
unsorted array and determine its final location in the sorted array This occurs when all values to the left of the element in the array are less than (or equal to) the element and all values to the right of the element are greater than (or equal to) the element We now have 1 element in its proper location and two unsorted subarrays
2) Recursive Step Perform step 1 on each unsorted subarray
Quick Sort Each time step 1 is performed on a
subarray another element is placed in its final location of the sorted array and two unsorted subarrays are created
When a subarray consists of one element that subarray is sorted
Therefore that element is in its final location
Quick Sort There are several partitioning strategies
used in practice (ie several ldquoversionsrdquo of quick sort) but the one we are about to describe is known to work well
For simplicity we will choose the last element to be the pivot element
We could also chose a different pivot element and swap it with the last element in the array
Quick Sort Below is the array we would like to
sort1 4 8 9 0 11 5 10 7 6
Quick Sort The index left starts at the first element
and right starts at the next-to-last element
We want to move all the elements smaller than the pivot to the left part of the array and all the elements larger than the pivot to the right part
1 4 8 9 0 11 5 10 7 6left righ
t
Quick Sort We move left to the right skipping
over elements that are smaller than the pivot
1 4 8 9 0 11 5 10 7 6left righ
t
Quick Sort We then move right to the left skipping
over elements that are greater than the pivot
When left and right have stopped left is on an element greater than (or equal to) the pivot and right is on an element smaller than (or equal to) the pivot
1 4 8 9 0 11 5 10 7 6left right
Quick Sort If left is to the left of right (or if left
= right) those elements are swapped
1 4 8 9 0 11 5 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
Quick Sort The effect is to push a large
element to the right and a small element to the left
We then repeat the process until left and right cross
Quick Sort
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6left righ
t
Quick Sort
1 4 5 0 9 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6righ
tleft
Quick Sort
At this point left and right have crossed so no swap is performed
The final part of the partitioning is to swap the pivot element with left
1 4 5 0 9 11 8 10 7 6righ
tleft
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort Note that all elements to the left of
the pivot are less than (or equal to) the pivot and all elements to the right of the pivot are greater than (or equal to) the pivot
Hence the pivot element has been placed in its final sorted position
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort We now repeat the process using
the sub-arrays to the left and right of the pivot
1 4 5 0 6 11 8 10 7 9
1 4 5 0 11 8 10 7 9
Analysis of quicksortmdashbest case
Suppose each partition operation divides the array almost exactly in half
Then the depth of the recursion in log2n Because thatrsquos how many times we can
halve n However there are many recursions at
each level How can we figure this out We note that
Each partition is linear over its subarray All the partitions at one level cover the array
Partitioning at various levels
Best case II We cut the array size in half each time So the depth of the recursion in log2n At each level of the recursion all the
partitions at that level do work that is linear in n
O(log2n) O(n) = O(n log2n)
What about the worst case
Worst case In the worst case partitioning always
divides the size n array into these three parts A length one part containing the pivot itself A length zero part and A length n-1 part containing everything else
We donrsquot recur on the zero-length part Recurring on the length n-1 part requires
(in the worst case) recurring to depth n-1
Worst case partitioning
Worst case for quicksort In the worst case recursion may be
O(n) levels deep (for an array of size n)
But the partitioning work done at each level is still O(n)
O(n) O(n) = O(n2) So worst case for Quicksort is O(n2) When does this happen
When the array is sorted to begin with
Typical case for quicksort If the array is sorted to begin with
Quicksort is terrible O(n2) It is possible to construct other bad
cases However Quicksort is usually O(n log2n) The constants are so good that Quicksort
is generally the fastest algorithm known Most real-world sorting is done by
Quicksort
Tweaking Quicksort Almost anything you can try to
ldquoimproverdquo Quicksort will actually slow it down
One good tweak is to switch to a different sorting method when the subarrays get small (say 10 or 12) Quicksort has too much overhead for small
array sizes For large arrays it might be a good
idea to check beforehand if the array is already sorted But there is a better tweak than this
Randomized Quick-Sort Select the pivot as a random element of the sequence The expected running time of randomized quick-sort on a
sequence of size n is O(n log n) The time spent at a level of the quick-sort tree is O(n) We show that the expected height of the quick-sort tree is
O(log n) good vs bad pivots
bull The probability of a good pivot is 12 thus we expect k2 good pivots out of k pivotsbull After a good pivot the size of each child sequence is at most 34 the size of the parent sequencebull After h pivots we expect (34)h2 n elementsbull The expected height h of the quick-sort tree is at most 2 log43n
Median of three Obviously it doesnrsquot make sense to sort the
array in order to find the median to use as a pivot
Instead compare just three elements of our (sub)arraymdashthe first the last and the middle Take the median (middle value) of these three as
pivot Itrsquos possible (but not easy) to construct cases
which will make this technique O(n2) Suppose we rearrange (sort) these three
numbers so that the smallest is in the first position the largest in the last position and the other in the middle This lets us simplify and speed up the partition
loop
Final comments Weissrsquos code shows some additional
optimizations on pp 246-247 Weiss chooses to stop both searches on
equality to pivot This design decision is debatable
Quicksort is the fastest known general sorting algorithm on average
For optimum speed the pivot must be chosen carefully
ldquoMedian of threerdquo is a good technique for choosing the pivot
There will be some cases where Quicksort runs in O(n2) time
Quick Sort A couple of notes about quick sort There are more optimal ways to
choose the pivot value (such as the median-of-three method)
Also when the subarrays get small it becomes more efficient to use the insertion sort as opposed to continued use of quick sort
Bubble Sort vs Quick Sort If we calculate the Big-O notation
we find that (in the average case) Bubble Sort O(n2) Quick Sort O(nlog2n)
(quick-sort runs in time O(n2) in the worst case)
Heap Sort Idea take the items that need to be
sorted and insert them into the heap By calling deleteHeap we remove the
smallest or largest element depending on whether or not we are working with a min- or max-heap respectively
Hence the elements are removed in ascending or descending order
Efficiency O(nlog2n)
Merge Sort Idea
Take the array you would like to sort and divide it in half to create 2 unsorted subarrays
Next sort each of the 2 subarrays Finally merge the 2 sorted subarrays
into 1 sorted array Efficiency O(nlog2n)
Merge Sort
Merge Sort Although the merge step produces
a sorted array we have overlooked a very important step
How did we sort the 2 halves before performing the merge step
We used merge sort
Merge Sort By continually calling the merge
sort algorithm we eventually get a subarray of size 1
Since an array with only 1 element is clearly sorted we can back out and merge 2 arrays of size 1
Merge Sort
Merge Sort The basic merging algorithm
consists of 2 input arrays (arrayA and arrayB) An ouput array (arrayC) 3 position holders (indexA indexB
indexC) which are initially set to the beginning of their respective arrays
Merge Sort The smaller of arrayA[indexA] and
arrayB[indexB] is copied into arrayC[indexC] and the appropriate position holders are advanced
When either input list is exhausted the remainder of the other list is copied into arrayC
Merge Sort
1 13 24 26index
A
indexC
arrayA
2 15 27 38index
B
arrayB
arrayC
We compare arrayA[indexA]with arrayB[indexB] Whichever value is smaller isplaced into arrayC[indexC]
1 lt 2 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1index
C
arrayA
2 15 27 38index
B
arrayB
arrayC
2 lt 13 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
13 lt 15 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
15 lt 24 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13 15indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
24 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
indexA
1 2 13 15 24indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
26 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
1 2 13 15 24 26indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
Since we have exhaustedone of the arrays arrayAwe simply copy the remaining items from theother array arrayB intoarrayC
Merge Sort
1 13 24
26
1 2 13 15 24 26 27 38
arrayA
2 15 27 38arrayB
arrayC
Efficiency Summary
Sort Worst Case Average Case
Insertion O(n2) O(n2)
Shell O(n15) O(n125)
Selection O(n2) O(n2)
Bubble O(n2) O(n2)
Quick O(n2) O(nlog2n)
Heap O(nlog2n) O(nlog2n)
Merge O(nlog2n) O(nlog2n)
More Sorting
radix sort bucket sort in-place sorting how fast can we sort
Radix Sort
Unlike other sorting methods radix sort considers the structure of the keys
Assume keys are represented in a base M number system (M = radix) ie if M = 2 the keys are represented in binary
Sorting is done by comparing bits in the same position
Extension to keys that are alphanumeric strings
Radix Exchange Sort
Examine bits from left to right1 Sort array with respect to leftmost bit
2 Partition array
3Recursionrecursively sort top subarray ignoring leftmost bitrecursively sort bottom subarray ignoring leftmost bit
Time to sort n b-bit numbers O(b n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Quick Sort Each time step 1 is performed on a
subarray another element is placed in its final location of the sorted array and two unsorted subarrays are created
When a subarray consists of one element that subarray is sorted
Therefore that element is in its final location
Quick Sort There are several partitioning strategies
used in practice (ie several ldquoversionsrdquo of quick sort) but the one we are about to describe is known to work well
For simplicity we will choose the last element to be the pivot element
We could also chose a different pivot element and swap it with the last element in the array
Quick Sort Below is the array we would like to
sort1 4 8 9 0 11 5 10 7 6
Quick Sort The index left starts at the first element
and right starts at the next-to-last element
We want to move all the elements smaller than the pivot to the left part of the array and all the elements larger than the pivot to the right part
1 4 8 9 0 11 5 10 7 6left righ
t
Quick Sort We move left to the right skipping
over elements that are smaller than the pivot
1 4 8 9 0 11 5 10 7 6left righ
t
Quick Sort We then move right to the left skipping
over elements that are greater than the pivot
When left and right have stopped left is on an element greater than (or equal to) the pivot and right is on an element smaller than (or equal to) the pivot
1 4 8 9 0 11 5 10 7 6left right
Quick Sort If left is to the left of right (or if left
= right) those elements are swapped
1 4 8 9 0 11 5 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
Quick Sort The effect is to push a large
element to the right and a small element to the left
We then repeat the process until left and right cross
Quick Sort
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6left righ
t
Quick Sort
1 4 5 0 9 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6righ
tleft
Quick Sort
At this point left and right have crossed so no swap is performed
The final part of the partitioning is to swap the pivot element with left
1 4 5 0 9 11 8 10 7 6righ
tleft
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort Note that all elements to the left of
the pivot are less than (or equal to) the pivot and all elements to the right of the pivot are greater than (or equal to) the pivot
Hence the pivot element has been placed in its final sorted position
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort We now repeat the process using
the sub-arrays to the left and right of the pivot
1 4 5 0 6 11 8 10 7 9
1 4 5 0 11 8 10 7 9
Analysis of quicksortmdashbest case
Suppose each partition operation divides the array almost exactly in half
Then the depth of the recursion in log2n Because thatrsquos how many times we can
halve n However there are many recursions at
each level How can we figure this out We note that
Each partition is linear over its subarray All the partitions at one level cover the array
Partitioning at various levels
Best case II We cut the array size in half each time So the depth of the recursion in log2n At each level of the recursion all the
partitions at that level do work that is linear in n
O(log2n) O(n) = O(n log2n)
What about the worst case
Worst case In the worst case partitioning always
divides the size n array into these three parts A length one part containing the pivot itself A length zero part and A length n-1 part containing everything else
We donrsquot recur on the zero-length part Recurring on the length n-1 part requires
(in the worst case) recurring to depth n-1
Worst case partitioning
Worst case for quicksort In the worst case recursion may be
O(n) levels deep (for an array of size n)
But the partitioning work done at each level is still O(n)
O(n) O(n) = O(n2) So worst case for Quicksort is O(n2) When does this happen
When the array is sorted to begin with
Typical case for quicksort If the array is sorted to begin with
Quicksort is terrible O(n2) It is possible to construct other bad
cases However Quicksort is usually O(n log2n) The constants are so good that Quicksort
is generally the fastest algorithm known Most real-world sorting is done by
Quicksort
Tweaking Quicksort Almost anything you can try to
ldquoimproverdquo Quicksort will actually slow it down
One good tweak is to switch to a different sorting method when the subarrays get small (say 10 or 12) Quicksort has too much overhead for small
array sizes For large arrays it might be a good
idea to check beforehand if the array is already sorted But there is a better tweak than this
Randomized Quick-Sort Select the pivot as a random element of the sequence The expected running time of randomized quick-sort on a
sequence of size n is O(n log n) The time spent at a level of the quick-sort tree is O(n) We show that the expected height of the quick-sort tree is
O(log n) good vs bad pivots
bull The probability of a good pivot is 12 thus we expect k2 good pivots out of k pivotsbull After a good pivot the size of each child sequence is at most 34 the size of the parent sequencebull After h pivots we expect (34)h2 n elementsbull The expected height h of the quick-sort tree is at most 2 log43n
Median of three Obviously it doesnrsquot make sense to sort the
array in order to find the median to use as a pivot
Instead compare just three elements of our (sub)arraymdashthe first the last and the middle Take the median (middle value) of these three as
pivot Itrsquos possible (but not easy) to construct cases
which will make this technique O(n2) Suppose we rearrange (sort) these three
numbers so that the smallest is in the first position the largest in the last position and the other in the middle This lets us simplify and speed up the partition
loop
Final comments Weissrsquos code shows some additional
optimizations on pp 246-247 Weiss chooses to stop both searches on
equality to pivot This design decision is debatable
Quicksort is the fastest known general sorting algorithm on average
For optimum speed the pivot must be chosen carefully
ldquoMedian of threerdquo is a good technique for choosing the pivot
There will be some cases where Quicksort runs in O(n2) time
Quick Sort A couple of notes about quick sort There are more optimal ways to
choose the pivot value (such as the median-of-three method)
Also when the subarrays get small it becomes more efficient to use the insertion sort as opposed to continued use of quick sort
Bubble Sort vs Quick Sort If we calculate the Big-O notation
we find that (in the average case) Bubble Sort O(n2) Quick Sort O(nlog2n)
(quick-sort runs in time O(n2) in the worst case)
Heap Sort Idea take the items that need to be
sorted and insert them into the heap By calling deleteHeap we remove the
smallest or largest element depending on whether or not we are working with a min- or max-heap respectively
Hence the elements are removed in ascending or descending order
Efficiency O(nlog2n)
Merge Sort Idea
Take the array you would like to sort and divide it in half to create 2 unsorted subarrays
Next sort each of the 2 subarrays Finally merge the 2 sorted subarrays
into 1 sorted array Efficiency O(nlog2n)
Merge Sort
Merge Sort Although the merge step produces
a sorted array we have overlooked a very important step
How did we sort the 2 halves before performing the merge step
We used merge sort
Merge Sort By continually calling the merge
sort algorithm we eventually get a subarray of size 1
Since an array with only 1 element is clearly sorted we can back out and merge 2 arrays of size 1
Merge Sort
Merge Sort The basic merging algorithm
consists of 2 input arrays (arrayA and arrayB) An ouput array (arrayC) 3 position holders (indexA indexB
indexC) which are initially set to the beginning of their respective arrays
Merge Sort The smaller of arrayA[indexA] and
arrayB[indexB] is copied into arrayC[indexC] and the appropriate position holders are advanced
When either input list is exhausted the remainder of the other list is copied into arrayC
Merge Sort
1 13 24 26index
A
indexC
arrayA
2 15 27 38index
B
arrayB
arrayC
We compare arrayA[indexA]with arrayB[indexB] Whichever value is smaller isplaced into arrayC[indexC]
1 lt 2 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1index
C
arrayA
2 15 27 38index
B
arrayB
arrayC
2 lt 13 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
13 lt 15 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
15 lt 24 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13 15indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
24 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
indexA
1 2 13 15 24indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
26 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
1 2 13 15 24 26indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
Since we have exhaustedone of the arrays arrayAwe simply copy the remaining items from theother array arrayB intoarrayC
Merge Sort
1 13 24
26
1 2 13 15 24 26 27 38
arrayA
2 15 27 38arrayB
arrayC
Efficiency Summary
Sort Worst Case Average Case
Insertion O(n2) O(n2)
Shell O(n15) O(n125)
Selection O(n2) O(n2)
Bubble O(n2) O(n2)
Quick O(n2) O(nlog2n)
Heap O(nlog2n) O(nlog2n)
Merge O(nlog2n) O(nlog2n)
More Sorting
radix sort bucket sort in-place sorting how fast can we sort
Radix Sort
Unlike other sorting methods radix sort considers the structure of the keys
Assume keys are represented in a base M number system (M = radix) ie if M = 2 the keys are represented in binary
Sorting is done by comparing bits in the same position
Extension to keys that are alphanumeric strings
Radix Exchange Sort
Examine bits from left to right1 Sort array with respect to leftmost bit
2 Partition array
3Recursionrecursively sort top subarray ignoring leftmost bitrecursively sort bottom subarray ignoring leftmost bit
Time to sort n b-bit numbers O(b n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Quick Sort There are several partitioning strategies
used in practice (ie several ldquoversionsrdquo of quick sort) but the one we are about to describe is known to work well
For simplicity we will choose the last element to be the pivot element
We could also chose a different pivot element and swap it with the last element in the array
Quick Sort Below is the array we would like to
sort1 4 8 9 0 11 5 10 7 6
Quick Sort The index left starts at the first element
and right starts at the next-to-last element
We want to move all the elements smaller than the pivot to the left part of the array and all the elements larger than the pivot to the right part
1 4 8 9 0 11 5 10 7 6left righ
t
Quick Sort We move left to the right skipping
over elements that are smaller than the pivot
1 4 8 9 0 11 5 10 7 6left righ
t
Quick Sort We then move right to the left skipping
over elements that are greater than the pivot
When left and right have stopped left is on an element greater than (or equal to) the pivot and right is on an element smaller than (or equal to) the pivot
1 4 8 9 0 11 5 10 7 6left right
Quick Sort If left is to the left of right (or if left
= right) those elements are swapped
1 4 8 9 0 11 5 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
Quick Sort The effect is to push a large
element to the right and a small element to the left
We then repeat the process until left and right cross
Quick Sort
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6left righ
t
Quick Sort
1 4 5 0 9 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6righ
tleft
Quick Sort
At this point left and right have crossed so no swap is performed
The final part of the partitioning is to swap the pivot element with left
1 4 5 0 9 11 8 10 7 6righ
tleft
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort Note that all elements to the left of
the pivot are less than (or equal to) the pivot and all elements to the right of the pivot are greater than (or equal to) the pivot
Hence the pivot element has been placed in its final sorted position
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort We now repeat the process using
the sub-arrays to the left and right of the pivot
1 4 5 0 6 11 8 10 7 9
1 4 5 0 11 8 10 7 9
Analysis of quicksortmdashbest case
Suppose each partition operation divides the array almost exactly in half
Then the depth of the recursion in log2n Because thatrsquos how many times we can
halve n However there are many recursions at
each level How can we figure this out We note that
Each partition is linear over its subarray All the partitions at one level cover the array
Partitioning at various levels
Best case II We cut the array size in half each time So the depth of the recursion in log2n At each level of the recursion all the
partitions at that level do work that is linear in n
O(log2n) O(n) = O(n log2n)
What about the worst case
Worst case In the worst case partitioning always
divides the size n array into these three parts A length one part containing the pivot itself A length zero part and A length n-1 part containing everything else
We donrsquot recur on the zero-length part Recurring on the length n-1 part requires
(in the worst case) recurring to depth n-1
Worst case partitioning
Worst case for quicksort In the worst case recursion may be
O(n) levels deep (for an array of size n)
But the partitioning work done at each level is still O(n)
O(n) O(n) = O(n2) So worst case for Quicksort is O(n2) When does this happen
When the array is sorted to begin with
Typical case for quicksort If the array is sorted to begin with
Quicksort is terrible O(n2) It is possible to construct other bad
cases However Quicksort is usually O(n log2n) The constants are so good that Quicksort
is generally the fastest algorithm known Most real-world sorting is done by
Quicksort
Tweaking Quicksort Almost anything you can try to
ldquoimproverdquo Quicksort will actually slow it down
One good tweak is to switch to a different sorting method when the subarrays get small (say 10 or 12) Quicksort has too much overhead for small
array sizes For large arrays it might be a good
idea to check beforehand if the array is already sorted But there is a better tweak than this
Randomized Quick-Sort Select the pivot as a random element of the sequence The expected running time of randomized quick-sort on a
sequence of size n is O(n log n) The time spent at a level of the quick-sort tree is O(n) We show that the expected height of the quick-sort tree is
O(log n) good vs bad pivots
bull The probability of a good pivot is 12 thus we expect k2 good pivots out of k pivotsbull After a good pivot the size of each child sequence is at most 34 the size of the parent sequencebull After h pivots we expect (34)h2 n elementsbull The expected height h of the quick-sort tree is at most 2 log43n
Median of three Obviously it doesnrsquot make sense to sort the
array in order to find the median to use as a pivot
Instead compare just three elements of our (sub)arraymdashthe first the last and the middle Take the median (middle value) of these three as
pivot Itrsquos possible (but not easy) to construct cases
which will make this technique O(n2) Suppose we rearrange (sort) these three
numbers so that the smallest is in the first position the largest in the last position and the other in the middle This lets us simplify and speed up the partition
loop
Final comments Weissrsquos code shows some additional
optimizations on pp 246-247 Weiss chooses to stop both searches on
equality to pivot This design decision is debatable
Quicksort is the fastest known general sorting algorithm on average
For optimum speed the pivot must be chosen carefully
ldquoMedian of threerdquo is a good technique for choosing the pivot
There will be some cases where Quicksort runs in O(n2) time
Quick Sort A couple of notes about quick sort There are more optimal ways to
choose the pivot value (such as the median-of-three method)
Also when the subarrays get small it becomes more efficient to use the insertion sort as opposed to continued use of quick sort
Bubble Sort vs Quick Sort If we calculate the Big-O notation
we find that (in the average case) Bubble Sort O(n2) Quick Sort O(nlog2n)
(quick-sort runs in time O(n2) in the worst case)
Heap Sort Idea take the items that need to be
sorted and insert them into the heap By calling deleteHeap we remove the
smallest or largest element depending on whether or not we are working with a min- or max-heap respectively
Hence the elements are removed in ascending or descending order
Efficiency O(nlog2n)
Merge Sort Idea
Take the array you would like to sort and divide it in half to create 2 unsorted subarrays
Next sort each of the 2 subarrays Finally merge the 2 sorted subarrays
into 1 sorted array Efficiency O(nlog2n)
Merge Sort
Merge Sort Although the merge step produces
a sorted array we have overlooked a very important step
How did we sort the 2 halves before performing the merge step
We used merge sort
Merge Sort By continually calling the merge
sort algorithm we eventually get a subarray of size 1
Since an array with only 1 element is clearly sorted we can back out and merge 2 arrays of size 1
Merge Sort
Merge Sort The basic merging algorithm
consists of 2 input arrays (arrayA and arrayB) An ouput array (arrayC) 3 position holders (indexA indexB
indexC) which are initially set to the beginning of their respective arrays
Merge Sort The smaller of arrayA[indexA] and
arrayB[indexB] is copied into arrayC[indexC] and the appropriate position holders are advanced
When either input list is exhausted the remainder of the other list is copied into arrayC
Merge Sort
1 13 24 26index
A
indexC
arrayA
2 15 27 38index
B
arrayB
arrayC
We compare arrayA[indexA]with arrayB[indexB] Whichever value is smaller isplaced into arrayC[indexC]
1 lt 2 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1index
C
arrayA
2 15 27 38index
B
arrayB
arrayC
2 lt 13 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
13 lt 15 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
15 lt 24 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13 15indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
24 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
indexA
1 2 13 15 24indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
26 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
1 2 13 15 24 26indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
Since we have exhaustedone of the arrays arrayAwe simply copy the remaining items from theother array arrayB intoarrayC
Merge Sort
1 13 24
26
1 2 13 15 24 26 27 38
arrayA
2 15 27 38arrayB
arrayC
Efficiency Summary
Sort Worst Case Average Case
Insertion O(n2) O(n2)
Shell O(n15) O(n125)
Selection O(n2) O(n2)
Bubble O(n2) O(n2)
Quick O(n2) O(nlog2n)
Heap O(nlog2n) O(nlog2n)
Merge O(nlog2n) O(nlog2n)
More Sorting
radix sort bucket sort in-place sorting how fast can we sort
Radix Sort
Unlike other sorting methods radix sort considers the structure of the keys
Assume keys are represented in a base M number system (M = radix) ie if M = 2 the keys are represented in binary
Sorting is done by comparing bits in the same position
Extension to keys that are alphanumeric strings
Radix Exchange Sort
Examine bits from left to right1 Sort array with respect to leftmost bit
2 Partition array
3Recursionrecursively sort top subarray ignoring leftmost bitrecursively sort bottom subarray ignoring leftmost bit
Time to sort n b-bit numbers O(b n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Quick Sort Below is the array we would like to
sort1 4 8 9 0 11 5 10 7 6
Quick Sort The index left starts at the first element
and right starts at the next-to-last element
We want to move all the elements smaller than the pivot to the left part of the array and all the elements larger than the pivot to the right part
1 4 8 9 0 11 5 10 7 6left righ
t
Quick Sort We move left to the right skipping
over elements that are smaller than the pivot
1 4 8 9 0 11 5 10 7 6left righ
t
Quick Sort We then move right to the left skipping
over elements that are greater than the pivot
When left and right have stopped left is on an element greater than (or equal to) the pivot and right is on an element smaller than (or equal to) the pivot
1 4 8 9 0 11 5 10 7 6left right
Quick Sort If left is to the left of right (or if left
= right) those elements are swapped
1 4 8 9 0 11 5 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
Quick Sort The effect is to push a large
element to the right and a small element to the left
We then repeat the process until left and right cross
Quick Sort
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6left righ
t
Quick Sort
1 4 5 0 9 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6righ
tleft
Quick Sort
At this point left and right have crossed so no swap is performed
The final part of the partitioning is to swap the pivot element with left
1 4 5 0 9 11 8 10 7 6righ
tleft
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort Note that all elements to the left of
the pivot are less than (or equal to) the pivot and all elements to the right of the pivot are greater than (or equal to) the pivot
Hence the pivot element has been placed in its final sorted position
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort We now repeat the process using
the sub-arrays to the left and right of the pivot
1 4 5 0 6 11 8 10 7 9
1 4 5 0 11 8 10 7 9
Analysis of quicksortmdashbest case
Suppose each partition operation divides the array almost exactly in half
Then the depth of the recursion in log2n Because thatrsquos how many times we can
halve n However there are many recursions at
each level How can we figure this out We note that
Each partition is linear over its subarray All the partitions at one level cover the array
Partitioning at various levels
Best case II We cut the array size in half each time So the depth of the recursion in log2n At each level of the recursion all the
partitions at that level do work that is linear in n
O(log2n) O(n) = O(n log2n)
What about the worst case
Worst case In the worst case partitioning always
divides the size n array into these three parts A length one part containing the pivot itself A length zero part and A length n-1 part containing everything else
We donrsquot recur on the zero-length part Recurring on the length n-1 part requires
(in the worst case) recurring to depth n-1
Worst case partitioning
Worst case for quicksort In the worst case recursion may be
O(n) levels deep (for an array of size n)
But the partitioning work done at each level is still O(n)
O(n) O(n) = O(n2) So worst case for Quicksort is O(n2) When does this happen
When the array is sorted to begin with
Typical case for quicksort If the array is sorted to begin with
Quicksort is terrible O(n2) It is possible to construct other bad
cases However Quicksort is usually O(n log2n) The constants are so good that Quicksort
is generally the fastest algorithm known Most real-world sorting is done by
Quicksort
Tweaking Quicksort Almost anything you can try to
ldquoimproverdquo Quicksort will actually slow it down
One good tweak is to switch to a different sorting method when the subarrays get small (say 10 or 12) Quicksort has too much overhead for small
array sizes For large arrays it might be a good
idea to check beforehand if the array is already sorted But there is a better tweak than this
Randomized Quick-Sort Select the pivot as a random element of the sequence The expected running time of randomized quick-sort on a
sequence of size n is O(n log n) The time spent at a level of the quick-sort tree is O(n) We show that the expected height of the quick-sort tree is
O(log n) good vs bad pivots
bull The probability of a good pivot is 12 thus we expect k2 good pivots out of k pivotsbull After a good pivot the size of each child sequence is at most 34 the size of the parent sequencebull After h pivots we expect (34)h2 n elementsbull The expected height h of the quick-sort tree is at most 2 log43n
Median of three Obviously it doesnrsquot make sense to sort the
array in order to find the median to use as a pivot
Instead compare just three elements of our (sub)arraymdashthe first the last and the middle Take the median (middle value) of these three as
pivot Itrsquos possible (but not easy) to construct cases
which will make this technique O(n2) Suppose we rearrange (sort) these three
numbers so that the smallest is in the first position the largest in the last position and the other in the middle This lets us simplify and speed up the partition
loop
Final comments Weissrsquos code shows some additional
optimizations on pp 246-247 Weiss chooses to stop both searches on
equality to pivot This design decision is debatable
Quicksort is the fastest known general sorting algorithm on average
For optimum speed the pivot must be chosen carefully
ldquoMedian of threerdquo is a good technique for choosing the pivot
There will be some cases where Quicksort runs in O(n2) time
Quick Sort A couple of notes about quick sort There are more optimal ways to
choose the pivot value (such as the median-of-three method)
Also when the subarrays get small it becomes more efficient to use the insertion sort as opposed to continued use of quick sort
Bubble Sort vs Quick Sort If we calculate the Big-O notation
we find that (in the average case) Bubble Sort O(n2) Quick Sort O(nlog2n)
(quick-sort runs in time O(n2) in the worst case)
Heap Sort Idea take the items that need to be
sorted and insert them into the heap By calling deleteHeap we remove the
smallest or largest element depending on whether or not we are working with a min- or max-heap respectively
Hence the elements are removed in ascending or descending order
Efficiency O(nlog2n)
Merge Sort Idea
Take the array you would like to sort and divide it in half to create 2 unsorted subarrays
Next sort each of the 2 subarrays Finally merge the 2 sorted subarrays
into 1 sorted array Efficiency O(nlog2n)
Merge Sort
Merge Sort Although the merge step produces
a sorted array we have overlooked a very important step
How did we sort the 2 halves before performing the merge step
We used merge sort
Merge Sort By continually calling the merge
sort algorithm we eventually get a subarray of size 1
Since an array with only 1 element is clearly sorted we can back out and merge 2 arrays of size 1
Merge Sort
Merge Sort The basic merging algorithm
consists of 2 input arrays (arrayA and arrayB) An ouput array (arrayC) 3 position holders (indexA indexB
indexC) which are initially set to the beginning of their respective arrays
Merge Sort The smaller of arrayA[indexA] and
arrayB[indexB] is copied into arrayC[indexC] and the appropriate position holders are advanced
When either input list is exhausted the remainder of the other list is copied into arrayC
Merge Sort
1 13 24 26index
A
indexC
arrayA
2 15 27 38index
B
arrayB
arrayC
We compare arrayA[indexA]with arrayB[indexB] Whichever value is smaller isplaced into arrayC[indexC]
1 lt 2 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1index
C
arrayA
2 15 27 38index
B
arrayB
arrayC
2 lt 13 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
13 lt 15 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
15 lt 24 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13 15indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
24 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
indexA
1 2 13 15 24indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
26 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
1 2 13 15 24 26indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
Since we have exhaustedone of the arrays arrayAwe simply copy the remaining items from theother array arrayB intoarrayC
Merge Sort
1 13 24
26
1 2 13 15 24 26 27 38
arrayA
2 15 27 38arrayB
arrayC
Efficiency Summary
Sort Worst Case Average Case
Insertion O(n2) O(n2)
Shell O(n15) O(n125)
Selection O(n2) O(n2)
Bubble O(n2) O(n2)
Quick O(n2) O(nlog2n)
Heap O(nlog2n) O(nlog2n)
Merge O(nlog2n) O(nlog2n)
More Sorting
radix sort bucket sort in-place sorting how fast can we sort
Radix Sort
Unlike other sorting methods radix sort considers the structure of the keys
Assume keys are represented in a base M number system (M = radix) ie if M = 2 the keys are represented in binary
Sorting is done by comparing bits in the same position
Extension to keys that are alphanumeric strings
Radix Exchange Sort
Examine bits from left to right1 Sort array with respect to leftmost bit
2 Partition array
3Recursionrecursively sort top subarray ignoring leftmost bitrecursively sort bottom subarray ignoring leftmost bit
Time to sort n b-bit numbers O(b n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Quick Sort The index left starts at the first element
and right starts at the next-to-last element
We want to move all the elements smaller than the pivot to the left part of the array and all the elements larger than the pivot to the right part
1 4 8 9 0 11 5 10 7 6left righ
t
Quick Sort We move left to the right skipping
over elements that are smaller than the pivot
1 4 8 9 0 11 5 10 7 6left righ
t
Quick Sort We then move right to the left skipping
over elements that are greater than the pivot
When left and right have stopped left is on an element greater than (or equal to) the pivot and right is on an element smaller than (or equal to) the pivot
1 4 8 9 0 11 5 10 7 6left right
Quick Sort If left is to the left of right (or if left
= right) those elements are swapped
1 4 8 9 0 11 5 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
Quick Sort The effect is to push a large
element to the right and a small element to the left
We then repeat the process until left and right cross
Quick Sort
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6left righ
t
Quick Sort
1 4 5 0 9 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6righ
tleft
Quick Sort
At this point left and right have crossed so no swap is performed
The final part of the partitioning is to swap the pivot element with left
1 4 5 0 9 11 8 10 7 6righ
tleft
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort Note that all elements to the left of
the pivot are less than (or equal to) the pivot and all elements to the right of the pivot are greater than (or equal to) the pivot
Hence the pivot element has been placed in its final sorted position
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort We now repeat the process using
the sub-arrays to the left and right of the pivot
1 4 5 0 6 11 8 10 7 9
1 4 5 0 11 8 10 7 9
Analysis of quicksortmdashbest case
Suppose each partition operation divides the array almost exactly in half
Then the depth of the recursion in log2n Because thatrsquos how many times we can
halve n However there are many recursions at
each level How can we figure this out We note that
Each partition is linear over its subarray All the partitions at one level cover the array
Partitioning at various levels
Best case II We cut the array size in half each time So the depth of the recursion in log2n At each level of the recursion all the
partitions at that level do work that is linear in n
O(log2n) O(n) = O(n log2n)
What about the worst case
Worst case In the worst case partitioning always
divides the size n array into these three parts A length one part containing the pivot itself A length zero part and A length n-1 part containing everything else
We donrsquot recur on the zero-length part Recurring on the length n-1 part requires
(in the worst case) recurring to depth n-1
Worst case partitioning
Worst case for quicksort In the worst case recursion may be
O(n) levels deep (for an array of size n)
But the partitioning work done at each level is still O(n)
O(n) O(n) = O(n2) So worst case for Quicksort is O(n2) When does this happen
When the array is sorted to begin with
Typical case for quicksort If the array is sorted to begin with
Quicksort is terrible O(n2) It is possible to construct other bad
cases However Quicksort is usually O(n log2n) The constants are so good that Quicksort
is generally the fastest algorithm known Most real-world sorting is done by
Quicksort
Tweaking Quicksort Almost anything you can try to
ldquoimproverdquo Quicksort will actually slow it down
One good tweak is to switch to a different sorting method when the subarrays get small (say 10 or 12) Quicksort has too much overhead for small
array sizes For large arrays it might be a good
idea to check beforehand if the array is already sorted But there is a better tweak than this
Randomized Quick-Sort Select the pivot as a random element of the sequence The expected running time of randomized quick-sort on a
sequence of size n is O(n log n) The time spent at a level of the quick-sort tree is O(n) We show that the expected height of the quick-sort tree is
O(log n) good vs bad pivots
bull The probability of a good pivot is 12 thus we expect k2 good pivots out of k pivotsbull After a good pivot the size of each child sequence is at most 34 the size of the parent sequencebull After h pivots we expect (34)h2 n elementsbull The expected height h of the quick-sort tree is at most 2 log43n
Median of three Obviously it doesnrsquot make sense to sort the
array in order to find the median to use as a pivot
Instead compare just three elements of our (sub)arraymdashthe first the last and the middle Take the median (middle value) of these three as
pivot Itrsquos possible (but not easy) to construct cases
which will make this technique O(n2) Suppose we rearrange (sort) these three
numbers so that the smallest is in the first position the largest in the last position and the other in the middle This lets us simplify and speed up the partition
loop
Final comments Weissrsquos code shows some additional
optimizations on pp 246-247 Weiss chooses to stop both searches on
equality to pivot This design decision is debatable
Quicksort is the fastest known general sorting algorithm on average
For optimum speed the pivot must be chosen carefully
ldquoMedian of threerdquo is a good technique for choosing the pivot
There will be some cases where Quicksort runs in O(n2) time
Quick Sort A couple of notes about quick sort There are more optimal ways to
choose the pivot value (such as the median-of-three method)
Also when the subarrays get small it becomes more efficient to use the insertion sort as opposed to continued use of quick sort
Bubble Sort vs Quick Sort If we calculate the Big-O notation
we find that (in the average case) Bubble Sort O(n2) Quick Sort O(nlog2n)
(quick-sort runs in time O(n2) in the worst case)
Heap Sort Idea take the items that need to be
sorted and insert them into the heap By calling deleteHeap we remove the
smallest or largest element depending on whether or not we are working with a min- or max-heap respectively
Hence the elements are removed in ascending or descending order
Efficiency O(nlog2n)
Merge Sort Idea
Take the array you would like to sort and divide it in half to create 2 unsorted subarrays
Next sort each of the 2 subarrays Finally merge the 2 sorted subarrays
into 1 sorted array Efficiency O(nlog2n)
Merge Sort
Merge Sort Although the merge step produces
a sorted array we have overlooked a very important step
How did we sort the 2 halves before performing the merge step
We used merge sort
Merge Sort By continually calling the merge
sort algorithm we eventually get a subarray of size 1
Since an array with only 1 element is clearly sorted we can back out and merge 2 arrays of size 1
Merge Sort
Merge Sort The basic merging algorithm
consists of 2 input arrays (arrayA and arrayB) An ouput array (arrayC) 3 position holders (indexA indexB
indexC) which are initially set to the beginning of their respective arrays
Merge Sort The smaller of arrayA[indexA] and
arrayB[indexB] is copied into arrayC[indexC] and the appropriate position holders are advanced
When either input list is exhausted the remainder of the other list is copied into arrayC
Merge Sort
1 13 24 26index
A
indexC
arrayA
2 15 27 38index
B
arrayB
arrayC
We compare arrayA[indexA]with arrayB[indexB] Whichever value is smaller isplaced into arrayC[indexC]
1 lt 2 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1index
C
arrayA
2 15 27 38index
B
arrayB
arrayC
2 lt 13 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
13 lt 15 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
15 lt 24 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13 15indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
24 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
indexA
1 2 13 15 24indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
26 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
1 2 13 15 24 26indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
Since we have exhaustedone of the arrays arrayAwe simply copy the remaining items from theother array arrayB intoarrayC
Merge Sort
1 13 24
26
1 2 13 15 24 26 27 38
arrayA
2 15 27 38arrayB
arrayC
Efficiency Summary
Sort Worst Case Average Case
Insertion O(n2) O(n2)
Shell O(n15) O(n125)
Selection O(n2) O(n2)
Bubble O(n2) O(n2)
Quick O(n2) O(nlog2n)
Heap O(nlog2n) O(nlog2n)
Merge O(nlog2n) O(nlog2n)
More Sorting
radix sort bucket sort in-place sorting how fast can we sort
Radix Sort
Unlike other sorting methods radix sort considers the structure of the keys
Assume keys are represented in a base M number system (M = radix) ie if M = 2 the keys are represented in binary
Sorting is done by comparing bits in the same position
Extension to keys that are alphanumeric strings
Radix Exchange Sort
Examine bits from left to right1 Sort array with respect to leftmost bit
2 Partition array
3Recursionrecursively sort top subarray ignoring leftmost bitrecursively sort bottom subarray ignoring leftmost bit
Time to sort n b-bit numbers O(b n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Quick Sort We move left to the right skipping
over elements that are smaller than the pivot
1 4 8 9 0 11 5 10 7 6left righ
t
Quick Sort We then move right to the left skipping
over elements that are greater than the pivot
When left and right have stopped left is on an element greater than (or equal to) the pivot and right is on an element smaller than (or equal to) the pivot
1 4 8 9 0 11 5 10 7 6left right
Quick Sort If left is to the left of right (or if left
= right) those elements are swapped
1 4 8 9 0 11 5 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
Quick Sort The effect is to push a large
element to the right and a small element to the left
We then repeat the process until left and right cross
Quick Sort
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6left righ
t
Quick Sort
1 4 5 0 9 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6righ
tleft
Quick Sort
At this point left and right have crossed so no swap is performed
The final part of the partitioning is to swap the pivot element with left
1 4 5 0 9 11 8 10 7 6righ
tleft
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort Note that all elements to the left of
the pivot are less than (or equal to) the pivot and all elements to the right of the pivot are greater than (or equal to) the pivot
Hence the pivot element has been placed in its final sorted position
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort We now repeat the process using
the sub-arrays to the left and right of the pivot
1 4 5 0 6 11 8 10 7 9
1 4 5 0 11 8 10 7 9
Analysis of quicksortmdashbest case
Suppose each partition operation divides the array almost exactly in half
Then the depth of the recursion in log2n Because thatrsquos how many times we can
halve n However there are many recursions at
each level How can we figure this out We note that
Each partition is linear over its subarray All the partitions at one level cover the array
Partitioning at various levels
Best case II We cut the array size in half each time So the depth of the recursion in log2n At each level of the recursion all the
partitions at that level do work that is linear in n
O(log2n) O(n) = O(n log2n)
What about the worst case
Worst case In the worst case partitioning always
divides the size n array into these three parts A length one part containing the pivot itself A length zero part and A length n-1 part containing everything else
We donrsquot recur on the zero-length part Recurring on the length n-1 part requires
(in the worst case) recurring to depth n-1
Worst case partitioning
Worst case for quicksort In the worst case recursion may be
O(n) levels deep (for an array of size n)
But the partitioning work done at each level is still O(n)
O(n) O(n) = O(n2) So worst case for Quicksort is O(n2) When does this happen
When the array is sorted to begin with
Typical case for quicksort If the array is sorted to begin with
Quicksort is terrible O(n2) It is possible to construct other bad
cases However Quicksort is usually O(n log2n) The constants are so good that Quicksort
is generally the fastest algorithm known Most real-world sorting is done by
Quicksort
Tweaking Quicksort Almost anything you can try to
ldquoimproverdquo Quicksort will actually slow it down
One good tweak is to switch to a different sorting method when the subarrays get small (say 10 or 12) Quicksort has too much overhead for small
array sizes For large arrays it might be a good
idea to check beforehand if the array is already sorted But there is a better tweak than this
Randomized Quick-Sort Select the pivot as a random element of the sequence The expected running time of randomized quick-sort on a
sequence of size n is O(n log n) The time spent at a level of the quick-sort tree is O(n) We show that the expected height of the quick-sort tree is
O(log n) good vs bad pivots
bull The probability of a good pivot is 12 thus we expect k2 good pivots out of k pivotsbull After a good pivot the size of each child sequence is at most 34 the size of the parent sequencebull After h pivots we expect (34)h2 n elementsbull The expected height h of the quick-sort tree is at most 2 log43n
Median of three Obviously it doesnrsquot make sense to sort the
array in order to find the median to use as a pivot
Instead compare just three elements of our (sub)arraymdashthe first the last and the middle Take the median (middle value) of these three as
pivot Itrsquos possible (but not easy) to construct cases
which will make this technique O(n2) Suppose we rearrange (sort) these three
numbers so that the smallest is in the first position the largest in the last position and the other in the middle This lets us simplify and speed up the partition
loop
Final comments Weissrsquos code shows some additional
optimizations on pp 246-247 Weiss chooses to stop both searches on
equality to pivot This design decision is debatable
Quicksort is the fastest known general sorting algorithm on average
For optimum speed the pivot must be chosen carefully
ldquoMedian of threerdquo is a good technique for choosing the pivot
There will be some cases where Quicksort runs in O(n2) time
Quick Sort A couple of notes about quick sort There are more optimal ways to
choose the pivot value (such as the median-of-three method)
Also when the subarrays get small it becomes more efficient to use the insertion sort as opposed to continued use of quick sort
Bubble Sort vs Quick Sort If we calculate the Big-O notation
we find that (in the average case) Bubble Sort O(n2) Quick Sort O(nlog2n)
(quick-sort runs in time O(n2) in the worst case)
Heap Sort Idea take the items that need to be
sorted and insert them into the heap By calling deleteHeap we remove the
smallest or largest element depending on whether or not we are working with a min- or max-heap respectively
Hence the elements are removed in ascending or descending order
Efficiency O(nlog2n)
Merge Sort Idea
Take the array you would like to sort and divide it in half to create 2 unsorted subarrays
Next sort each of the 2 subarrays Finally merge the 2 sorted subarrays
into 1 sorted array Efficiency O(nlog2n)
Merge Sort
Merge Sort Although the merge step produces
a sorted array we have overlooked a very important step
How did we sort the 2 halves before performing the merge step
We used merge sort
Merge Sort By continually calling the merge
sort algorithm we eventually get a subarray of size 1
Since an array with only 1 element is clearly sorted we can back out and merge 2 arrays of size 1
Merge Sort
Merge Sort The basic merging algorithm
consists of 2 input arrays (arrayA and arrayB) An ouput array (arrayC) 3 position holders (indexA indexB
indexC) which are initially set to the beginning of their respective arrays
Merge Sort The smaller of arrayA[indexA] and
arrayB[indexB] is copied into arrayC[indexC] and the appropriate position holders are advanced
When either input list is exhausted the remainder of the other list is copied into arrayC
Merge Sort
1 13 24 26index
A
indexC
arrayA
2 15 27 38index
B
arrayB
arrayC
We compare arrayA[indexA]with arrayB[indexB] Whichever value is smaller isplaced into arrayC[indexC]
1 lt 2 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1index
C
arrayA
2 15 27 38index
B
arrayB
arrayC
2 lt 13 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
13 lt 15 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
15 lt 24 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13 15indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
24 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
indexA
1 2 13 15 24indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
26 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
1 2 13 15 24 26indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
Since we have exhaustedone of the arrays arrayAwe simply copy the remaining items from theother array arrayB intoarrayC
Merge Sort
1 13 24
26
1 2 13 15 24 26 27 38
arrayA
2 15 27 38arrayB
arrayC
Efficiency Summary
Sort Worst Case Average Case
Insertion O(n2) O(n2)
Shell O(n15) O(n125)
Selection O(n2) O(n2)
Bubble O(n2) O(n2)
Quick O(n2) O(nlog2n)
Heap O(nlog2n) O(nlog2n)
Merge O(nlog2n) O(nlog2n)
More Sorting
radix sort bucket sort in-place sorting how fast can we sort
Radix Sort
Unlike other sorting methods radix sort considers the structure of the keys
Assume keys are represented in a base M number system (M = radix) ie if M = 2 the keys are represented in binary
Sorting is done by comparing bits in the same position
Extension to keys that are alphanumeric strings
Radix Exchange Sort
Examine bits from left to right1 Sort array with respect to leftmost bit
2 Partition array
3Recursionrecursively sort top subarray ignoring leftmost bitrecursively sort bottom subarray ignoring leftmost bit
Time to sort n b-bit numbers O(b n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Quick Sort We then move right to the left skipping
over elements that are greater than the pivot
When left and right have stopped left is on an element greater than (or equal to) the pivot and right is on an element smaller than (or equal to) the pivot
1 4 8 9 0 11 5 10 7 6left right
Quick Sort If left is to the left of right (or if left
= right) those elements are swapped
1 4 8 9 0 11 5 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
Quick Sort The effect is to push a large
element to the right and a small element to the left
We then repeat the process until left and right cross
Quick Sort
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6left righ
t
Quick Sort
1 4 5 0 9 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6righ
tleft
Quick Sort
At this point left and right have crossed so no swap is performed
The final part of the partitioning is to swap the pivot element with left
1 4 5 0 9 11 8 10 7 6righ
tleft
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort Note that all elements to the left of
the pivot are less than (or equal to) the pivot and all elements to the right of the pivot are greater than (or equal to) the pivot
Hence the pivot element has been placed in its final sorted position
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort We now repeat the process using
the sub-arrays to the left and right of the pivot
1 4 5 0 6 11 8 10 7 9
1 4 5 0 11 8 10 7 9
Analysis of quicksortmdashbest case
Suppose each partition operation divides the array almost exactly in half
Then the depth of the recursion in log2n Because thatrsquos how many times we can
halve n However there are many recursions at
each level How can we figure this out We note that
Each partition is linear over its subarray All the partitions at one level cover the array
Partitioning at various levels
Best case II We cut the array size in half each time So the depth of the recursion in log2n At each level of the recursion all the
partitions at that level do work that is linear in n
O(log2n) O(n) = O(n log2n)
What about the worst case
Worst case In the worst case partitioning always
divides the size n array into these three parts A length one part containing the pivot itself A length zero part and A length n-1 part containing everything else
We donrsquot recur on the zero-length part Recurring on the length n-1 part requires
(in the worst case) recurring to depth n-1
Worst case partitioning
Worst case for quicksort In the worst case recursion may be
O(n) levels deep (for an array of size n)
But the partitioning work done at each level is still O(n)
O(n) O(n) = O(n2) So worst case for Quicksort is O(n2) When does this happen
When the array is sorted to begin with
Typical case for quicksort If the array is sorted to begin with
Quicksort is terrible O(n2) It is possible to construct other bad
cases However Quicksort is usually O(n log2n) The constants are so good that Quicksort
is generally the fastest algorithm known Most real-world sorting is done by
Quicksort
Tweaking Quicksort Almost anything you can try to
ldquoimproverdquo Quicksort will actually slow it down
One good tweak is to switch to a different sorting method when the subarrays get small (say 10 or 12) Quicksort has too much overhead for small
array sizes For large arrays it might be a good
idea to check beforehand if the array is already sorted But there is a better tweak than this
Randomized Quick-Sort Select the pivot as a random element of the sequence The expected running time of randomized quick-sort on a
sequence of size n is O(n log n) The time spent at a level of the quick-sort tree is O(n) We show that the expected height of the quick-sort tree is
O(log n) good vs bad pivots
bull The probability of a good pivot is 12 thus we expect k2 good pivots out of k pivotsbull After a good pivot the size of each child sequence is at most 34 the size of the parent sequencebull After h pivots we expect (34)h2 n elementsbull The expected height h of the quick-sort tree is at most 2 log43n
Median of three Obviously it doesnrsquot make sense to sort the
array in order to find the median to use as a pivot
Instead compare just three elements of our (sub)arraymdashthe first the last and the middle Take the median (middle value) of these three as
pivot Itrsquos possible (but not easy) to construct cases
which will make this technique O(n2) Suppose we rearrange (sort) these three
numbers so that the smallest is in the first position the largest in the last position and the other in the middle This lets us simplify and speed up the partition
loop
Final comments Weissrsquos code shows some additional
optimizations on pp 246-247 Weiss chooses to stop both searches on
equality to pivot This design decision is debatable
Quicksort is the fastest known general sorting algorithm on average
For optimum speed the pivot must be chosen carefully
ldquoMedian of threerdquo is a good technique for choosing the pivot
There will be some cases where Quicksort runs in O(n2) time
Quick Sort A couple of notes about quick sort There are more optimal ways to
choose the pivot value (such as the median-of-three method)
Also when the subarrays get small it becomes more efficient to use the insertion sort as opposed to continued use of quick sort
Bubble Sort vs Quick Sort If we calculate the Big-O notation
we find that (in the average case) Bubble Sort O(n2) Quick Sort O(nlog2n)
(quick-sort runs in time O(n2) in the worst case)
Heap Sort Idea take the items that need to be
sorted and insert them into the heap By calling deleteHeap we remove the
smallest or largest element depending on whether or not we are working with a min- or max-heap respectively
Hence the elements are removed in ascending or descending order
Efficiency O(nlog2n)
Merge Sort Idea
Take the array you would like to sort and divide it in half to create 2 unsorted subarrays
Next sort each of the 2 subarrays Finally merge the 2 sorted subarrays
into 1 sorted array Efficiency O(nlog2n)
Merge Sort
Merge Sort Although the merge step produces
a sorted array we have overlooked a very important step
How did we sort the 2 halves before performing the merge step
We used merge sort
Merge Sort By continually calling the merge
sort algorithm we eventually get a subarray of size 1
Since an array with only 1 element is clearly sorted we can back out and merge 2 arrays of size 1
Merge Sort
Merge Sort The basic merging algorithm
consists of 2 input arrays (arrayA and arrayB) An ouput array (arrayC) 3 position holders (indexA indexB
indexC) which are initially set to the beginning of their respective arrays
Merge Sort The smaller of arrayA[indexA] and
arrayB[indexB] is copied into arrayC[indexC] and the appropriate position holders are advanced
When either input list is exhausted the remainder of the other list is copied into arrayC
Merge Sort
1 13 24 26index
A
indexC
arrayA
2 15 27 38index
B
arrayB
arrayC
We compare arrayA[indexA]with arrayB[indexB] Whichever value is smaller isplaced into arrayC[indexC]
1 lt 2 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1index
C
arrayA
2 15 27 38index
B
arrayB
arrayC
2 lt 13 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
13 lt 15 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
15 lt 24 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13 15indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
24 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
indexA
1 2 13 15 24indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
26 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
1 2 13 15 24 26indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
Since we have exhaustedone of the arrays arrayAwe simply copy the remaining items from theother array arrayB intoarrayC
Merge Sort
1 13 24
26
1 2 13 15 24 26 27 38
arrayA
2 15 27 38arrayB
arrayC
Efficiency Summary
Sort Worst Case Average Case
Insertion O(n2) O(n2)
Shell O(n15) O(n125)
Selection O(n2) O(n2)
Bubble O(n2) O(n2)
Quick O(n2) O(nlog2n)
Heap O(nlog2n) O(nlog2n)
Merge O(nlog2n) O(nlog2n)
More Sorting
radix sort bucket sort in-place sorting how fast can we sort
Radix Sort
Unlike other sorting methods radix sort considers the structure of the keys
Assume keys are represented in a base M number system (M = radix) ie if M = 2 the keys are represented in binary
Sorting is done by comparing bits in the same position
Extension to keys that are alphanumeric strings
Radix Exchange Sort
Examine bits from left to right1 Sort array with respect to leftmost bit
2 Partition array
3Recursionrecursively sort top subarray ignoring leftmost bitrecursively sort bottom subarray ignoring leftmost bit
Time to sort n b-bit numbers O(b n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Quick Sort If left is to the left of right (or if left
= right) those elements are swapped
1 4 8 9 0 11 5 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
Quick Sort The effect is to push a large
element to the right and a small element to the left
We then repeat the process until left and right cross
Quick Sort
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6left righ
t
Quick Sort
1 4 5 0 9 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6righ
tleft
Quick Sort
At this point left and right have crossed so no swap is performed
The final part of the partitioning is to swap the pivot element with left
1 4 5 0 9 11 8 10 7 6righ
tleft
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort Note that all elements to the left of
the pivot are less than (or equal to) the pivot and all elements to the right of the pivot are greater than (or equal to) the pivot
Hence the pivot element has been placed in its final sorted position
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort We now repeat the process using
the sub-arrays to the left and right of the pivot
1 4 5 0 6 11 8 10 7 9
1 4 5 0 11 8 10 7 9
Analysis of quicksortmdashbest case
Suppose each partition operation divides the array almost exactly in half
Then the depth of the recursion in log2n Because thatrsquos how many times we can
halve n However there are many recursions at
each level How can we figure this out We note that
Each partition is linear over its subarray All the partitions at one level cover the array
Partitioning at various levels
Best case II We cut the array size in half each time So the depth of the recursion in log2n At each level of the recursion all the
partitions at that level do work that is linear in n
O(log2n) O(n) = O(n log2n)
What about the worst case
Worst case In the worst case partitioning always
divides the size n array into these three parts A length one part containing the pivot itself A length zero part and A length n-1 part containing everything else
We donrsquot recur on the zero-length part Recurring on the length n-1 part requires
(in the worst case) recurring to depth n-1
Worst case partitioning
Worst case for quicksort In the worst case recursion may be
O(n) levels deep (for an array of size n)
But the partitioning work done at each level is still O(n)
O(n) O(n) = O(n2) So worst case for Quicksort is O(n2) When does this happen
When the array is sorted to begin with
Typical case for quicksort If the array is sorted to begin with
Quicksort is terrible O(n2) It is possible to construct other bad
cases However Quicksort is usually O(n log2n) The constants are so good that Quicksort
is generally the fastest algorithm known Most real-world sorting is done by
Quicksort
Tweaking Quicksort Almost anything you can try to
ldquoimproverdquo Quicksort will actually slow it down
One good tweak is to switch to a different sorting method when the subarrays get small (say 10 or 12) Quicksort has too much overhead for small
array sizes For large arrays it might be a good
idea to check beforehand if the array is already sorted But there is a better tweak than this
Randomized Quick-Sort Select the pivot as a random element of the sequence The expected running time of randomized quick-sort on a
sequence of size n is O(n log n) The time spent at a level of the quick-sort tree is O(n) We show that the expected height of the quick-sort tree is
O(log n) good vs bad pivots
bull The probability of a good pivot is 12 thus we expect k2 good pivots out of k pivotsbull After a good pivot the size of each child sequence is at most 34 the size of the parent sequencebull After h pivots we expect (34)h2 n elementsbull The expected height h of the quick-sort tree is at most 2 log43n
Median of three Obviously it doesnrsquot make sense to sort the
array in order to find the median to use as a pivot
Instead compare just three elements of our (sub)arraymdashthe first the last and the middle Take the median (middle value) of these three as
pivot Itrsquos possible (but not easy) to construct cases
which will make this technique O(n2) Suppose we rearrange (sort) these three
numbers so that the smallest is in the first position the largest in the last position and the other in the middle This lets us simplify and speed up the partition
loop
Final comments Weissrsquos code shows some additional
optimizations on pp 246-247 Weiss chooses to stop both searches on
equality to pivot This design decision is debatable
Quicksort is the fastest known general sorting algorithm on average
For optimum speed the pivot must be chosen carefully
ldquoMedian of threerdquo is a good technique for choosing the pivot
There will be some cases where Quicksort runs in O(n2) time
Quick Sort A couple of notes about quick sort There are more optimal ways to
choose the pivot value (such as the median-of-three method)
Also when the subarrays get small it becomes more efficient to use the insertion sort as opposed to continued use of quick sort
Bubble Sort vs Quick Sort If we calculate the Big-O notation
we find that (in the average case) Bubble Sort O(n2) Quick Sort O(nlog2n)
(quick-sort runs in time O(n2) in the worst case)
Heap Sort Idea take the items that need to be
sorted and insert them into the heap By calling deleteHeap we remove the
smallest or largest element depending on whether or not we are working with a min- or max-heap respectively
Hence the elements are removed in ascending or descending order
Efficiency O(nlog2n)
Merge Sort Idea
Take the array you would like to sort and divide it in half to create 2 unsorted subarrays
Next sort each of the 2 subarrays Finally merge the 2 sorted subarrays
into 1 sorted array Efficiency O(nlog2n)
Merge Sort
Merge Sort Although the merge step produces
a sorted array we have overlooked a very important step
How did we sort the 2 halves before performing the merge step
We used merge sort
Merge Sort By continually calling the merge
sort algorithm we eventually get a subarray of size 1
Since an array with only 1 element is clearly sorted we can back out and merge 2 arrays of size 1
Merge Sort
Merge Sort The basic merging algorithm
consists of 2 input arrays (arrayA and arrayB) An ouput array (arrayC) 3 position holders (indexA indexB
indexC) which are initially set to the beginning of their respective arrays
Merge Sort The smaller of arrayA[indexA] and
arrayB[indexB] is copied into arrayC[indexC] and the appropriate position holders are advanced
When either input list is exhausted the remainder of the other list is copied into arrayC
Merge Sort
1 13 24 26index
A
indexC
arrayA
2 15 27 38index
B
arrayB
arrayC
We compare arrayA[indexA]with arrayB[indexB] Whichever value is smaller isplaced into arrayC[indexC]
1 lt 2 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1index
C
arrayA
2 15 27 38index
B
arrayB
arrayC
2 lt 13 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
13 lt 15 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
15 lt 24 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13 15indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
24 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
indexA
1 2 13 15 24indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
26 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
1 2 13 15 24 26indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
Since we have exhaustedone of the arrays arrayAwe simply copy the remaining items from theother array arrayB intoarrayC
Merge Sort
1 13 24
26
1 2 13 15 24 26 27 38
arrayA
2 15 27 38arrayB
arrayC
Efficiency Summary
Sort Worst Case Average Case
Insertion O(n2) O(n2)
Shell O(n15) O(n125)
Selection O(n2) O(n2)
Bubble O(n2) O(n2)
Quick O(n2) O(nlog2n)
Heap O(nlog2n) O(nlog2n)
Merge O(nlog2n) O(nlog2n)
More Sorting
radix sort bucket sort in-place sorting how fast can we sort
Radix Sort
Unlike other sorting methods radix sort considers the structure of the keys
Assume keys are represented in a base M number system (M = radix) ie if M = 2 the keys are represented in binary
Sorting is done by comparing bits in the same position
Extension to keys that are alphanumeric strings
Radix Exchange Sort
Examine bits from left to right1 Sort array with respect to leftmost bit
2 Partition array
3Recursionrecursively sort top subarray ignoring leftmost bitrecursively sort bottom subarray ignoring leftmost bit
Time to sort n b-bit numbers O(b n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Quick Sort The effect is to push a large
element to the right and a small element to the left
We then repeat the process until left and right cross
Quick Sort
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6left righ
t
Quick Sort
1 4 5 0 9 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6righ
tleft
Quick Sort
At this point left and right have crossed so no swap is performed
The final part of the partitioning is to swap the pivot element with left
1 4 5 0 9 11 8 10 7 6righ
tleft
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort Note that all elements to the left of
the pivot are less than (or equal to) the pivot and all elements to the right of the pivot are greater than (or equal to) the pivot
Hence the pivot element has been placed in its final sorted position
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort We now repeat the process using
the sub-arrays to the left and right of the pivot
1 4 5 0 6 11 8 10 7 9
1 4 5 0 11 8 10 7 9
Analysis of quicksortmdashbest case
Suppose each partition operation divides the array almost exactly in half
Then the depth of the recursion in log2n Because thatrsquos how many times we can
halve n However there are many recursions at
each level How can we figure this out We note that
Each partition is linear over its subarray All the partitions at one level cover the array
Partitioning at various levels
Best case II We cut the array size in half each time So the depth of the recursion in log2n At each level of the recursion all the
partitions at that level do work that is linear in n
O(log2n) O(n) = O(n log2n)
What about the worst case
Worst case In the worst case partitioning always
divides the size n array into these three parts A length one part containing the pivot itself A length zero part and A length n-1 part containing everything else
We donrsquot recur on the zero-length part Recurring on the length n-1 part requires
(in the worst case) recurring to depth n-1
Worst case partitioning
Worst case for quicksort In the worst case recursion may be
O(n) levels deep (for an array of size n)
But the partitioning work done at each level is still O(n)
O(n) O(n) = O(n2) So worst case for Quicksort is O(n2) When does this happen
When the array is sorted to begin with
Typical case for quicksort If the array is sorted to begin with
Quicksort is terrible O(n2) It is possible to construct other bad
cases However Quicksort is usually O(n log2n) The constants are so good that Quicksort
is generally the fastest algorithm known Most real-world sorting is done by
Quicksort
Tweaking Quicksort Almost anything you can try to
ldquoimproverdquo Quicksort will actually slow it down
One good tweak is to switch to a different sorting method when the subarrays get small (say 10 or 12) Quicksort has too much overhead for small
array sizes For large arrays it might be a good
idea to check beforehand if the array is already sorted But there is a better tweak than this
Randomized Quick-Sort Select the pivot as a random element of the sequence The expected running time of randomized quick-sort on a
sequence of size n is O(n log n) The time spent at a level of the quick-sort tree is O(n) We show that the expected height of the quick-sort tree is
O(log n) good vs bad pivots
bull The probability of a good pivot is 12 thus we expect k2 good pivots out of k pivotsbull After a good pivot the size of each child sequence is at most 34 the size of the parent sequencebull After h pivots we expect (34)h2 n elementsbull The expected height h of the quick-sort tree is at most 2 log43n
Median of three Obviously it doesnrsquot make sense to sort the
array in order to find the median to use as a pivot
Instead compare just three elements of our (sub)arraymdashthe first the last and the middle Take the median (middle value) of these three as
pivot Itrsquos possible (but not easy) to construct cases
which will make this technique O(n2) Suppose we rearrange (sort) these three
numbers so that the smallest is in the first position the largest in the last position and the other in the middle This lets us simplify and speed up the partition
loop
Final comments Weissrsquos code shows some additional
optimizations on pp 246-247 Weiss chooses to stop both searches on
equality to pivot This design decision is debatable
Quicksort is the fastest known general sorting algorithm on average
For optimum speed the pivot must be chosen carefully
ldquoMedian of threerdquo is a good technique for choosing the pivot
There will be some cases where Quicksort runs in O(n2) time
Quick Sort A couple of notes about quick sort There are more optimal ways to
choose the pivot value (such as the median-of-three method)
Also when the subarrays get small it becomes more efficient to use the insertion sort as opposed to continued use of quick sort
Bubble Sort vs Quick Sort If we calculate the Big-O notation
we find that (in the average case) Bubble Sort O(n2) Quick Sort O(nlog2n)
(quick-sort runs in time O(n2) in the worst case)
Heap Sort Idea take the items that need to be
sorted and insert them into the heap By calling deleteHeap we remove the
smallest or largest element depending on whether or not we are working with a min- or max-heap respectively
Hence the elements are removed in ascending or descending order
Efficiency O(nlog2n)
Merge Sort Idea
Take the array you would like to sort and divide it in half to create 2 unsorted subarrays
Next sort each of the 2 subarrays Finally merge the 2 sorted subarrays
into 1 sorted array Efficiency O(nlog2n)
Merge Sort
Merge Sort Although the merge step produces
a sorted array we have overlooked a very important step
How did we sort the 2 halves before performing the merge step
We used merge sort
Merge Sort By continually calling the merge
sort algorithm we eventually get a subarray of size 1
Since an array with only 1 element is clearly sorted we can back out and merge 2 arrays of size 1
Merge Sort
Merge Sort The basic merging algorithm
consists of 2 input arrays (arrayA and arrayB) An ouput array (arrayC) 3 position holders (indexA indexB
indexC) which are initially set to the beginning of their respective arrays
Merge Sort The smaller of arrayA[indexA] and
arrayB[indexB] is copied into arrayC[indexC] and the appropriate position holders are advanced
When either input list is exhausted the remainder of the other list is copied into arrayC
Merge Sort
1 13 24 26index
A
indexC
arrayA
2 15 27 38index
B
arrayB
arrayC
We compare arrayA[indexA]with arrayB[indexB] Whichever value is smaller isplaced into arrayC[indexC]
1 lt 2 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1index
C
arrayA
2 15 27 38index
B
arrayB
arrayC
2 lt 13 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
13 lt 15 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
15 lt 24 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13 15indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
24 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
indexA
1 2 13 15 24indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
26 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
1 2 13 15 24 26indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
Since we have exhaustedone of the arrays arrayAwe simply copy the remaining items from theother array arrayB intoarrayC
Merge Sort
1 13 24
26
1 2 13 15 24 26 27 38
arrayA
2 15 27 38arrayB
arrayC
Efficiency Summary
Sort Worst Case Average Case
Insertion O(n2) O(n2)
Shell O(n15) O(n125)
Selection O(n2) O(n2)
Bubble O(n2) O(n2)
Quick O(n2) O(nlog2n)
Heap O(nlog2n) O(nlog2n)
Merge O(nlog2n) O(nlog2n)
More Sorting
radix sort bucket sort in-place sorting how fast can we sort
Radix Sort
Unlike other sorting methods radix sort considers the structure of the keys
Assume keys are represented in a base M number system (M = radix) ie if M = 2 the keys are represented in binary
Sorting is done by comparing bits in the same position
Extension to keys that are alphanumeric strings
Radix Exchange Sort
Examine bits from left to right1 Sort array with respect to leftmost bit
2 Partition array
3Recursionrecursively sort top subarray ignoring leftmost bitrecursively sort bottom subarray ignoring leftmost bit
Time to sort n b-bit numbers O(b n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Quick Sort
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 9 0 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6left righ
t
Quick Sort
1 4 5 0 9 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6righ
tleft
Quick Sort
At this point left and right have crossed so no swap is performed
The final part of the partitioning is to swap the pivot element with left
1 4 5 0 9 11 8 10 7 6righ
tleft
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort Note that all elements to the left of
the pivot are less than (or equal to) the pivot and all elements to the right of the pivot are greater than (or equal to) the pivot
Hence the pivot element has been placed in its final sorted position
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort We now repeat the process using
the sub-arrays to the left and right of the pivot
1 4 5 0 6 11 8 10 7 9
1 4 5 0 11 8 10 7 9
Analysis of quicksortmdashbest case
Suppose each partition operation divides the array almost exactly in half
Then the depth of the recursion in log2n Because thatrsquos how many times we can
halve n However there are many recursions at
each level How can we figure this out We note that
Each partition is linear over its subarray All the partitions at one level cover the array
Partitioning at various levels
Best case II We cut the array size in half each time So the depth of the recursion in log2n At each level of the recursion all the
partitions at that level do work that is linear in n
O(log2n) O(n) = O(n log2n)
What about the worst case
Worst case In the worst case partitioning always
divides the size n array into these three parts A length one part containing the pivot itself A length zero part and A length n-1 part containing everything else
We donrsquot recur on the zero-length part Recurring on the length n-1 part requires
(in the worst case) recurring to depth n-1
Worst case partitioning
Worst case for quicksort In the worst case recursion may be
O(n) levels deep (for an array of size n)
But the partitioning work done at each level is still O(n)
O(n) O(n) = O(n2) So worst case for Quicksort is O(n2) When does this happen
When the array is sorted to begin with
Typical case for quicksort If the array is sorted to begin with
Quicksort is terrible O(n2) It is possible to construct other bad
cases However Quicksort is usually O(n log2n) The constants are so good that Quicksort
is generally the fastest algorithm known Most real-world sorting is done by
Quicksort
Tweaking Quicksort Almost anything you can try to
ldquoimproverdquo Quicksort will actually slow it down
One good tweak is to switch to a different sorting method when the subarrays get small (say 10 or 12) Quicksort has too much overhead for small
array sizes For large arrays it might be a good
idea to check beforehand if the array is already sorted But there is a better tweak than this
Randomized Quick-Sort Select the pivot as a random element of the sequence The expected running time of randomized quick-sort on a
sequence of size n is O(n log n) The time spent at a level of the quick-sort tree is O(n) We show that the expected height of the quick-sort tree is
O(log n) good vs bad pivots
bull The probability of a good pivot is 12 thus we expect k2 good pivots out of k pivotsbull After a good pivot the size of each child sequence is at most 34 the size of the parent sequencebull After h pivots we expect (34)h2 n elementsbull The expected height h of the quick-sort tree is at most 2 log43n
Median of three Obviously it doesnrsquot make sense to sort the
array in order to find the median to use as a pivot
Instead compare just three elements of our (sub)arraymdashthe first the last and the middle Take the median (middle value) of these three as
pivot Itrsquos possible (but not easy) to construct cases
which will make this technique O(n2) Suppose we rearrange (sort) these three
numbers so that the smallest is in the first position the largest in the last position and the other in the middle This lets us simplify and speed up the partition
loop
Final comments Weissrsquos code shows some additional
optimizations on pp 246-247 Weiss chooses to stop both searches on
equality to pivot This design decision is debatable
Quicksort is the fastest known general sorting algorithm on average
For optimum speed the pivot must be chosen carefully
ldquoMedian of threerdquo is a good technique for choosing the pivot
There will be some cases where Quicksort runs in O(n2) time
Quick Sort A couple of notes about quick sort There are more optimal ways to
choose the pivot value (such as the median-of-three method)
Also when the subarrays get small it becomes more efficient to use the insertion sort as opposed to continued use of quick sort
Bubble Sort vs Quick Sort If we calculate the Big-O notation
we find that (in the average case) Bubble Sort O(n2) Quick Sort O(nlog2n)
(quick-sort runs in time O(n2) in the worst case)
Heap Sort Idea take the items that need to be
sorted and insert them into the heap By calling deleteHeap we remove the
smallest or largest element depending on whether or not we are working with a min- or max-heap respectively
Hence the elements are removed in ascending or descending order
Efficiency O(nlog2n)
Merge Sort Idea
Take the array you would like to sort and divide it in half to create 2 unsorted subarrays
Next sort each of the 2 subarrays Finally merge the 2 sorted subarrays
into 1 sorted array Efficiency O(nlog2n)
Merge Sort
Merge Sort Although the merge step produces
a sorted array we have overlooked a very important step
How did we sort the 2 halves before performing the merge step
We used merge sort
Merge Sort By continually calling the merge
sort algorithm we eventually get a subarray of size 1
Since an array with only 1 element is clearly sorted we can back out and merge 2 arrays of size 1
Merge Sort
Merge Sort The basic merging algorithm
consists of 2 input arrays (arrayA and arrayB) An ouput array (arrayC) 3 position holders (indexA indexB
indexC) which are initially set to the beginning of their respective arrays
Merge Sort The smaller of arrayA[indexA] and
arrayB[indexB] is copied into arrayC[indexC] and the appropriate position holders are advanced
When either input list is exhausted the remainder of the other list is copied into arrayC
Merge Sort
1 13 24 26index
A
indexC
arrayA
2 15 27 38index
B
arrayB
arrayC
We compare arrayA[indexA]with arrayB[indexB] Whichever value is smaller isplaced into arrayC[indexC]
1 lt 2 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1index
C
arrayA
2 15 27 38index
B
arrayB
arrayC
2 lt 13 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
13 lt 15 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
15 lt 24 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13 15indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
24 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
indexA
1 2 13 15 24indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
26 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
1 2 13 15 24 26indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
Since we have exhaustedone of the arrays arrayAwe simply copy the remaining items from theother array arrayB intoarrayC
Merge Sort
1 13 24
26
1 2 13 15 24 26 27 38
arrayA
2 15 27 38arrayB
arrayC
Efficiency Summary
Sort Worst Case Average Case
Insertion O(n2) O(n2)
Shell O(n15) O(n125)
Selection O(n2) O(n2)
Bubble O(n2) O(n2)
Quick O(n2) O(nlog2n)
Heap O(nlog2n) O(nlog2n)
Merge O(nlog2n) O(nlog2n)
More Sorting
radix sort bucket sort in-place sorting how fast can we sort
Radix Sort
Unlike other sorting methods radix sort considers the structure of the keys
Assume keys are represented in a base M number system (M = radix) ie if M = 2 the keys are represented in binary
Sorting is done by comparing bits in the same position
Extension to keys that are alphanumeric strings
Radix Exchange Sort
Examine bits from left to right1 Sort array with respect to leftmost bit
2 Partition array
3Recursionrecursively sort top subarray ignoring leftmost bitrecursively sort bottom subarray ignoring leftmost bit
Time to sort n b-bit numbers O(b n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Quick Sort
1 4 5 0 9 11 8 10 7 6left righ
t
1 4 5 0 9 11 8 10 7 6righ
tleft
Quick Sort
At this point left and right have crossed so no swap is performed
The final part of the partitioning is to swap the pivot element with left
1 4 5 0 9 11 8 10 7 6righ
tleft
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort Note that all elements to the left of
the pivot are less than (or equal to) the pivot and all elements to the right of the pivot are greater than (or equal to) the pivot
Hence the pivot element has been placed in its final sorted position
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort We now repeat the process using
the sub-arrays to the left and right of the pivot
1 4 5 0 6 11 8 10 7 9
1 4 5 0 11 8 10 7 9
Analysis of quicksortmdashbest case
Suppose each partition operation divides the array almost exactly in half
Then the depth of the recursion in log2n Because thatrsquos how many times we can
halve n However there are many recursions at
each level How can we figure this out We note that
Each partition is linear over its subarray All the partitions at one level cover the array
Partitioning at various levels
Best case II We cut the array size in half each time So the depth of the recursion in log2n At each level of the recursion all the
partitions at that level do work that is linear in n
O(log2n) O(n) = O(n log2n)
What about the worst case
Worst case In the worst case partitioning always
divides the size n array into these three parts A length one part containing the pivot itself A length zero part and A length n-1 part containing everything else
We donrsquot recur on the zero-length part Recurring on the length n-1 part requires
(in the worst case) recurring to depth n-1
Worst case partitioning
Worst case for quicksort In the worst case recursion may be
O(n) levels deep (for an array of size n)
But the partitioning work done at each level is still O(n)
O(n) O(n) = O(n2) So worst case for Quicksort is O(n2) When does this happen
When the array is sorted to begin with
Typical case for quicksort If the array is sorted to begin with
Quicksort is terrible O(n2) It is possible to construct other bad
cases However Quicksort is usually O(n log2n) The constants are so good that Quicksort
is generally the fastest algorithm known Most real-world sorting is done by
Quicksort
Tweaking Quicksort Almost anything you can try to
ldquoimproverdquo Quicksort will actually slow it down
One good tweak is to switch to a different sorting method when the subarrays get small (say 10 or 12) Quicksort has too much overhead for small
array sizes For large arrays it might be a good
idea to check beforehand if the array is already sorted But there is a better tweak than this
Randomized Quick-Sort Select the pivot as a random element of the sequence The expected running time of randomized quick-sort on a
sequence of size n is O(n log n) The time spent at a level of the quick-sort tree is O(n) We show that the expected height of the quick-sort tree is
O(log n) good vs bad pivots
bull The probability of a good pivot is 12 thus we expect k2 good pivots out of k pivotsbull After a good pivot the size of each child sequence is at most 34 the size of the parent sequencebull After h pivots we expect (34)h2 n elementsbull The expected height h of the quick-sort tree is at most 2 log43n
Median of three Obviously it doesnrsquot make sense to sort the
array in order to find the median to use as a pivot
Instead compare just three elements of our (sub)arraymdashthe first the last and the middle Take the median (middle value) of these three as
pivot Itrsquos possible (but not easy) to construct cases
which will make this technique O(n2) Suppose we rearrange (sort) these three
numbers so that the smallest is in the first position the largest in the last position and the other in the middle This lets us simplify and speed up the partition
loop
Final comments Weissrsquos code shows some additional
optimizations on pp 246-247 Weiss chooses to stop both searches on
equality to pivot This design decision is debatable
Quicksort is the fastest known general sorting algorithm on average
For optimum speed the pivot must be chosen carefully
ldquoMedian of threerdquo is a good technique for choosing the pivot
There will be some cases where Quicksort runs in O(n2) time
Quick Sort A couple of notes about quick sort There are more optimal ways to
choose the pivot value (such as the median-of-three method)
Also when the subarrays get small it becomes more efficient to use the insertion sort as opposed to continued use of quick sort
Bubble Sort vs Quick Sort If we calculate the Big-O notation
we find that (in the average case) Bubble Sort O(n2) Quick Sort O(nlog2n)
(quick-sort runs in time O(n2) in the worst case)
Heap Sort Idea take the items that need to be
sorted and insert them into the heap By calling deleteHeap we remove the
smallest or largest element depending on whether or not we are working with a min- or max-heap respectively
Hence the elements are removed in ascending or descending order
Efficiency O(nlog2n)
Merge Sort Idea
Take the array you would like to sort and divide it in half to create 2 unsorted subarrays
Next sort each of the 2 subarrays Finally merge the 2 sorted subarrays
into 1 sorted array Efficiency O(nlog2n)
Merge Sort
Merge Sort Although the merge step produces
a sorted array we have overlooked a very important step
How did we sort the 2 halves before performing the merge step
We used merge sort
Merge Sort By continually calling the merge
sort algorithm we eventually get a subarray of size 1
Since an array with only 1 element is clearly sorted we can back out and merge 2 arrays of size 1
Merge Sort
Merge Sort The basic merging algorithm
consists of 2 input arrays (arrayA and arrayB) An ouput array (arrayC) 3 position holders (indexA indexB
indexC) which are initially set to the beginning of their respective arrays
Merge Sort The smaller of arrayA[indexA] and
arrayB[indexB] is copied into arrayC[indexC] and the appropriate position holders are advanced
When either input list is exhausted the remainder of the other list is copied into arrayC
Merge Sort
1 13 24 26index
A
indexC
arrayA
2 15 27 38index
B
arrayB
arrayC
We compare arrayA[indexA]with arrayB[indexB] Whichever value is smaller isplaced into arrayC[indexC]
1 lt 2 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1index
C
arrayA
2 15 27 38index
B
arrayB
arrayC
2 lt 13 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
13 lt 15 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
15 lt 24 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13 15indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
24 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
indexA
1 2 13 15 24indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
26 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
1 2 13 15 24 26indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
Since we have exhaustedone of the arrays arrayAwe simply copy the remaining items from theother array arrayB intoarrayC
Merge Sort
1 13 24
26
1 2 13 15 24 26 27 38
arrayA
2 15 27 38arrayB
arrayC
Efficiency Summary
Sort Worst Case Average Case
Insertion O(n2) O(n2)
Shell O(n15) O(n125)
Selection O(n2) O(n2)
Bubble O(n2) O(n2)
Quick O(n2) O(nlog2n)
Heap O(nlog2n) O(nlog2n)
Merge O(nlog2n) O(nlog2n)
More Sorting
radix sort bucket sort in-place sorting how fast can we sort
Radix Sort
Unlike other sorting methods radix sort considers the structure of the keys
Assume keys are represented in a base M number system (M = radix) ie if M = 2 the keys are represented in binary
Sorting is done by comparing bits in the same position
Extension to keys that are alphanumeric strings
Radix Exchange Sort
Examine bits from left to right1 Sort array with respect to leftmost bit
2 Partition array
3Recursionrecursively sort top subarray ignoring leftmost bitrecursively sort bottom subarray ignoring leftmost bit
Time to sort n b-bit numbers O(b n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Quick Sort
At this point left and right have crossed so no swap is performed
The final part of the partitioning is to swap the pivot element with left
1 4 5 0 9 11 8 10 7 6righ
tleft
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort Note that all elements to the left of
the pivot are less than (or equal to) the pivot and all elements to the right of the pivot are greater than (or equal to) the pivot
Hence the pivot element has been placed in its final sorted position
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort We now repeat the process using
the sub-arrays to the left and right of the pivot
1 4 5 0 6 11 8 10 7 9
1 4 5 0 11 8 10 7 9
Analysis of quicksortmdashbest case
Suppose each partition operation divides the array almost exactly in half
Then the depth of the recursion in log2n Because thatrsquos how many times we can
halve n However there are many recursions at
each level How can we figure this out We note that
Each partition is linear over its subarray All the partitions at one level cover the array
Partitioning at various levels
Best case II We cut the array size in half each time So the depth of the recursion in log2n At each level of the recursion all the
partitions at that level do work that is linear in n
O(log2n) O(n) = O(n log2n)
What about the worst case
Worst case In the worst case partitioning always
divides the size n array into these three parts A length one part containing the pivot itself A length zero part and A length n-1 part containing everything else
We donrsquot recur on the zero-length part Recurring on the length n-1 part requires
(in the worst case) recurring to depth n-1
Worst case partitioning
Worst case for quicksort In the worst case recursion may be
O(n) levels deep (for an array of size n)
But the partitioning work done at each level is still O(n)
O(n) O(n) = O(n2) So worst case for Quicksort is O(n2) When does this happen
When the array is sorted to begin with
Typical case for quicksort If the array is sorted to begin with
Quicksort is terrible O(n2) It is possible to construct other bad
cases However Quicksort is usually O(n log2n) The constants are so good that Quicksort
is generally the fastest algorithm known Most real-world sorting is done by
Quicksort
Tweaking Quicksort Almost anything you can try to
ldquoimproverdquo Quicksort will actually slow it down
One good tweak is to switch to a different sorting method when the subarrays get small (say 10 or 12) Quicksort has too much overhead for small
array sizes For large arrays it might be a good
idea to check beforehand if the array is already sorted But there is a better tweak than this
Randomized Quick-Sort Select the pivot as a random element of the sequence The expected running time of randomized quick-sort on a
sequence of size n is O(n log n) The time spent at a level of the quick-sort tree is O(n) We show that the expected height of the quick-sort tree is
O(log n) good vs bad pivots
bull The probability of a good pivot is 12 thus we expect k2 good pivots out of k pivotsbull After a good pivot the size of each child sequence is at most 34 the size of the parent sequencebull After h pivots we expect (34)h2 n elementsbull The expected height h of the quick-sort tree is at most 2 log43n
Median of three Obviously it doesnrsquot make sense to sort the
array in order to find the median to use as a pivot
Instead compare just three elements of our (sub)arraymdashthe first the last and the middle Take the median (middle value) of these three as
pivot Itrsquos possible (but not easy) to construct cases
which will make this technique O(n2) Suppose we rearrange (sort) these three
numbers so that the smallest is in the first position the largest in the last position and the other in the middle This lets us simplify and speed up the partition
loop
Final comments Weissrsquos code shows some additional
optimizations on pp 246-247 Weiss chooses to stop both searches on
equality to pivot This design decision is debatable
Quicksort is the fastest known general sorting algorithm on average
For optimum speed the pivot must be chosen carefully
ldquoMedian of threerdquo is a good technique for choosing the pivot
There will be some cases where Quicksort runs in O(n2) time
Quick Sort A couple of notes about quick sort There are more optimal ways to
choose the pivot value (such as the median-of-three method)
Also when the subarrays get small it becomes more efficient to use the insertion sort as opposed to continued use of quick sort
Bubble Sort vs Quick Sort If we calculate the Big-O notation
we find that (in the average case) Bubble Sort O(n2) Quick Sort O(nlog2n)
(quick-sort runs in time O(n2) in the worst case)
Heap Sort Idea take the items that need to be
sorted and insert them into the heap By calling deleteHeap we remove the
smallest or largest element depending on whether or not we are working with a min- or max-heap respectively
Hence the elements are removed in ascending or descending order
Efficiency O(nlog2n)
Merge Sort Idea
Take the array you would like to sort and divide it in half to create 2 unsorted subarrays
Next sort each of the 2 subarrays Finally merge the 2 sorted subarrays
into 1 sorted array Efficiency O(nlog2n)
Merge Sort
Merge Sort Although the merge step produces
a sorted array we have overlooked a very important step
How did we sort the 2 halves before performing the merge step
We used merge sort
Merge Sort By continually calling the merge
sort algorithm we eventually get a subarray of size 1
Since an array with only 1 element is clearly sorted we can back out and merge 2 arrays of size 1
Merge Sort
Merge Sort The basic merging algorithm
consists of 2 input arrays (arrayA and arrayB) An ouput array (arrayC) 3 position holders (indexA indexB
indexC) which are initially set to the beginning of their respective arrays
Merge Sort The smaller of arrayA[indexA] and
arrayB[indexB] is copied into arrayC[indexC] and the appropriate position holders are advanced
When either input list is exhausted the remainder of the other list is copied into arrayC
Merge Sort
1 13 24 26index
A
indexC
arrayA
2 15 27 38index
B
arrayB
arrayC
We compare arrayA[indexA]with arrayB[indexB] Whichever value is smaller isplaced into arrayC[indexC]
1 lt 2 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1index
C
arrayA
2 15 27 38index
B
arrayB
arrayC
2 lt 13 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
13 lt 15 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
15 lt 24 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13 15indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
24 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
indexA
1 2 13 15 24indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
26 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
1 2 13 15 24 26indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
Since we have exhaustedone of the arrays arrayAwe simply copy the remaining items from theother array arrayB intoarrayC
Merge Sort
1 13 24
26
1 2 13 15 24 26 27 38
arrayA
2 15 27 38arrayB
arrayC
Efficiency Summary
Sort Worst Case Average Case
Insertion O(n2) O(n2)
Shell O(n15) O(n125)
Selection O(n2) O(n2)
Bubble O(n2) O(n2)
Quick O(n2) O(nlog2n)
Heap O(nlog2n) O(nlog2n)
Merge O(nlog2n) O(nlog2n)
More Sorting
radix sort bucket sort in-place sorting how fast can we sort
Radix Sort
Unlike other sorting methods radix sort considers the structure of the keys
Assume keys are represented in a base M number system (M = radix) ie if M = 2 the keys are represented in binary
Sorting is done by comparing bits in the same position
Extension to keys that are alphanumeric strings
Radix Exchange Sort
Examine bits from left to right1 Sort array with respect to leftmost bit
2 Partition array
3Recursionrecursively sort top subarray ignoring leftmost bitrecursively sort bottom subarray ignoring leftmost bit
Time to sort n b-bit numbers O(b n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Quick Sort Note that all elements to the left of
the pivot are less than (or equal to) the pivot and all elements to the right of the pivot are greater than (or equal to) the pivot
Hence the pivot element has been placed in its final sorted position
1 4 5 0 6 11 8 10 7 9righ
tleft
Quick Sort We now repeat the process using
the sub-arrays to the left and right of the pivot
1 4 5 0 6 11 8 10 7 9
1 4 5 0 11 8 10 7 9
Analysis of quicksortmdashbest case
Suppose each partition operation divides the array almost exactly in half
Then the depth of the recursion in log2n Because thatrsquos how many times we can
halve n However there are many recursions at
each level How can we figure this out We note that
Each partition is linear over its subarray All the partitions at one level cover the array
Partitioning at various levels
Best case II We cut the array size in half each time So the depth of the recursion in log2n At each level of the recursion all the
partitions at that level do work that is linear in n
O(log2n) O(n) = O(n log2n)
What about the worst case
Worst case In the worst case partitioning always
divides the size n array into these three parts A length one part containing the pivot itself A length zero part and A length n-1 part containing everything else
We donrsquot recur on the zero-length part Recurring on the length n-1 part requires
(in the worst case) recurring to depth n-1
Worst case partitioning
Worst case for quicksort In the worst case recursion may be
O(n) levels deep (for an array of size n)
But the partitioning work done at each level is still O(n)
O(n) O(n) = O(n2) So worst case for Quicksort is O(n2) When does this happen
When the array is sorted to begin with
Typical case for quicksort If the array is sorted to begin with
Quicksort is terrible O(n2) It is possible to construct other bad
cases However Quicksort is usually O(n log2n) The constants are so good that Quicksort
is generally the fastest algorithm known Most real-world sorting is done by
Quicksort
Tweaking Quicksort Almost anything you can try to
ldquoimproverdquo Quicksort will actually slow it down
One good tweak is to switch to a different sorting method when the subarrays get small (say 10 or 12) Quicksort has too much overhead for small
array sizes For large arrays it might be a good
idea to check beforehand if the array is already sorted But there is a better tweak than this
Randomized Quick-Sort Select the pivot as a random element of the sequence The expected running time of randomized quick-sort on a
sequence of size n is O(n log n) The time spent at a level of the quick-sort tree is O(n) We show that the expected height of the quick-sort tree is
O(log n) good vs bad pivots
bull The probability of a good pivot is 12 thus we expect k2 good pivots out of k pivotsbull After a good pivot the size of each child sequence is at most 34 the size of the parent sequencebull After h pivots we expect (34)h2 n elementsbull The expected height h of the quick-sort tree is at most 2 log43n
Median of three Obviously it doesnrsquot make sense to sort the
array in order to find the median to use as a pivot
Instead compare just three elements of our (sub)arraymdashthe first the last and the middle Take the median (middle value) of these three as
pivot Itrsquos possible (but not easy) to construct cases
which will make this technique O(n2) Suppose we rearrange (sort) these three
numbers so that the smallest is in the first position the largest in the last position and the other in the middle This lets us simplify and speed up the partition
loop
Final comments Weissrsquos code shows some additional
optimizations on pp 246-247 Weiss chooses to stop both searches on
equality to pivot This design decision is debatable
Quicksort is the fastest known general sorting algorithm on average
For optimum speed the pivot must be chosen carefully
ldquoMedian of threerdquo is a good technique for choosing the pivot
There will be some cases where Quicksort runs in O(n2) time
Quick Sort A couple of notes about quick sort There are more optimal ways to
choose the pivot value (such as the median-of-three method)
Also when the subarrays get small it becomes more efficient to use the insertion sort as opposed to continued use of quick sort
Bubble Sort vs Quick Sort If we calculate the Big-O notation
we find that (in the average case) Bubble Sort O(n2) Quick Sort O(nlog2n)
(quick-sort runs in time O(n2) in the worst case)
Heap Sort Idea take the items that need to be
sorted and insert them into the heap By calling deleteHeap we remove the
smallest or largest element depending on whether or not we are working with a min- or max-heap respectively
Hence the elements are removed in ascending or descending order
Efficiency O(nlog2n)
Merge Sort Idea
Take the array you would like to sort and divide it in half to create 2 unsorted subarrays
Next sort each of the 2 subarrays Finally merge the 2 sorted subarrays
into 1 sorted array Efficiency O(nlog2n)
Merge Sort
Merge Sort Although the merge step produces
a sorted array we have overlooked a very important step
How did we sort the 2 halves before performing the merge step
We used merge sort
Merge Sort By continually calling the merge
sort algorithm we eventually get a subarray of size 1
Since an array with only 1 element is clearly sorted we can back out and merge 2 arrays of size 1
Merge Sort
Merge Sort The basic merging algorithm
consists of 2 input arrays (arrayA and arrayB) An ouput array (arrayC) 3 position holders (indexA indexB
indexC) which are initially set to the beginning of their respective arrays
Merge Sort The smaller of arrayA[indexA] and
arrayB[indexB] is copied into arrayC[indexC] and the appropriate position holders are advanced
When either input list is exhausted the remainder of the other list is copied into arrayC
Merge Sort
1 13 24 26index
A
indexC
arrayA
2 15 27 38index
B
arrayB
arrayC
We compare arrayA[indexA]with arrayB[indexB] Whichever value is smaller isplaced into arrayC[indexC]
1 lt 2 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1index
C
arrayA
2 15 27 38index
B
arrayB
arrayC
2 lt 13 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
13 lt 15 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
15 lt 24 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13 15indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
24 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
indexA
1 2 13 15 24indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
26 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
1 2 13 15 24 26indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
Since we have exhaustedone of the arrays arrayAwe simply copy the remaining items from theother array arrayB intoarrayC
Merge Sort
1 13 24
26
1 2 13 15 24 26 27 38
arrayA
2 15 27 38arrayB
arrayC
Efficiency Summary
Sort Worst Case Average Case
Insertion O(n2) O(n2)
Shell O(n15) O(n125)
Selection O(n2) O(n2)
Bubble O(n2) O(n2)
Quick O(n2) O(nlog2n)
Heap O(nlog2n) O(nlog2n)
Merge O(nlog2n) O(nlog2n)
More Sorting
radix sort bucket sort in-place sorting how fast can we sort
Radix Sort
Unlike other sorting methods radix sort considers the structure of the keys
Assume keys are represented in a base M number system (M = radix) ie if M = 2 the keys are represented in binary
Sorting is done by comparing bits in the same position
Extension to keys that are alphanumeric strings
Radix Exchange Sort
Examine bits from left to right1 Sort array with respect to leftmost bit
2 Partition array
3Recursionrecursively sort top subarray ignoring leftmost bitrecursively sort bottom subarray ignoring leftmost bit
Time to sort n b-bit numbers O(b n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Quick Sort We now repeat the process using
the sub-arrays to the left and right of the pivot
1 4 5 0 6 11 8 10 7 9
1 4 5 0 11 8 10 7 9
Analysis of quicksortmdashbest case
Suppose each partition operation divides the array almost exactly in half
Then the depth of the recursion in log2n Because thatrsquos how many times we can
halve n However there are many recursions at
each level How can we figure this out We note that
Each partition is linear over its subarray All the partitions at one level cover the array
Partitioning at various levels
Best case II We cut the array size in half each time So the depth of the recursion in log2n At each level of the recursion all the
partitions at that level do work that is linear in n
O(log2n) O(n) = O(n log2n)
What about the worst case
Worst case In the worst case partitioning always
divides the size n array into these three parts A length one part containing the pivot itself A length zero part and A length n-1 part containing everything else
We donrsquot recur on the zero-length part Recurring on the length n-1 part requires
(in the worst case) recurring to depth n-1
Worst case partitioning
Worst case for quicksort In the worst case recursion may be
O(n) levels deep (for an array of size n)
But the partitioning work done at each level is still O(n)
O(n) O(n) = O(n2) So worst case for Quicksort is O(n2) When does this happen
When the array is sorted to begin with
Typical case for quicksort If the array is sorted to begin with
Quicksort is terrible O(n2) It is possible to construct other bad
cases However Quicksort is usually O(n log2n) The constants are so good that Quicksort
is generally the fastest algorithm known Most real-world sorting is done by
Quicksort
Tweaking Quicksort Almost anything you can try to
ldquoimproverdquo Quicksort will actually slow it down
One good tweak is to switch to a different sorting method when the subarrays get small (say 10 or 12) Quicksort has too much overhead for small
array sizes For large arrays it might be a good
idea to check beforehand if the array is already sorted But there is a better tweak than this
Randomized Quick-Sort Select the pivot as a random element of the sequence The expected running time of randomized quick-sort on a
sequence of size n is O(n log n) The time spent at a level of the quick-sort tree is O(n) We show that the expected height of the quick-sort tree is
O(log n) good vs bad pivots
bull The probability of a good pivot is 12 thus we expect k2 good pivots out of k pivotsbull After a good pivot the size of each child sequence is at most 34 the size of the parent sequencebull After h pivots we expect (34)h2 n elementsbull The expected height h of the quick-sort tree is at most 2 log43n
Median of three Obviously it doesnrsquot make sense to sort the
array in order to find the median to use as a pivot
Instead compare just three elements of our (sub)arraymdashthe first the last and the middle Take the median (middle value) of these three as
pivot Itrsquos possible (but not easy) to construct cases
which will make this technique O(n2) Suppose we rearrange (sort) these three
numbers so that the smallest is in the first position the largest in the last position and the other in the middle This lets us simplify and speed up the partition
loop
Final comments Weissrsquos code shows some additional
optimizations on pp 246-247 Weiss chooses to stop both searches on
equality to pivot This design decision is debatable
Quicksort is the fastest known general sorting algorithm on average
For optimum speed the pivot must be chosen carefully
ldquoMedian of threerdquo is a good technique for choosing the pivot
There will be some cases where Quicksort runs in O(n2) time
Quick Sort A couple of notes about quick sort There are more optimal ways to
choose the pivot value (such as the median-of-three method)
Also when the subarrays get small it becomes more efficient to use the insertion sort as opposed to continued use of quick sort
Bubble Sort vs Quick Sort If we calculate the Big-O notation
we find that (in the average case) Bubble Sort O(n2) Quick Sort O(nlog2n)
(quick-sort runs in time O(n2) in the worst case)
Heap Sort Idea take the items that need to be
sorted and insert them into the heap By calling deleteHeap we remove the
smallest or largest element depending on whether or not we are working with a min- or max-heap respectively
Hence the elements are removed in ascending or descending order
Efficiency O(nlog2n)
Merge Sort Idea
Take the array you would like to sort and divide it in half to create 2 unsorted subarrays
Next sort each of the 2 subarrays Finally merge the 2 sorted subarrays
into 1 sorted array Efficiency O(nlog2n)
Merge Sort
Merge Sort Although the merge step produces
a sorted array we have overlooked a very important step
How did we sort the 2 halves before performing the merge step
We used merge sort
Merge Sort By continually calling the merge
sort algorithm we eventually get a subarray of size 1
Since an array with only 1 element is clearly sorted we can back out and merge 2 arrays of size 1
Merge Sort
Merge Sort The basic merging algorithm
consists of 2 input arrays (arrayA and arrayB) An ouput array (arrayC) 3 position holders (indexA indexB
indexC) which are initially set to the beginning of their respective arrays
Merge Sort The smaller of arrayA[indexA] and
arrayB[indexB] is copied into arrayC[indexC] and the appropriate position holders are advanced
When either input list is exhausted the remainder of the other list is copied into arrayC
Merge Sort
1 13 24 26index
A
indexC
arrayA
2 15 27 38index
B
arrayB
arrayC
We compare arrayA[indexA]with arrayB[indexB] Whichever value is smaller isplaced into arrayC[indexC]
1 lt 2 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1index
C
arrayA
2 15 27 38index
B
arrayB
arrayC
2 lt 13 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
13 lt 15 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
15 lt 24 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13 15indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
24 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
indexA
1 2 13 15 24indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
26 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
1 2 13 15 24 26indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
Since we have exhaustedone of the arrays arrayAwe simply copy the remaining items from theother array arrayB intoarrayC
Merge Sort
1 13 24
26
1 2 13 15 24 26 27 38
arrayA
2 15 27 38arrayB
arrayC
Efficiency Summary
Sort Worst Case Average Case
Insertion O(n2) O(n2)
Shell O(n15) O(n125)
Selection O(n2) O(n2)
Bubble O(n2) O(n2)
Quick O(n2) O(nlog2n)
Heap O(nlog2n) O(nlog2n)
Merge O(nlog2n) O(nlog2n)
More Sorting
radix sort bucket sort in-place sorting how fast can we sort
Radix Sort
Unlike other sorting methods radix sort considers the structure of the keys
Assume keys are represented in a base M number system (M = radix) ie if M = 2 the keys are represented in binary
Sorting is done by comparing bits in the same position
Extension to keys that are alphanumeric strings
Radix Exchange Sort
Examine bits from left to right1 Sort array with respect to leftmost bit
2 Partition array
3Recursionrecursively sort top subarray ignoring leftmost bitrecursively sort bottom subarray ignoring leftmost bit
Time to sort n b-bit numbers O(b n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Analysis of quicksortmdashbest case
Suppose each partition operation divides the array almost exactly in half
Then the depth of the recursion in log2n Because thatrsquos how many times we can
halve n However there are many recursions at
each level How can we figure this out We note that
Each partition is linear over its subarray All the partitions at one level cover the array
Partitioning at various levels
Best case II We cut the array size in half each time So the depth of the recursion in log2n At each level of the recursion all the
partitions at that level do work that is linear in n
O(log2n) O(n) = O(n log2n)
What about the worst case
Worst case In the worst case partitioning always
divides the size n array into these three parts A length one part containing the pivot itself A length zero part and A length n-1 part containing everything else
We donrsquot recur on the zero-length part Recurring on the length n-1 part requires
(in the worst case) recurring to depth n-1
Worst case partitioning
Worst case for quicksort In the worst case recursion may be
O(n) levels deep (for an array of size n)
But the partitioning work done at each level is still O(n)
O(n) O(n) = O(n2) So worst case for Quicksort is O(n2) When does this happen
When the array is sorted to begin with
Typical case for quicksort If the array is sorted to begin with
Quicksort is terrible O(n2) It is possible to construct other bad
cases However Quicksort is usually O(n log2n) The constants are so good that Quicksort
is generally the fastest algorithm known Most real-world sorting is done by
Quicksort
Tweaking Quicksort Almost anything you can try to
ldquoimproverdquo Quicksort will actually slow it down
One good tweak is to switch to a different sorting method when the subarrays get small (say 10 or 12) Quicksort has too much overhead for small
array sizes For large arrays it might be a good
idea to check beforehand if the array is already sorted But there is a better tweak than this
Randomized Quick-Sort Select the pivot as a random element of the sequence The expected running time of randomized quick-sort on a
sequence of size n is O(n log n) The time spent at a level of the quick-sort tree is O(n) We show that the expected height of the quick-sort tree is
O(log n) good vs bad pivots
bull The probability of a good pivot is 12 thus we expect k2 good pivots out of k pivotsbull After a good pivot the size of each child sequence is at most 34 the size of the parent sequencebull After h pivots we expect (34)h2 n elementsbull The expected height h of the quick-sort tree is at most 2 log43n
Median of three Obviously it doesnrsquot make sense to sort the
array in order to find the median to use as a pivot
Instead compare just three elements of our (sub)arraymdashthe first the last and the middle Take the median (middle value) of these three as
pivot Itrsquos possible (but not easy) to construct cases
which will make this technique O(n2) Suppose we rearrange (sort) these three
numbers so that the smallest is in the first position the largest in the last position and the other in the middle This lets us simplify and speed up the partition
loop
Final comments Weissrsquos code shows some additional
optimizations on pp 246-247 Weiss chooses to stop both searches on
equality to pivot This design decision is debatable
Quicksort is the fastest known general sorting algorithm on average
For optimum speed the pivot must be chosen carefully
ldquoMedian of threerdquo is a good technique for choosing the pivot
There will be some cases where Quicksort runs in O(n2) time
Quick Sort A couple of notes about quick sort There are more optimal ways to
choose the pivot value (such as the median-of-three method)
Also when the subarrays get small it becomes more efficient to use the insertion sort as opposed to continued use of quick sort
Bubble Sort vs Quick Sort If we calculate the Big-O notation
we find that (in the average case) Bubble Sort O(n2) Quick Sort O(nlog2n)
(quick-sort runs in time O(n2) in the worst case)
Heap Sort Idea take the items that need to be
sorted and insert them into the heap By calling deleteHeap we remove the
smallest or largest element depending on whether or not we are working with a min- or max-heap respectively
Hence the elements are removed in ascending or descending order
Efficiency O(nlog2n)
Merge Sort Idea
Take the array you would like to sort and divide it in half to create 2 unsorted subarrays
Next sort each of the 2 subarrays Finally merge the 2 sorted subarrays
into 1 sorted array Efficiency O(nlog2n)
Merge Sort
Merge Sort Although the merge step produces
a sorted array we have overlooked a very important step
How did we sort the 2 halves before performing the merge step
We used merge sort
Merge Sort By continually calling the merge
sort algorithm we eventually get a subarray of size 1
Since an array with only 1 element is clearly sorted we can back out and merge 2 arrays of size 1
Merge Sort
Merge Sort The basic merging algorithm
consists of 2 input arrays (arrayA and arrayB) An ouput array (arrayC) 3 position holders (indexA indexB
indexC) which are initially set to the beginning of their respective arrays
Merge Sort The smaller of arrayA[indexA] and
arrayB[indexB] is copied into arrayC[indexC] and the appropriate position holders are advanced
When either input list is exhausted the remainder of the other list is copied into arrayC
Merge Sort
1 13 24 26index
A
indexC
arrayA
2 15 27 38index
B
arrayB
arrayC
We compare arrayA[indexA]with arrayB[indexB] Whichever value is smaller isplaced into arrayC[indexC]
1 lt 2 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1index
C
arrayA
2 15 27 38index
B
arrayB
arrayC
2 lt 13 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
13 lt 15 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
15 lt 24 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13 15indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
24 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
indexA
1 2 13 15 24indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
26 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
1 2 13 15 24 26indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
Since we have exhaustedone of the arrays arrayAwe simply copy the remaining items from theother array arrayB intoarrayC
Merge Sort
1 13 24
26
1 2 13 15 24 26 27 38
arrayA
2 15 27 38arrayB
arrayC
Efficiency Summary
Sort Worst Case Average Case
Insertion O(n2) O(n2)
Shell O(n15) O(n125)
Selection O(n2) O(n2)
Bubble O(n2) O(n2)
Quick O(n2) O(nlog2n)
Heap O(nlog2n) O(nlog2n)
Merge O(nlog2n) O(nlog2n)
More Sorting
radix sort bucket sort in-place sorting how fast can we sort
Radix Sort
Unlike other sorting methods radix sort considers the structure of the keys
Assume keys are represented in a base M number system (M = radix) ie if M = 2 the keys are represented in binary
Sorting is done by comparing bits in the same position
Extension to keys that are alphanumeric strings
Radix Exchange Sort
Examine bits from left to right1 Sort array with respect to leftmost bit
2 Partition array
3Recursionrecursively sort top subarray ignoring leftmost bitrecursively sort bottom subarray ignoring leftmost bit
Time to sort n b-bit numbers O(b n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Partitioning at various levels
Best case II We cut the array size in half each time So the depth of the recursion in log2n At each level of the recursion all the
partitions at that level do work that is linear in n
O(log2n) O(n) = O(n log2n)
What about the worst case
Worst case In the worst case partitioning always
divides the size n array into these three parts A length one part containing the pivot itself A length zero part and A length n-1 part containing everything else
We donrsquot recur on the zero-length part Recurring on the length n-1 part requires
(in the worst case) recurring to depth n-1
Worst case partitioning
Worst case for quicksort In the worst case recursion may be
O(n) levels deep (for an array of size n)
But the partitioning work done at each level is still O(n)
O(n) O(n) = O(n2) So worst case for Quicksort is O(n2) When does this happen
When the array is sorted to begin with
Typical case for quicksort If the array is sorted to begin with
Quicksort is terrible O(n2) It is possible to construct other bad
cases However Quicksort is usually O(n log2n) The constants are so good that Quicksort
is generally the fastest algorithm known Most real-world sorting is done by
Quicksort
Tweaking Quicksort Almost anything you can try to
ldquoimproverdquo Quicksort will actually slow it down
One good tweak is to switch to a different sorting method when the subarrays get small (say 10 or 12) Quicksort has too much overhead for small
array sizes For large arrays it might be a good
idea to check beforehand if the array is already sorted But there is a better tweak than this
Randomized Quick-Sort Select the pivot as a random element of the sequence The expected running time of randomized quick-sort on a
sequence of size n is O(n log n) The time spent at a level of the quick-sort tree is O(n) We show that the expected height of the quick-sort tree is
O(log n) good vs bad pivots
bull The probability of a good pivot is 12 thus we expect k2 good pivots out of k pivotsbull After a good pivot the size of each child sequence is at most 34 the size of the parent sequencebull After h pivots we expect (34)h2 n elementsbull The expected height h of the quick-sort tree is at most 2 log43n
Median of three Obviously it doesnrsquot make sense to sort the
array in order to find the median to use as a pivot
Instead compare just three elements of our (sub)arraymdashthe first the last and the middle Take the median (middle value) of these three as
pivot Itrsquos possible (but not easy) to construct cases
which will make this technique O(n2) Suppose we rearrange (sort) these three
numbers so that the smallest is in the first position the largest in the last position and the other in the middle This lets us simplify and speed up the partition
loop
Final comments Weissrsquos code shows some additional
optimizations on pp 246-247 Weiss chooses to stop both searches on
equality to pivot This design decision is debatable
Quicksort is the fastest known general sorting algorithm on average
For optimum speed the pivot must be chosen carefully
ldquoMedian of threerdquo is a good technique for choosing the pivot
There will be some cases where Quicksort runs in O(n2) time
Quick Sort A couple of notes about quick sort There are more optimal ways to
choose the pivot value (such as the median-of-three method)
Also when the subarrays get small it becomes more efficient to use the insertion sort as opposed to continued use of quick sort
Bubble Sort vs Quick Sort If we calculate the Big-O notation
we find that (in the average case) Bubble Sort O(n2) Quick Sort O(nlog2n)
(quick-sort runs in time O(n2) in the worst case)
Heap Sort Idea take the items that need to be
sorted and insert them into the heap By calling deleteHeap we remove the
smallest or largest element depending on whether or not we are working with a min- or max-heap respectively
Hence the elements are removed in ascending or descending order
Efficiency O(nlog2n)
Merge Sort Idea
Take the array you would like to sort and divide it in half to create 2 unsorted subarrays
Next sort each of the 2 subarrays Finally merge the 2 sorted subarrays
into 1 sorted array Efficiency O(nlog2n)
Merge Sort
Merge Sort Although the merge step produces
a sorted array we have overlooked a very important step
How did we sort the 2 halves before performing the merge step
We used merge sort
Merge Sort By continually calling the merge
sort algorithm we eventually get a subarray of size 1
Since an array with only 1 element is clearly sorted we can back out and merge 2 arrays of size 1
Merge Sort
Merge Sort The basic merging algorithm
consists of 2 input arrays (arrayA and arrayB) An ouput array (arrayC) 3 position holders (indexA indexB
indexC) which are initially set to the beginning of their respective arrays
Merge Sort The smaller of arrayA[indexA] and
arrayB[indexB] is copied into arrayC[indexC] and the appropriate position holders are advanced
When either input list is exhausted the remainder of the other list is copied into arrayC
Merge Sort
1 13 24 26index
A
indexC
arrayA
2 15 27 38index
B
arrayB
arrayC
We compare arrayA[indexA]with arrayB[indexB] Whichever value is smaller isplaced into arrayC[indexC]
1 lt 2 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1index
C
arrayA
2 15 27 38index
B
arrayB
arrayC
2 lt 13 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
13 lt 15 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
15 lt 24 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13 15indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
24 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
indexA
1 2 13 15 24indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
26 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
1 2 13 15 24 26indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
Since we have exhaustedone of the arrays arrayAwe simply copy the remaining items from theother array arrayB intoarrayC
Merge Sort
1 13 24
26
1 2 13 15 24 26 27 38
arrayA
2 15 27 38arrayB
arrayC
Efficiency Summary
Sort Worst Case Average Case
Insertion O(n2) O(n2)
Shell O(n15) O(n125)
Selection O(n2) O(n2)
Bubble O(n2) O(n2)
Quick O(n2) O(nlog2n)
Heap O(nlog2n) O(nlog2n)
Merge O(nlog2n) O(nlog2n)
More Sorting
radix sort bucket sort in-place sorting how fast can we sort
Radix Sort
Unlike other sorting methods radix sort considers the structure of the keys
Assume keys are represented in a base M number system (M = radix) ie if M = 2 the keys are represented in binary
Sorting is done by comparing bits in the same position
Extension to keys that are alphanumeric strings
Radix Exchange Sort
Examine bits from left to right1 Sort array with respect to leftmost bit
2 Partition array
3Recursionrecursively sort top subarray ignoring leftmost bitrecursively sort bottom subarray ignoring leftmost bit
Time to sort n b-bit numbers O(b n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Best case II We cut the array size in half each time So the depth of the recursion in log2n At each level of the recursion all the
partitions at that level do work that is linear in n
O(log2n) O(n) = O(n log2n)
What about the worst case
Worst case In the worst case partitioning always
divides the size n array into these three parts A length one part containing the pivot itself A length zero part and A length n-1 part containing everything else
We donrsquot recur on the zero-length part Recurring on the length n-1 part requires
(in the worst case) recurring to depth n-1
Worst case partitioning
Worst case for quicksort In the worst case recursion may be
O(n) levels deep (for an array of size n)
But the partitioning work done at each level is still O(n)
O(n) O(n) = O(n2) So worst case for Quicksort is O(n2) When does this happen
When the array is sorted to begin with
Typical case for quicksort If the array is sorted to begin with
Quicksort is terrible O(n2) It is possible to construct other bad
cases However Quicksort is usually O(n log2n) The constants are so good that Quicksort
is generally the fastest algorithm known Most real-world sorting is done by
Quicksort
Tweaking Quicksort Almost anything you can try to
ldquoimproverdquo Quicksort will actually slow it down
One good tweak is to switch to a different sorting method when the subarrays get small (say 10 or 12) Quicksort has too much overhead for small
array sizes For large arrays it might be a good
idea to check beforehand if the array is already sorted But there is a better tweak than this
Randomized Quick-Sort Select the pivot as a random element of the sequence The expected running time of randomized quick-sort on a
sequence of size n is O(n log n) The time spent at a level of the quick-sort tree is O(n) We show that the expected height of the quick-sort tree is
O(log n) good vs bad pivots
bull The probability of a good pivot is 12 thus we expect k2 good pivots out of k pivotsbull After a good pivot the size of each child sequence is at most 34 the size of the parent sequencebull After h pivots we expect (34)h2 n elementsbull The expected height h of the quick-sort tree is at most 2 log43n
Median of three Obviously it doesnrsquot make sense to sort the
array in order to find the median to use as a pivot
Instead compare just three elements of our (sub)arraymdashthe first the last and the middle Take the median (middle value) of these three as
pivot Itrsquos possible (but not easy) to construct cases
which will make this technique O(n2) Suppose we rearrange (sort) these three
numbers so that the smallest is in the first position the largest in the last position and the other in the middle This lets us simplify and speed up the partition
loop
Final comments Weissrsquos code shows some additional
optimizations on pp 246-247 Weiss chooses to stop both searches on
equality to pivot This design decision is debatable
Quicksort is the fastest known general sorting algorithm on average
For optimum speed the pivot must be chosen carefully
ldquoMedian of threerdquo is a good technique for choosing the pivot
There will be some cases where Quicksort runs in O(n2) time
Quick Sort A couple of notes about quick sort There are more optimal ways to
choose the pivot value (such as the median-of-three method)
Also when the subarrays get small it becomes more efficient to use the insertion sort as opposed to continued use of quick sort
Bubble Sort vs Quick Sort If we calculate the Big-O notation
we find that (in the average case) Bubble Sort O(n2) Quick Sort O(nlog2n)
(quick-sort runs in time O(n2) in the worst case)
Heap Sort Idea take the items that need to be
sorted and insert them into the heap By calling deleteHeap we remove the
smallest or largest element depending on whether or not we are working with a min- or max-heap respectively
Hence the elements are removed in ascending or descending order
Efficiency O(nlog2n)
Merge Sort Idea
Take the array you would like to sort and divide it in half to create 2 unsorted subarrays
Next sort each of the 2 subarrays Finally merge the 2 sorted subarrays
into 1 sorted array Efficiency O(nlog2n)
Merge Sort
Merge Sort Although the merge step produces
a sorted array we have overlooked a very important step
How did we sort the 2 halves before performing the merge step
We used merge sort
Merge Sort By continually calling the merge
sort algorithm we eventually get a subarray of size 1
Since an array with only 1 element is clearly sorted we can back out and merge 2 arrays of size 1
Merge Sort
Merge Sort The basic merging algorithm
consists of 2 input arrays (arrayA and arrayB) An ouput array (arrayC) 3 position holders (indexA indexB
indexC) which are initially set to the beginning of their respective arrays
Merge Sort The smaller of arrayA[indexA] and
arrayB[indexB] is copied into arrayC[indexC] and the appropriate position holders are advanced
When either input list is exhausted the remainder of the other list is copied into arrayC
Merge Sort
1 13 24 26index
A
indexC
arrayA
2 15 27 38index
B
arrayB
arrayC
We compare arrayA[indexA]with arrayB[indexB] Whichever value is smaller isplaced into arrayC[indexC]
1 lt 2 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1index
C
arrayA
2 15 27 38index
B
arrayB
arrayC
2 lt 13 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
13 lt 15 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
15 lt 24 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13 15indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
24 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
indexA
1 2 13 15 24indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
26 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
1 2 13 15 24 26indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
Since we have exhaustedone of the arrays arrayAwe simply copy the remaining items from theother array arrayB intoarrayC
Merge Sort
1 13 24
26
1 2 13 15 24 26 27 38
arrayA
2 15 27 38arrayB
arrayC
Efficiency Summary
Sort Worst Case Average Case
Insertion O(n2) O(n2)
Shell O(n15) O(n125)
Selection O(n2) O(n2)
Bubble O(n2) O(n2)
Quick O(n2) O(nlog2n)
Heap O(nlog2n) O(nlog2n)
Merge O(nlog2n) O(nlog2n)
More Sorting
radix sort bucket sort in-place sorting how fast can we sort
Radix Sort
Unlike other sorting methods radix sort considers the structure of the keys
Assume keys are represented in a base M number system (M = radix) ie if M = 2 the keys are represented in binary
Sorting is done by comparing bits in the same position
Extension to keys that are alphanumeric strings
Radix Exchange Sort
Examine bits from left to right1 Sort array with respect to leftmost bit
2 Partition array
3Recursionrecursively sort top subarray ignoring leftmost bitrecursively sort bottom subarray ignoring leftmost bit
Time to sort n b-bit numbers O(b n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Worst case In the worst case partitioning always
divides the size n array into these three parts A length one part containing the pivot itself A length zero part and A length n-1 part containing everything else
We donrsquot recur on the zero-length part Recurring on the length n-1 part requires
(in the worst case) recurring to depth n-1
Worst case partitioning
Worst case for quicksort In the worst case recursion may be
O(n) levels deep (for an array of size n)
But the partitioning work done at each level is still O(n)
O(n) O(n) = O(n2) So worst case for Quicksort is O(n2) When does this happen
When the array is sorted to begin with
Typical case for quicksort If the array is sorted to begin with
Quicksort is terrible O(n2) It is possible to construct other bad
cases However Quicksort is usually O(n log2n) The constants are so good that Quicksort
is generally the fastest algorithm known Most real-world sorting is done by
Quicksort
Tweaking Quicksort Almost anything you can try to
ldquoimproverdquo Quicksort will actually slow it down
One good tweak is to switch to a different sorting method when the subarrays get small (say 10 or 12) Quicksort has too much overhead for small
array sizes For large arrays it might be a good
idea to check beforehand if the array is already sorted But there is a better tweak than this
Randomized Quick-Sort Select the pivot as a random element of the sequence The expected running time of randomized quick-sort on a
sequence of size n is O(n log n) The time spent at a level of the quick-sort tree is O(n) We show that the expected height of the quick-sort tree is
O(log n) good vs bad pivots
bull The probability of a good pivot is 12 thus we expect k2 good pivots out of k pivotsbull After a good pivot the size of each child sequence is at most 34 the size of the parent sequencebull After h pivots we expect (34)h2 n elementsbull The expected height h of the quick-sort tree is at most 2 log43n
Median of three Obviously it doesnrsquot make sense to sort the
array in order to find the median to use as a pivot
Instead compare just three elements of our (sub)arraymdashthe first the last and the middle Take the median (middle value) of these three as
pivot Itrsquos possible (but not easy) to construct cases
which will make this technique O(n2) Suppose we rearrange (sort) these three
numbers so that the smallest is in the first position the largest in the last position and the other in the middle This lets us simplify and speed up the partition
loop
Final comments Weissrsquos code shows some additional
optimizations on pp 246-247 Weiss chooses to stop both searches on
equality to pivot This design decision is debatable
Quicksort is the fastest known general sorting algorithm on average
For optimum speed the pivot must be chosen carefully
ldquoMedian of threerdquo is a good technique for choosing the pivot
There will be some cases where Quicksort runs in O(n2) time
Quick Sort A couple of notes about quick sort There are more optimal ways to
choose the pivot value (such as the median-of-three method)
Also when the subarrays get small it becomes more efficient to use the insertion sort as opposed to continued use of quick sort
Bubble Sort vs Quick Sort If we calculate the Big-O notation
we find that (in the average case) Bubble Sort O(n2) Quick Sort O(nlog2n)
(quick-sort runs in time O(n2) in the worst case)
Heap Sort Idea take the items that need to be
sorted and insert them into the heap By calling deleteHeap we remove the
smallest or largest element depending on whether or not we are working with a min- or max-heap respectively
Hence the elements are removed in ascending or descending order
Efficiency O(nlog2n)
Merge Sort Idea
Take the array you would like to sort and divide it in half to create 2 unsorted subarrays
Next sort each of the 2 subarrays Finally merge the 2 sorted subarrays
into 1 sorted array Efficiency O(nlog2n)
Merge Sort
Merge Sort Although the merge step produces
a sorted array we have overlooked a very important step
How did we sort the 2 halves before performing the merge step
We used merge sort
Merge Sort By continually calling the merge
sort algorithm we eventually get a subarray of size 1
Since an array with only 1 element is clearly sorted we can back out and merge 2 arrays of size 1
Merge Sort
Merge Sort The basic merging algorithm
consists of 2 input arrays (arrayA and arrayB) An ouput array (arrayC) 3 position holders (indexA indexB
indexC) which are initially set to the beginning of their respective arrays
Merge Sort The smaller of arrayA[indexA] and
arrayB[indexB] is copied into arrayC[indexC] and the appropriate position holders are advanced
When either input list is exhausted the remainder of the other list is copied into arrayC
Merge Sort
1 13 24 26index
A
indexC
arrayA
2 15 27 38index
B
arrayB
arrayC
We compare arrayA[indexA]with arrayB[indexB] Whichever value is smaller isplaced into arrayC[indexC]
1 lt 2 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1index
C
arrayA
2 15 27 38index
B
arrayB
arrayC
2 lt 13 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
13 lt 15 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
15 lt 24 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13 15indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
24 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
indexA
1 2 13 15 24indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
26 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
1 2 13 15 24 26indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
Since we have exhaustedone of the arrays arrayAwe simply copy the remaining items from theother array arrayB intoarrayC
Merge Sort
1 13 24
26
1 2 13 15 24 26 27 38
arrayA
2 15 27 38arrayB
arrayC
Efficiency Summary
Sort Worst Case Average Case
Insertion O(n2) O(n2)
Shell O(n15) O(n125)
Selection O(n2) O(n2)
Bubble O(n2) O(n2)
Quick O(n2) O(nlog2n)
Heap O(nlog2n) O(nlog2n)
Merge O(nlog2n) O(nlog2n)
More Sorting
radix sort bucket sort in-place sorting how fast can we sort
Radix Sort
Unlike other sorting methods radix sort considers the structure of the keys
Assume keys are represented in a base M number system (M = radix) ie if M = 2 the keys are represented in binary
Sorting is done by comparing bits in the same position
Extension to keys that are alphanumeric strings
Radix Exchange Sort
Examine bits from left to right1 Sort array with respect to leftmost bit
2 Partition array
3Recursionrecursively sort top subarray ignoring leftmost bitrecursively sort bottom subarray ignoring leftmost bit
Time to sort n b-bit numbers O(b n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Worst case partitioning
Worst case for quicksort In the worst case recursion may be
O(n) levels deep (for an array of size n)
But the partitioning work done at each level is still O(n)
O(n) O(n) = O(n2) So worst case for Quicksort is O(n2) When does this happen
When the array is sorted to begin with
Typical case for quicksort If the array is sorted to begin with
Quicksort is terrible O(n2) It is possible to construct other bad
cases However Quicksort is usually O(n log2n) The constants are so good that Quicksort
is generally the fastest algorithm known Most real-world sorting is done by
Quicksort
Tweaking Quicksort Almost anything you can try to
ldquoimproverdquo Quicksort will actually slow it down
One good tweak is to switch to a different sorting method when the subarrays get small (say 10 or 12) Quicksort has too much overhead for small
array sizes For large arrays it might be a good
idea to check beforehand if the array is already sorted But there is a better tweak than this
Randomized Quick-Sort Select the pivot as a random element of the sequence The expected running time of randomized quick-sort on a
sequence of size n is O(n log n) The time spent at a level of the quick-sort tree is O(n) We show that the expected height of the quick-sort tree is
O(log n) good vs bad pivots
bull The probability of a good pivot is 12 thus we expect k2 good pivots out of k pivotsbull After a good pivot the size of each child sequence is at most 34 the size of the parent sequencebull After h pivots we expect (34)h2 n elementsbull The expected height h of the quick-sort tree is at most 2 log43n
Median of three Obviously it doesnrsquot make sense to sort the
array in order to find the median to use as a pivot
Instead compare just three elements of our (sub)arraymdashthe first the last and the middle Take the median (middle value) of these three as
pivot Itrsquos possible (but not easy) to construct cases
which will make this technique O(n2) Suppose we rearrange (sort) these three
numbers so that the smallest is in the first position the largest in the last position and the other in the middle This lets us simplify and speed up the partition
loop
Final comments Weissrsquos code shows some additional
optimizations on pp 246-247 Weiss chooses to stop both searches on
equality to pivot This design decision is debatable
Quicksort is the fastest known general sorting algorithm on average
For optimum speed the pivot must be chosen carefully
ldquoMedian of threerdquo is a good technique for choosing the pivot
There will be some cases where Quicksort runs in O(n2) time
Quick Sort A couple of notes about quick sort There are more optimal ways to
choose the pivot value (such as the median-of-three method)
Also when the subarrays get small it becomes more efficient to use the insertion sort as opposed to continued use of quick sort
Bubble Sort vs Quick Sort If we calculate the Big-O notation
we find that (in the average case) Bubble Sort O(n2) Quick Sort O(nlog2n)
(quick-sort runs in time O(n2) in the worst case)
Heap Sort Idea take the items that need to be
sorted and insert them into the heap By calling deleteHeap we remove the
smallest or largest element depending on whether or not we are working with a min- or max-heap respectively
Hence the elements are removed in ascending or descending order
Efficiency O(nlog2n)
Merge Sort Idea
Take the array you would like to sort and divide it in half to create 2 unsorted subarrays
Next sort each of the 2 subarrays Finally merge the 2 sorted subarrays
into 1 sorted array Efficiency O(nlog2n)
Merge Sort
Merge Sort Although the merge step produces
a sorted array we have overlooked a very important step
How did we sort the 2 halves before performing the merge step
We used merge sort
Merge Sort By continually calling the merge
sort algorithm we eventually get a subarray of size 1
Since an array with only 1 element is clearly sorted we can back out and merge 2 arrays of size 1
Merge Sort
Merge Sort The basic merging algorithm
consists of 2 input arrays (arrayA and arrayB) An ouput array (arrayC) 3 position holders (indexA indexB
indexC) which are initially set to the beginning of their respective arrays
Merge Sort The smaller of arrayA[indexA] and
arrayB[indexB] is copied into arrayC[indexC] and the appropriate position holders are advanced
When either input list is exhausted the remainder of the other list is copied into arrayC
Merge Sort
1 13 24 26index
A
indexC
arrayA
2 15 27 38index
B
arrayB
arrayC
We compare arrayA[indexA]with arrayB[indexB] Whichever value is smaller isplaced into arrayC[indexC]
1 lt 2 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1index
C
arrayA
2 15 27 38index
B
arrayB
arrayC
2 lt 13 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
13 lt 15 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
15 lt 24 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13 15indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
24 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
indexA
1 2 13 15 24indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
26 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
1 2 13 15 24 26indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
Since we have exhaustedone of the arrays arrayAwe simply copy the remaining items from theother array arrayB intoarrayC
Merge Sort
1 13 24
26
1 2 13 15 24 26 27 38
arrayA
2 15 27 38arrayB
arrayC
Efficiency Summary
Sort Worst Case Average Case
Insertion O(n2) O(n2)
Shell O(n15) O(n125)
Selection O(n2) O(n2)
Bubble O(n2) O(n2)
Quick O(n2) O(nlog2n)
Heap O(nlog2n) O(nlog2n)
Merge O(nlog2n) O(nlog2n)
More Sorting
radix sort bucket sort in-place sorting how fast can we sort
Radix Sort
Unlike other sorting methods radix sort considers the structure of the keys
Assume keys are represented in a base M number system (M = radix) ie if M = 2 the keys are represented in binary
Sorting is done by comparing bits in the same position
Extension to keys that are alphanumeric strings
Radix Exchange Sort
Examine bits from left to right1 Sort array with respect to leftmost bit
2 Partition array
3Recursionrecursively sort top subarray ignoring leftmost bitrecursively sort bottom subarray ignoring leftmost bit
Time to sort n b-bit numbers O(b n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Worst case for quicksort In the worst case recursion may be
O(n) levels deep (for an array of size n)
But the partitioning work done at each level is still O(n)
O(n) O(n) = O(n2) So worst case for Quicksort is O(n2) When does this happen
When the array is sorted to begin with
Typical case for quicksort If the array is sorted to begin with
Quicksort is terrible O(n2) It is possible to construct other bad
cases However Quicksort is usually O(n log2n) The constants are so good that Quicksort
is generally the fastest algorithm known Most real-world sorting is done by
Quicksort
Tweaking Quicksort Almost anything you can try to
ldquoimproverdquo Quicksort will actually slow it down
One good tweak is to switch to a different sorting method when the subarrays get small (say 10 or 12) Quicksort has too much overhead for small
array sizes For large arrays it might be a good
idea to check beforehand if the array is already sorted But there is a better tweak than this
Randomized Quick-Sort Select the pivot as a random element of the sequence The expected running time of randomized quick-sort on a
sequence of size n is O(n log n) The time spent at a level of the quick-sort tree is O(n) We show that the expected height of the quick-sort tree is
O(log n) good vs bad pivots
bull The probability of a good pivot is 12 thus we expect k2 good pivots out of k pivotsbull After a good pivot the size of each child sequence is at most 34 the size of the parent sequencebull After h pivots we expect (34)h2 n elementsbull The expected height h of the quick-sort tree is at most 2 log43n
Median of three Obviously it doesnrsquot make sense to sort the
array in order to find the median to use as a pivot
Instead compare just three elements of our (sub)arraymdashthe first the last and the middle Take the median (middle value) of these three as
pivot Itrsquos possible (but not easy) to construct cases
which will make this technique O(n2) Suppose we rearrange (sort) these three
numbers so that the smallest is in the first position the largest in the last position and the other in the middle This lets us simplify and speed up the partition
loop
Final comments Weissrsquos code shows some additional
optimizations on pp 246-247 Weiss chooses to stop both searches on
equality to pivot This design decision is debatable
Quicksort is the fastest known general sorting algorithm on average
For optimum speed the pivot must be chosen carefully
ldquoMedian of threerdquo is a good technique for choosing the pivot
There will be some cases where Quicksort runs in O(n2) time
Quick Sort A couple of notes about quick sort There are more optimal ways to
choose the pivot value (such as the median-of-three method)
Also when the subarrays get small it becomes more efficient to use the insertion sort as opposed to continued use of quick sort
Bubble Sort vs Quick Sort If we calculate the Big-O notation
we find that (in the average case) Bubble Sort O(n2) Quick Sort O(nlog2n)
(quick-sort runs in time O(n2) in the worst case)
Heap Sort Idea take the items that need to be
sorted and insert them into the heap By calling deleteHeap we remove the
smallest or largest element depending on whether or not we are working with a min- or max-heap respectively
Hence the elements are removed in ascending or descending order
Efficiency O(nlog2n)
Merge Sort Idea
Take the array you would like to sort and divide it in half to create 2 unsorted subarrays
Next sort each of the 2 subarrays Finally merge the 2 sorted subarrays
into 1 sorted array Efficiency O(nlog2n)
Merge Sort
Merge Sort Although the merge step produces
a sorted array we have overlooked a very important step
How did we sort the 2 halves before performing the merge step
We used merge sort
Merge Sort By continually calling the merge
sort algorithm we eventually get a subarray of size 1
Since an array with only 1 element is clearly sorted we can back out and merge 2 arrays of size 1
Merge Sort
Merge Sort The basic merging algorithm
consists of 2 input arrays (arrayA and arrayB) An ouput array (arrayC) 3 position holders (indexA indexB
indexC) which are initially set to the beginning of their respective arrays
Merge Sort The smaller of arrayA[indexA] and
arrayB[indexB] is copied into arrayC[indexC] and the appropriate position holders are advanced
When either input list is exhausted the remainder of the other list is copied into arrayC
Merge Sort
1 13 24 26index
A
indexC
arrayA
2 15 27 38index
B
arrayB
arrayC
We compare arrayA[indexA]with arrayB[indexB] Whichever value is smaller isplaced into arrayC[indexC]
1 lt 2 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1index
C
arrayA
2 15 27 38index
B
arrayB
arrayC
2 lt 13 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
13 lt 15 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
15 lt 24 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13 15indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
24 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
indexA
1 2 13 15 24indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
26 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
1 2 13 15 24 26indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
Since we have exhaustedone of the arrays arrayAwe simply copy the remaining items from theother array arrayB intoarrayC
Merge Sort
1 13 24
26
1 2 13 15 24 26 27 38
arrayA
2 15 27 38arrayB
arrayC
Efficiency Summary
Sort Worst Case Average Case
Insertion O(n2) O(n2)
Shell O(n15) O(n125)
Selection O(n2) O(n2)
Bubble O(n2) O(n2)
Quick O(n2) O(nlog2n)
Heap O(nlog2n) O(nlog2n)
Merge O(nlog2n) O(nlog2n)
More Sorting
radix sort bucket sort in-place sorting how fast can we sort
Radix Sort
Unlike other sorting methods radix sort considers the structure of the keys
Assume keys are represented in a base M number system (M = radix) ie if M = 2 the keys are represented in binary
Sorting is done by comparing bits in the same position
Extension to keys that are alphanumeric strings
Radix Exchange Sort
Examine bits from left to right1 Sort array with respect to leftmost bit
2 Partition array
3Recursionrecursively sort top subarray ignoring leftmost bitrecursively sort bottom subarray ignoring leftmost bit
Time to sort n b-bit numbers O(b n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Typical case for quicksort If the array is sorted to begin with
Quicksort is terrible O(n2) It is possible to construct other bad
cases However Quicksort is usually O(n log2n) The constants are so good that Quicksort
is generally the fastest algorithm known Most real-world sorting is done by
Quicksort
Tweaking Quicksort Almost anything you can try to
ldquoimproverdquo Quicksort will actually slow it down
One good tweak is to switch to a different sorting method when the subarrays get small (say 10 or 12) Quicksort has too much overhead for small
array sizes For large arrays it might be a good
idea to check beforehand if the array is already sorted But there is a better tweak than this
Randomized Quick-Sort Select the pivot as a random element of the sequence The expected running time of randomized quick-sort on a
sequence of size n is O(n log n) The time spent at a level of the quick-sort tree is O(n) We show that the expected height of the quick-sort tree is
O(log n) good vs bad pivots
bull The probability of a good pivot is 12 thus we expect k2 good pivots out of k pivotsbull After a good pivot the size of each child sequence is at most 34 the size of the parent sequencebull After h pivots we expect (34)h2 n elementsbull The expected height h of the quick-sort tree is at most 2 log43n
Median of three Obviously it doesnrsquot make sense to sort the
array in order to find the median to use as a pivot
Instead compare just three elements of our (sub)arraymdashthe first the last and the middle Take the median (middle value) of these three as
pivot Itrsquos possible (but not easy) to construct cases
which will make this technique O(n2) Suppose we rearrange (sort) these three
numbers so that the smallest is in the first position the largest in the last position and the other in the middle This lets us simplify and speed up the partition
loop
Final comments Weissrsquos code shows some additional
optimizations on pp 246-247 Weiss chooses to stop both searches on
equality to pivot This design decision is debatable
Quicksort is the fastest known general sorting algorithm on average
For optimum speed the pivot must be chosen carefully
ldquoMedian of threerdquo is a good technique for choosing the pivot
There will be some cases where Quicksort runs in O(n2) time
Quick Sort A couple of notes about quick sort There are more optimal ways to
choose the pivot value (such as the median-of-three method)
Also when the subarrays get small it becomes more efficient to use the insertion sort as opposed to continued use of quick sort
Bubble Sort vs Quick Sort If we calculate the Big-O notation
we find that (in the average case) Bubble Sort O(n2) Quick Sort O(nlog2n)
(quick-sort runs in time O(n2) in the worst case)
Heap Sort Idea take the items that need to be
sorted and insert them into the heap By calling deleteHeap we remove the
smallest or largest element depending on whether or not we are working with a min- or max-heap respectively
Hence the elements are removed in ascending or descending order
Efficiency O(nlog2n)
Merge Sort Idea
Take the array you would like to sort and divide it in half to create 2 unsorted subarrays
Next sort each of the 2 subarrays Finally merge the 2 sorted subarrays
into 1 sorted array Efficiency O(nlog2n)
Merge Sort
Merge Sort Although the merge step produces
a sorted array we have overlooked a very important step
How did we sort the 2 halves before performing the merge step
We used merge sort
Merge Sort By continually calling the merge
sort algorithm we eventually get a subarray of size 1
Since an array with only 1 element is clearly sorted we can back out and merge 2 arrays of size 1
Merge Sort
Merge Sort The basic merging algorithm
consists of 2 input arrays (arrayA and arrayB) An ouput array (arrayC) 3 position holders (indexA indexB
indexC) which are initially set to the beginning of their respective arrays
Merge Sort The smaller of arrayA[indexA] and
arrayB[indexB] is copied into arrayC[indexC] and the appropriate position holders are advanced
When either input list is exhausted the remainder of the other list is copied into arrayC
Merge Sort
1 13 24 26index
A
indexC
arrayA
2 15 27 38index
B
arrayB
arrayC
We compare arrayA[indexA]with arrayB[indexB] Whichever value is smaller isplaced into arrayC[indexC]
1 lt 2 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1index
C
arrayA
2 15 27 38index
B
arrayB
arrayC
2 lt 13 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
13 lt 15 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
15 lt 24 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13 15indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
24 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
indexA
1 2 13 15 24indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
26 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
1 2 13 15 24 26indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
Since we have exhaustedone of the arrays arrayAwe simply copy the remaining items from theother array arrayB intoarrayC
Merge Sort
1 13 24
26
1 2 13 15 24 26 27 38
arrayA
2 15 27 38arrayB
arrayC
Efficiency Summary
Sort Worst Case Average Case
Insertion O(n2) O(n2)
Shell O(n15) O(n125)
Selection O(n2) O(n2)
Bubble O(n2) O(n2)
Quick O(n2) O(nlog2n)
Heap O(nlog2n) O(nlog2n)
Merge O(nlog2n) O(nlog2n)
More Sorting
radix sort bucket sort in-place sorting how fast can we sort
Radix Sort
Unlike other sorting methods radix sort considers the structure of the keys
Assume keys are represented in a base M number system (M = radix) ie if M = 2 the keys are represented in binary
Sorting is done by comparing bits in the same position
Extension to keys that are alphanumeric strings
Radix Exchange Sort
Examine bits from left to right1 Sort array with respect to leftmost bit
2 Partition array
3Recursionrecursively sort top subarray ignoring leftmost bitrecursively sort bottom subarray ignoring leftmost bit
Time to sort n b-bit numbers O(b n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Tweaking Quicksort Almost anything you can try to
ldquoimproverdquo Quicksort will actually slow it down
One good tweak is to switch to a different sorting method when the subarrays get small (say 10 or 12) Quicksort has too much overhead for small
array sizes For large arrays it might be a good
idea to check beforehand if the array is already sorted But there is a better tweak than this
Randomized Quick-Sort Select the pivot as a random element of the sequence The expected running time of randomized quick-sort on a
sequence of size n is O(n log n) The time spent at a level of the quick-sort tree is O(n) We show that the expected height of the quick-sort tree is
O(log n) good vs bad pivots
bull The probability of a good pivot is 12 thus we expect k2 good pivots out of k pivotsbull After a good pivot the size of each child sequence is at most 34 the size of the parent sequencebull After h pivots we expect (34)h2 n elementsbull The expected height h of the quick-sort tree is at most 2 log43n
Median of three Obviously it doesnrsquot make sense to sort the
array in order to find the median to use as a pivot
Instead compare just three elements of our (sub)arraymdashthe first the last and the middle Take the median (middle value) of these three as
pivot Itrsquos possible (but not easy) to construct cases
which will make this technique O(n2) Suppose we rearrange (sort) these three
numbers so that the smallest is in the first position the largest in the last position and the other in the middle This lets us simplify and speed up the partition
loop
Final comments Weissrsquos code shows some additional
optimizations on pp 246-247 Weiss chooses to stop both searches on
equality to pivot This design decision is debatable
Quicksort is the fastest known general sorting algorithm on average
For optimum speed the pivot must be chosen carefully
ldquoMedian of threerdquo is a good technique for choosing the pivot
There will be some cases where Quicksort runs in O(n2) time
Quick Sort A couple of notes about quick sort There are more optimal ways to
choose the pivot value (such as the median-of-three method)
Also when the subarrays get small it becomes more efficient to use the insertion sort as opposed to continued use of quick sort
Bubble Sort vs Quick Sort If we calculate the Big-O notation
we find that (in the average case) Bubble Sort O(n2) Quick Sort O(nlog2n)
(quick-sort runs in time O(n2) in the worst case)
Heap Sort Idea take the items that need to be
sorted and insert them into the heap By calling deleteHeap we remove the
smallest or largest element depending on whether or not we are working with a min- or max-heap respectively
Hence the elements are removed in ascending or descending order
Efficiency O(nlog2n)
Merge Sort Idea
Take the array you would like to sort and divide it in half to create 2 unsorted subarrays
Next sort each of the 2 subarrays Finally merge the 2 sorted subarrays
into 1 sorted array Efficiency O(nlog2n)
Merge Sort
Merge Sort Although the merge step produces
a sorted array we have overlooked a very important step
How did we sort the 2 halves before performing the merge step
We used merge sort
Merge Sort By continually calling the merge
sort algorithm we eventually get a subarray of size 1
Since an array with only 1 element is clearly sorted we can back out and merge 2 arrays of size 1
Merge Sort
Merge Sort The basic merging algorithm
consists of 2 input arrays (arrayA and arrayB) An ouput array (arrayC) 3 position holders (indexA indexB
indexC) which are initially set to the beginning of their respective arrays
Merge Sort The smaller of arrayA[indexA] and
arrayB[indexB] is copied into arrayC[indexC] and the appropriate position holders are advanced
When either input list is exhausted the remainder of the other list is copied into arrayC
Merge Sort
1 13 24 26index
A
indexC
arrayA
2 15 27 38index
B
arrayB
arrayC
We compare arrayA[indexA]with arrayB[indexB] Whichever value is smaller isplaced into arrayC[indexC]
1 lt 2 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1index
C
arrayA
2 15 27 38index
B
arrayB
arrayC
2 lt 13 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
13 lt 15 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
15 lt 24 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13 15indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
24 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
indexA
1 2 13 15 24indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
26 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
1 2 13 15 24 26indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
Since we have exhaustedone of the arrays arrayAwe simply copy the remaining items from theother array arrayB intoarrayC
Merge Sort
1 13 24
26
1 2 13 15 24 26 27 38
arrayA
2 15 27 38arrayB
arrayC
Efficiency Summary
Sort Worst Case Average Case
Insertion O(n2) O(n2)
Shell O(n15) O(n125)
Selection O(n2) O(n2)
Bubble O(n2) O(n2)
Quick O(n2) O(nlog2n)
Heap O(nlog2n) O(nlog2n)
Merge O(nlog2n) O(nlog2n)
More Sorting
radix sort bucket sort in-place sorting how fast can we sort
Radix Sort
Unlike other sorting methods radix sort considers the structure of the keys
Assume keys are represented in a base M number system (M = radix) ie if M = 2 the keys are represented in binary
Sorting is done by comparing bits in the same position
Extension to keys that are alphanumeric strings
Radix Exchange Sort
Examine bits from left to right1 Sort array with respect to leftmost bit
2 Partition array
3Recursionrecursively sort top subarray ignoring leftmost bitrecursively sort bottom subarray ignoring leftmost bit
Time to sort n b-bit numbers O(b n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Randomized Quick-Sort Select the pivot as a random element of the sequence The expected running time of randomized quick-sort on a
sequence of size n is O(n log n) The time spent at a level of the quick-sort tree is O(n) We show that the expected height of the quick-sort tree is
O(log n) good vs bad pivots
bull The probability of a good pivot is 12 thus we expect k2 good pivots out of k pivotsbull After a good pivot the size of each child sequence is at most 34 the size of the parent sequencebull After h pivots we expect (34)h2 n elementsbull The expected height h of the quick-sort tree is at most 2 log43n
Median of three Obviously it doesnrsquot make sense to sort the
array in order to find the median to use as a pivot
Instead compare just three elements of our (sub)arraymdashthe first the last and the middle Take the median (middle value) of these three as
pivot Itrsquos possible (but not easy) to construct cases
which will make this technique O(n2) Suppose we rearrange (sort) these three
numbers so that the smallest is in the first position the largest in the last position and the other in the middle This lets us simplify and speed up the partition
loop
Final comments Weissrsquos code shows some additional
optimizations on pp 246-247 Weiss chooses to stop both searches on
equality to pivot This design decision is debatable
Quicksort is the fastest known general sorting algorithm on average
For optimum speed the pivot must be chosen carefully
ldquoMedian of threerdquo is a good technique for choosing the pivot
There will be some cases where Quicksort runs in O(n2) time
Quick Sort A couple of notes about quick sort There are more optimal ways to
choose the pivot value (such as the median-of-three method)
Also when the subarrays get small it becomes more efficient to use the insertion sort as opposed to continued use of quick sort
Bubble Sort vs Quick Sort If we calculate the Big-O notation
we find that (in the average case) Bubble Sort O(n2) Quick Sort O(nlog2n)
(quick-sort runs in time O(n2) in the worst case)
Heap Sort Idea take the items that need to be
sorted and insert them into the heap By calling deleteHeap we remove the
smallest or largest element depending on whether or not we are working with a min- or max-heap respectively
Hence the elements are removed in ascending or descending order
Efficiency O(nlog2n)
Merge Sort Idea
Take the array you would like to sort and divide it in half to create 2 unsorted subarrays
Next sort each of the 2 subarrays Finally merge the 2 sorted subarrays
into 1 sorted array Efficiency O(nlog2n)
Merge Sort
Merge Sort Although the merge step produces
a sorted array we have overlooked a very important step
How did we sort the 2 halves before performing the merge step
We used merge sort
Merge Sort By continually calling the merge
sort algorithm we eventually get a subarray of size 1
Since an array with only 1 element is clearly sorted we can back out and merge 2 arrays of size 1
Merge Sort
Merge Sort The basic merging algorithm
consists of 2 input arrays (arrayA and arrayB) An ouput array (arrayC) 3 position holders (indexA indexB
indexC) which are initially set to the beginning of their respective arrays
Merge Sort The smaller of arrayA[indexA] and
arrayB[indexB] is copied into arrayC[indexC] and the appropriate position holders are advanced
When either input list is exhausted the remainder of the other list is copied into arrayC
Merge Sort
1 13 24 26index
A
indexC
arrayA
2 15 27 38index
B
arrayB
arrayC
We compare arrayA[indexA]with arrayB[indexB] Whichever value is smaller isplaced into arrayC[indexC]
1 lt 2 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1index
C
arrayA
2 15 27 38index
B
arrayB
arrayC
2 lt 13 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
13 lt 15 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
15 lt 24 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13 15indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
24 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
indexA
1 2 13 15 24indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
26 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
1 2 13 15 24 26indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
Since we have exhaustedone of the arrays arrayAwe simply copy the remaining items from theother array arrayB intoarrayC
Merge Sort
1 13 24
26
1 2 13 15 24 26 27 38
arrayA
2 15 27 38arrayB
arrayC
Efficiency Summary
Sort Worst Case Average Case
Insertion O(n2) O(n2)
Shell O(n15) O(n125)
Selection O(n2) O(n2)
Bubble O(n2) O(n2)
Quick O(n2) O(nlog2n)
Heap O(nlog2n) O(nlog2n)
Merge O(nlog2n) O(nlog2n)
More Sorting
radix sort bucket sort in-place sorting how fast can we sort
Radix Sort
Unlike other sorting methods radix sort considers the structure of the keys
Assume keys are represented in a base M number system (M = radix) ie if M = 2 the keys are represented in binary
Sorting is done by comparing bits in the same position
Extension to keys that are alphanumeric strings
Radix Exchange Sort
Examine bits from left to right1 Sort array with respect to leftmost bit
2 Partition array
3Recursionrecursively sort top subarray ignoring leftmost bitrecursively sort bottom subarray ignoring leftmost bit
Time to sort n b-bit numbers O(b n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Median of three Obviously it doesnrsquot make sense to sort the
array in order to find the median to use as a pivot
Instead compare just three elements of our (sub)arraymdashthe first the last and the middle Take the median (middle value) of these three as
pivot Itrsquos possible (but not easy) to construct cases
which will make this technique O(n2) Suppose we rearrange (sort) these three
numbers so that the smallest is in the first position the largest in the last position and the other in the middle This lets us simplify and speed up the partition
loop
Final comments Weissrsquos code shows some additional
optimizations on pp 246-247 Weiss chooses to stop both searches on
equality to pivot This design decision is debatable
Quicksort is the fastest known general sorting algorithm on average
For optimum speed the pivot must be chosen carefully
ldquoMedian of threerdquo is a good technique for choosing the pivot
There will be some cases where Quicksort runs in O(n2) time
Quick Sort A couple of notes about quick sort There are more optimal ways to
choose the pivot value (such as the median-of-three method)
Also when the subarrays get small it becomes more efficient to use the insertion sort as opposed to continued use of quick sort
Bubble Sort vs Quick Sort If we calculate the Big-O notation
we find that (in the average case) Bubble Sort O(n2) Quick Sort O(nlog2n)
(quick-sort runs in time O(n2) in the worst case)
Heap Sort Idea take the items that need to be
sorted and insert them into the heap By calling deleteHeap we remove the
smallest or largest element depending on whether or not we are working with a min- or max-heap respectively
Hence the elements are removed in ascending or descending order
Efficiency O(nlog2n)
Merge Sort Idea
Take the array you would like to sort and divide it in half to create 2 unsorted subarrays
Next sort each of the 2 subarrays Finally merge the 2 sorted subarrays
into 1 sorted array Efficiency O(nlog2n)
Merge Sort
Merge Sort Although the merge step produces
a sorted array we have overlooked a very important step
How did we sort the 2 halves before performing the merge step
We used merge sort
Merge Sort By continually calling the merge
sort algorithm we eventually get a subarray of size 1
Since an array with only 1 element is clearly sorted we can back out and merge 2 arrays of size 1
Merge Sort
Merge Sort The basic merging algorithm
consists of 2 input arrays (arrayA and arrayB) An ouput array (arrayC) 3 position holders (indexA indexB
indexC) which are initially set to the beginning of their respective arrays
Merge Sort The smaller of arrayA[indexA] and
arrayB[indexB] is copied into arrayC[indexC] and the appropriate position holders are advanced
When either input list is exhausted the remainder of the other list is copied into arrayC
Merge Sort
1 13 24 26index
A
indexC
arrayA
2 15 27 38index
B
arrayB
arrayC
We compare arrayA[indexA]with arrayB[indexB] Whichever value is smaller isplaced into arrayC[indexC]
1 lt 2 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1index
C
arrayA
2 15 27 38index
B
arrayB
arrayC
2 lt 13 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
13 lt 15 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
15 lt 24 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13 15indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
24 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
indexA
1 2 13 15 24indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
26 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
1 2 13 15 24 26indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
Since we have exhaustedone of the arrays arrayAwe simply copy the remaining items from theother array arrayB intoarrayC
Merge Sort
1 13 24
26
1 2 13 15 24 26 27 38
arrayA
2 15 27 38arrayB
arrayC
Efficiency Summary
Sort Worst Case Average Case
Insertion O(n2) O(n2)
Shell O(n15) O(n125)
Selection O(n2) O(n2)
Bubble O(n2) O(n2)
Quick O(n2) O(nlog2n)
Heap O(nlog2n) O(nlog2n)
Merge O(nlog2n) O(nlog2n)
More Sorting
radix sort bucket sort in-place sorting how fast can we sort
Radix Sort
Unlike other sorting methods radix sort considers the structure of the keys
Assume keys are represented in a base M number system (M = radix) ie if M = 2 the keys are represented in binary
Sorting is done by comparing bits in the same position
Extension to keys that are alphanumeric strings
Radix Exchange Sort
Examine bits from left to right1 Sort array with respect to leftmost bit
2 Partition array
3Recursionrecursively sort top subarray ignoring leftmost bitrecursively sort bottom subarray ignoring leftmost bit
Time to sort n b-bit numbers O(b n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Final comments Weissrsquos code shows some additional
optimizations on pp 246-247 Weiss chooses to stop both searches on
equality to pivot This design decision is debatable
Quicksort is the fastest known general sorting algorithm on average
For optimum speed the pivot must be chosen carefully
ldquoMedian of threerdquo is a good technique for choosing the pivot
There will be some cases where Quicksort runs in O(n2) time
Quick Sort A couple of notes about quick sort There are more optimal ways to
choose the pivot value (such as the median-of-three method)
Also when the subarrays get small it becomes more efficient to use the insertion sort as opposed to continued use of quick sort
Bubble Sort vs Quick Sort If we calculate the Big-O notation
we find that (in the average case) Bubble Sort O(n2) Quick Sort O(nlog2n)
(quick-sort runs in time O(n2) in the worst case)
Heap Sort Idea take the items that need to be
sorted and insert them into the heap By calling deleteHeap we remove the
smallest or largest element depending on whether or not we are working with a min- or max-heap respectively
Hence the elements are removed in ascending or descending order
Efficiency O(nlog2n)
Merge Sort Idea
Take the array you would like to sort and divide it in half to create 2 unsorted subarrays
Next sort each of the 2 subarrays Finally merge the 2 sorted subarrays
into 1 sorted array Efficiency O(nlog2n)
Merge Sort
Merge Sort Although the merge step produces
a sorted array we have overlooked a very important step
How did we sort the 2 halves before performing the merge step
We used merge sort
Merge Sort By continually calling the merge
sort algorithm we eventually get a subarray of size 1
Since an array with only 1 element is clearly sorted we can back out and merge 2 arrays of size 1
Merge Sort
Merge Sort The basic merging algorithm
consists of 2 input arrays (arrayA and arrayB) An ouput array (arrayC) 3 position holders (indexA indexB
indexC) which are initially set to the beginning of their respective arrays
Merge Sort The smaller of arrayA[indexA] and
arrayB[indexB] is copied into arrayC[indexC] and the appropriate position holders are advanced
When either input list is exhausted the remainder of the other list is copied into arrayC
Merge Sort
1 13 24 26index
A
indexC
arrayA
2 15 27 38index
B
arrayB
arrayC
We compare arrayA[indexA]with arrayB[indexB] Whichever value is smaller isplaced into arrayC[indexC]
1 lt 2 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1index
C
arrayA
2 15 27 38index
B
arrayB
arrayC
2 lt 13 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
13 lt 15 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
15 lt 24 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13 15indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
24 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
indexA
1 2 13 15 24indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
26 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
1 2 13 15 24 26indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
Since we have exhaustedone of the arrays arrayAwe simply copy the remaining items from theother array arrayB intoarrayC
Merge Sort
1 13 24
26
1 2 13 15 24 26 27 38
arrayA
2 15 27 38arrayB
arrayC
Efficiency Summary
Sort Worst Case Average Case
Insertion O(n2) O(n2)
Shell O(n15) O(n125)
Selection O(n2) O(n2)
Bubble O(n2) O(n2)
Quick O(n2) O(nlog2n)
Heap O(nlog2n) O(nlog2n)
Merge O(nlog2n) O(nlog2n)
More Sorting
radix sort bucket sort in-place sorting how fast can we sort
Radix Sort
Unlike other sorting methods radix sort considers the structure of the keys
Assume keys are represented in a base M number system (M = radix) ie if M = 2 the keys are represented in binary
Sorting is done by comparing bits in the same position
Extension to keys that are alphanumeric strings
Radix Exchange Sort
Examine bits from left to right1 Sort array with respect to leftmost bit
2 Partition array
3Recursionrecursively sort top subarray ignoring leftmost bitrecursively sort bottom subarray ignoring leftmost bit
Time to sort n b-bit numbers O(b n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Quick Sort A couple of notes about quick sort There are more optimal ways to
choose the pivot value (such as the median-of-three method)
Also when the subarrays get small it becomes more efficient to use the insertion sort as opposed to continued use of quick sort
Bubble Sort vs Quick Sort If we calculate the Big-O notation
we find that (in the average case) Bubble Sort O(n2) Quick Sort O(nlog2n)
(quick-sort runs in time O(n2) in the worst case)
Heap Sort Idea take the items that need to be
sorted and insert them into the heap By calling deleteHeap we remove the
smallest or largest element depending on whether or not we are working with a min- or max-heap respectively
Hence the elements are removed in ascending or descending order
Efficiency O(nlog2n)
Merge Sort Idea
Take the array you would like to sort and divide it in half to create 2 unsorted subarrays
Next sort each of the 2 subarrays Finally merge the 2 sorted subarrays
into 1 sorted array Efficiency O(nlog2n)
Merge Sort
Merge Sort Although the merge step produces
a sorted array we have overlooked a very important step
How did we sort the 2 halves before performing the merge step
We used merge sort
Merge Sort By continually calling the merge
sort algorithm we eventually get a subarray of size 1
Since an array with only 1 element is clearly sorted we can back out and merge 2 arrays of size 1
Merge Sort
Merge Sort The basic merging algorithm
consists of 2 input arrays (arrayA and arrayB) An ouput array (arrayC) 3 position holders (indexA indexB
indexC) which are initially set to the beginning of their respective arrays
Merge Sort The smaller of arrayA[indexA] and
arrayB[indexB] is copied into arrayC[indexC] and the appropriate position holders are advanced
When either input list is exhausted the remainder of the other list is copied into arrayC
Merge Sort
1 13 24 26index
A
indexC
arrayA
2 15 27 38index
B
arrayB
arrayC
We compare arrayA[indexA]with arrayB[indexB] Whichever value is smaller isplaced into arrayC[indexC]
1 lt 2 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1index
C
arrayA
2 15 27 38index
B
arrayB
arrayC
2 lt 13 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
13 lt 15 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
15 lt 24 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13 15indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
24 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
indexA
1 2 13 15 24indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
26 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
1 2 13 15 24 26indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
Since we have exhaustedone of the arrays arrayAwe simply copy the remaining items from theother array arrayB intoarrayC
Merge Sort
1 13 24
26
1 2 13 15 24 26 27 38
arrayA
2 15 27 38arrayB
arrayC
Efficiency Summary
Sort Worst Case Average Case
Insertion O(n2) O(n2)
Shell O(n15) O(n125)
Selection O(n2) O(n2)
Bubble O(n2) O(n2)
Quick O(n2) O(nlog2n)
Heap O(nlog2n) O(nlog2n)
Merge O(nlog2n) O(nlog2n)
More Sorting
radix sort bucket sort in-place sorting how fast can we sort
Radix Sort
Unlike other sorting methods radix sort considers the structure of the keys
Assume keys are represented in a base M number system (M = radix) ie if M = 2 the keys are represented in binary
Sorting is done by comparing bits in the same position
Extension to keys that are alphanumeric strings
Radix Exchange Sort
Examine bits from left to right1 Sort array with respect to leftmost bit
2 Partition array
3Recursionrecursively sort top subarray ignoring leftmost bitrecursively sort bottom subarray ignoring leftmost bit
Time to sort n b-bit numbers O(b n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Bubble Sort vs Quick Sort If we calculate the Big-O notation
we find that (in the average case) Bubble Sort O(n2) Quick Sort O(nlog2n)
(quick-sort runs in time O(n2) in the worst case)
Heap Sort Idea take the items that need to be
sorted and insert them into the heap By calling deleteHeap we remove the
smallest or largest element depending on whether or not we are working with a min- or max-heap respectively
Hence the elements are removed in ascending or descending order
Efficiency O(nlog2n)
Merge Sort Idea
Take the array you would like to sort and divide it in half to create 2 unsorted subarrays
Next sort each of the 2 subarrays Finally merge the 2 sorted subarrays
into 1 sorted array Efficiency O(nlog2n)
Merge Sort
Merge Sort Although the merge step produces
a sorted array we have overlooked a very important step
How did we sort the 2 halves before performing the merge step
We used merge sort
Merge Sort By continually calling the merge
sort algorithm we eventually get a subarray of size 1
Since an array with only 1 element is clearly sorted we can back out and merge 2 arrays of size 1
Merge Sort
Merge Sort The basic merging algorithm
consists of 2 input arrays (arrayA and arrayB) An ouput array (arrayC) 3 position holders (indexA indexB
indexC) which are initially set to the beginning of their respective arrays
Merge Sort The smaller of arrayA[indexA] and
arrayB[indexB] is copied into arrayC[indexC] and the appropriate position holders are advanced
When either input list is exhausted the remainder of the other list is copied into arrayC
Merge Sort
1 13 24 26index
A
indexC
arrayA
2 15 27 38index
B
arrayB
arrayC
We compare arrayA[indexA]with arrayB[indexB] Whichever value is smaller isplaced into arrayC[indexC]
1 lt 2 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1index
C
arrayA
2 15 27 38index
B
arrayB
arrayC
2 lt 13 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
13 lt 15 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
15 lt 24 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13 15indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
24 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
indexA
1 2 13 15 24indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
26 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
1 2 13 15 24 26indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
Since we have exhaustedone of the arrays arrayAwe simply copy the remaining items from theother array arrayB intoarrayC
Merge Sort
1 13 24
26
1 2 13 15 24 26 27 38
arrayA
2 15 27 38arrayB
arrayC
Efficiency Summary
Sort Worst Case Average Case
Insertion O(n2) O(n2)
Shell O(n15) O(n125)
Selection O(n2) O(n2)
Bubble O(n2) O(n2)
Quick O(n2) O(nlog2n)
Heap O(nlog2n) O(nlog2n)
Merge O(nlog2n) O(nlog2n)
More Sorting
radix sort bucket sort in-place sorting how fast can we sort
Radix Sort
Unlike other sorting methods radix sort considers the structure of the keys
Assume keys are represented in a base M number system (M = radix) ie if M = 2 the keys are represented in binary
Sorting is done by comparing bits in the same position
Extension to keys that are alphanumeric strings
Radix Exchange Sort
Examine bits from left to right1 Sort array with respect to leftmost bit
2 Partition array
3Recursionrecursively sort top subarray ignoring leftmost bitrecursively sort bottom subarray ignoring leftmost bit
Time to sort n b-bit numbers O(b n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Heap Sort Idea take the items that need to be
sorted and insert them into the heap By calling deleteHeap we remove the
smallest or largest element depending on whether or not we are working with a min- or max-heap respectively
Hence the elements are removed in ascending or descending order
Efficiency O(nlog2n)
Merge Sort Idea
Take the array you would like to sort and divide it in half to create 2 unsorted subarrays
Next sort each of the 2 subarrays Finally merge the 2 sorted subarrays
into 1 sorted array Efficiency O(nlog2n)
Merge Sort
Merge Sort Although the merge step produces
a sorted array we have overlooked a very important step
How did we sort the 2 halves before performing the merge step
We used merge sort
Merge Sort By continually calling the merge
sort algorithm we eventually get a subarray of size 1
Since an array with only 1 element is clearly sorted we can back out and merge 2 arrays of size 1
Merge Sort
Merge Sort The basic merging algorithm
consists of 2 input arrays (arrayA and arrayB) An ouput array (arrayC) 3 position holders (indexA indexB
indexC) which are initially set to the beginning of their respective arrays
Merge Sort The smaller of arrayA[indexA] and
arrayB[indexB] is copied into arrayC[indexC] and the appropriate position holders are advanced
When either input list is exhausted the remainder of the other list is copied into arrayC
Merge Sort
1 13 24 26index
A
indexC
arrayA
2 15 27 38index
B
arrayB
arrayC
We compare arrayA[indexA]with arrayB[indexB] Whichever value is smaller isplaced into arrayC[indexC]
1 lt 2 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1index
C
arrayA
2 15 27 38index
B
arrayB
arrayC
2 lt 13 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
13 lt 15 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
15 lt 24 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13 15indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
24 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
indexA
1 2 13 15 24indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
26 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
1 2 13 15 24 26indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
Since we have exhaustedone of the arrays arrayAwe simply copy the remaining items from theother array arrayB intoarrayC
Merge Sort
1 13 24
26
1 2 13 15 24 26 27 38
arrayA
2 15 27 38arrayB
arrayC
Efficiency Summary
Sort Worst Case Average Case
Insertion O(n2) O(n2)
Shell O(n15) O(n125)
Selection O(n2) O(n2)
Bubble O(n2) O(n2)
Quick O(n2) O(nlog2n)
Heap O(nlog2n) O(nlog2n)
Merge O(nlog2n) O(nlog2n)
More Sorting
radix sort bucket sort in-place sorting how fast can we sort
Radix Sort
Unlike other sorting methods radix sort considers the structure of the keys
Assume keys are represented in a base M number system (M = radix) ie if M = 2 the keys are represented in binary
Sorting is done by comparing bits in the same position
Extension to keys that are alphanumeric strings
Radix Exchange Sort
Examine bits from left to right1 Sort array with respect to leftmost bit
2 Partition array
3Recursionrecursively sort top subarray ignoring leftmost bitrecursively sort bottom subarray ignoring leftmost bit
Time to sort n b-bit numbers O(b n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Merge Sort Idea
Take the array you would like to sort and divide it in half to create 2 unsorted subarrays
Next sort each of the 2 subarrays Finally merge the 2 sorted subarrays
into 1 sorted array Efficiency O(nlog2n)
Merge Sort
Merge Sort Although the merge step produces
a sorted array we have overlooked a very important step
How did we sort the 2 halves before performing the merge step
We used merge sort
Merge Sort By continually calling the merge
sort algorithm we eventually get a subarray of size 1
Since an array with only 1 element is clearly sorted we can back out and merge 2 arrays of size 1
Merge Sort
Merge Sort The basic merging algorithm
consists of 2 input arrays (arrayA and arrayB) An ouput array (arrayC) 3 position holders (indexA indexB
indexC) which are initially set to the beginning of their respective arrays
Merge Sort The smaller of arrayA[indexA] and
arrayB[indexB] is copied into arrayC[indexC] and the appropriate position holders are advanced
When either input list is exhausted the remainder of the other list is copied into arrayC
Merge Sort
1 13 24 26index
A
indexC
arrayA
2 15 27 38index
B
arrayB
arrayC
We compare arrayA[indexA]with arrayB[indexB] Whichever value is smaller isplaced into arrayC[indexC]
1 lt 2 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1index
C
arrayA
2 15 27 38index
B
arrayB
arrayC
2 lt 13 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
13 lt 15 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
15 lt 24 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13 15indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
24 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
indexA
1 2 13 15 24indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
26 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
1 2 13 15 24 26indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
Since we have exhaustedone of the arrays arrayAwe simply copy the remaining items from theother array arrayB intoarrayC
Merge Sort
1 13 24
26
1 2 13 15 24 26 27 38
arrayA
2 15 27 38arrayB
arrayC
Efficiency Summary
Sort Worst Case Average Case
Insertion O(n2) O(n2)
Shell O(n15) O(n125)
Selection O(n2) O(n2)
Bubble O(n2) O(n2)
Quick O(n2) O(nlog2n)
Heap O(nlog2n) O(nlog2n)
Merge O(nlog2n) O(nlog2n)
More Sorting
radix sort bucket sort in-place sorting how fast can we sort
Radix Sort
Unlike other sorting methods radix sort considers the structure of the keys
Assume keys are represented in a base M number system (M = radix) ie if M = 2 the keys are represented in binary
Sorting is done by comparing bits in the same position
Extension to keys that are alphanumeric strings
Radix Exchange Sort
Examine bits from left to right1 Sort array with respect to leftmost bit
2 Partition array
3Recursionrecursively sort top subarray ignoring leftmost bitrecursively sort bottom subarray ignoring leftmost bit
Time to sort n b-bit numbers O(b n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Merge Sort
Merge Sort Although the merge step produces
a sorted array we have overlooked a very important step
How did we sort the 2 halves before performing the merge step
We used merge sort
Merge Sort By continually calling the merge
sort algorithm we eventually get a subarray of size 1
Since an array with only 1 element is clearly sorted we can back out and merge 2 arrays of size 1
Merge Sort
Merge Sort The basic merging algorithm
consists of 2 input arrays (arrayA and arrayB) An ouput array (arrayC) 3 position holders (indexA indexB
indexC) which are initially set to the beginning of their respective arrays
Merge Sort The smaller of arrayA[indexA] and
arrayB[indexB] is copied into arrayC[indexC] and the appropriate position holders are advanced
When either input list is exhausted the remainder of the other list is copied into arrayC
Merge Sort
1 13 24 26index
A
indexC
arrayA
2 15 27 38index
B
arrayB
arrayC
We compare arrayA[indexA]with arrayB[indexB] Whichever value is smaller isplaced into arrayC[indexC]
1 lt 2 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1index
C
arrayA
2 15 27 38index
B
arrayB
arrayC
2 lt 13 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
13 lt 15 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
15 lt 24 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13 15indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
24 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
indexA
1 2 13 15 24indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
26 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
1 2 13 15 24 26indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
Since we have exhaustedone of the arrays arrayAwe simply copy the remaining items from theother array arrayB intoarrayC
Merge Sort
1 13 24
26
1 2 13 15 24 26 27 38
arrayA
2 15 27 38arrayB
arrayC
Efficiency Summary
Sort Worst Case Average Case
Insertion O(n2) O(n2)
Shell O(n15) O(n125)
Selection O(n2) O(n2)
Bubble O(n2) O(n2)
Quick O(n2) O(nlog2n)
Heap O(nlog2n) O(nlog2n)
Merge O(nlog2n) O(nlog2n)
More Sorting
radix sort bucket sort in-place sorting how fast can we sort
Radix Sort
Unlike other sorting methods radix sort considers the structure of the keys
Assume keys are represented in a base M number system (M = radix) ie if M = 2 the keys are represented in binary
Sorting is done by comparing bits in the same position
Extension to keys that are alphanumeric strings
Radix Exchange Sort
Examine bits from left to right1 Sort array with respect to leftmost bit
2 Partition array
3Recursionrecursively sort top subarray ignoring leftmost bitrecursively sort bottom subarray ignoring leftmost bit
Time to sort n b-bit numbers O(b n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Merge Sort Although the merge step produces
a sorted array we have overlooked a very important step
How did we sort the 2 halves before performing the merge step
We used merge sort
Merge Sort By continually calling the merge
sort algorithm we eventually get a subarray of size 1
Since an array with only 1 element is clearly sorted we can back out and merge 2 arrays of size 1
Merge Sort
Merge Sort The basic merging algorithm
consists of 2 input arrays (arrayA and arrayB) An ouput array (arrayC) 3 position holders (indexA indexB
indexC) which are initially set to the beginning of their respective arrays
Merge Sort The smaller of arrayA[indexA] and
arrayB[indexB] is copied into arrayC[indexC] and the appropriate position holders are advanced
When either input list is exhausted the remainder of the other list is copied into arrayC
Merge Sort
1 13 24 26index
A
indexC
arrayA
2 15 27 38index
B
arrayB
arrayC
We compare arrayA[indexA]with arrayB[indexB] Whichever value is smaller isplaced into arrayC[indexC]
1 lt 2 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1index
C
arrayA
2 15 27 38index
B
arrayB
arrayC
2 lt 13 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
13 lt 15 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
15 lt 24 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13 15indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
24 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
indexA
1 2 13 15 24indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
26 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
1 2 13 15 24 26indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
Since we have exhaustedone of the arrays arrayAwe simply copy the remaining items from theother array arrayB intoarrayC
Merge Sort
1 13 24
26
1 2 13 15 24 26 27 38
arrayA
2 15 27 38arrayB
arrayC
Efficiency Summary
Sort Worst Case Average Case
Insertion O(n2) O(n2)
Shell O(n15) O(n125)
Selection O(n2) O(n2)
Bubble O(n2) O(n2)
Quick O(n2) O(nlog2n)
Heap O(nlog2n) O(nlog2n)
Merge O(nlog2n) O(nlog2n)
More Sorting
radix sort bucket sort in-place sorting how fast can we sort
Radix Sort
Unlike other sorting methods radix sort considers the structure of the keys
Assume keys are represented in a base M number system (M = radix) ie if M = 2 the keys are represented in binary
Sorting is done by comparing bits in the same position
Extension to keys that are alphanumeric strings
Radix Exchange Sort
Examine bits from left to right1 Sort array with respect to leftmost bit
2 Partition array
3Recursionrecursively sort top subarray ignoring leftmost bitrecursively sort bottom subarray ignoring leftmost bit
Time to sort n b-bit numbers O(b n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Merge Sort By continually calling the merge
sort algorithm we eventually get a subarray of size 1
Since an array with only 1 element is clearly sorted we can back out and merge 2 arrays of size 1
Merge Sort
Merge Sort The basic merging algorithm
consists of 2 input arrays (arrayA and arrayB) An ouput array (arrayC) 3 position holders (indexA indexB
indexC) which are initially set to the beginning of their respective arrays
Merge Sort The smaller of arrayA[indexA] and
arrayB[indexB] is copied into arrayC[indexC] and the appropriate position holders are advanced
When either input list is exhausted the remainder of the other list is copied into arrayC
Merge Sort
1 13 24 26index
A
indexC
arrayA
2 15 27 38index
B
arrayB
arrayC
We compare arrayA[indexA]with arrayB[indexB] Whichever value is smaller isplaced into arrayC[indexC]
1 lt 2 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1index
C
arrayA
2 15 27 38index
B
arrayB
arrayC
2 lt 13 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
13 lt 15 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
15 lt 24 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13 15indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
24 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
indexA
1 2 13 15 24indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
26 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
1 2 13 15 24 26indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
Since we have exhaustedone of the arrays arrayAwe simply copy the remaining items from theother array arrayB intoarrayC
Merge Sort
1 13 24
26
1 2 13 15 24 26 27 38
arrayA
2 15 27 38arrayB
arrayC
Efficiency Summary
Sort Worst Case Average Case
Insertion O(n2) O(n2)
Shell O(n15) O(n125)
Selection O(n2) O(n2)
Bubble O(n2) O(n2)
Quick O(n2) O(nlog2n)
Heap O(nlog2n) O(nlog2n)
Merge O(nlog2n) O(nlog2n)
More Sorting
radix sort bucket sort in-place sorting how fast can we sort
Radix Sort
Unlike other sorting methods radix sort considers the structure of the keys
Assume keys are represented in a base M number system (M = radix) ie if M = 2 the keys are represented in binary
Sorting is done by comparing bits in the same position
Extension to keys that are alphanumeric strings
Radix Exchange Sort
Examine bits from left to right1 Sort array with respect to leftmost bit
2 Partition array
3Recursionrecursively sort top subarray ignoring leftmost bitrecursively sort bottom subarray ignoring leftmost bit
Time to sort n b-bit numbers O(b n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Merge Sort
Merge Sort The basic merging algorithm
consists of 2 input arrays (arrayA and arrayB) An ouput array (arrayC) 3 position holders (indexA indexB
indexC) which are initially set to the beginning of their respective arrays
Merge Sort The smaller of arrayA[indexA] and
arrayB[indexB] is copied into arrayC[indexC] and the appropriate position holders are advanced
When either input list is exhausted the remainder of the other list is copied into arrayC
Merge Sort
1 13 24 26index
A
indexC
arrayA
2 15 27 38index
B
arrayB
arrayC
We compare arrayA[indexA]with arrayB[indexB] Whichever value is smaller isplaced into arrayC[indexC]
1 lt 2 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1index
C
arrayA
2 15 27 38index
B
arrayB
arrayC
2 lt 13 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
13 lt 15 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
15 lt 24 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13 15indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
24 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
indexA
1 2 13 15 24indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
26 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
1 2 13 15 24 26indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
Since we have exhaustedone of the arrays arrayAwe simply copy the remaining items from theother array arrayB intoarrayC
Merge Sort
1 13 24
26
1 2 13 15 24 26 27 38
arrayA
2 15 27 38arrayB
arrayC
Efficiency Summary
Sort Worst Case Average Case
Insertion O(n2) O(n2)
Shell O(n15) O(n125)
Selection O(n2) O(n2)
Bubble O(n2) O(n2)
Quick O(n2) O(nlog2n)
Heap O(nlog2n) O(nlog2n)
Merge O(nlog2n) O(nlog2n)
More Sorting
radix sort bucket sort in-place sorting how fast can we sort
Radix Sort
Unlike other sorting methods radix sort considers the structure of the keys
Assume keys are represented in a base M number system (M = radix) ie if M = 2 the keys are represented in binary
Sorting is done by comparing bits in the same position
Extension to keys that are alphanumeric strings
Radix Exchange Sort
Examine bits from left to right1 Sort array with respect to leftmost bit
2 Partition array
3Recursionrecursively sort top subarray ignoring leftmost bitrecursively sort bottom subarray ignoring leftmost bit
Time to sort n b-bit numbers O(b n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Merge Sort The basic merging algorithm
consists of 2 input arrays (arrayA and arrayB) An ouput array (arrayC) 3 position holders (indexA indexB
indexC) which are initially set to the beginning of their respective arrays
Merge Sort The smaller of arrayA[indexA] and
arrayB[indexB] is copied into arrayC[indexC] and the appropriate position holders are advanced
When either input list is exhausted the remainder of the other list is copied into arrayC
Merge Sort
1 13 24 26index
A
indexC
arrayA
2 15 27 38index
B
arrayB
arrayC
We compare arrayA[indexA]with arrayB[indexB] Whichever value is smaller isplaced into arrayC[indexC]
1 lt 2 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1index
C
arrayA
2 15 27 38index
B
arrayB
arrayC
2 lt 13 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
13 lt 15 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
15 lt 24 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13 15indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
24 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
indexA
1 2 13 15 24indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
26 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
1 2 13 15 24 26indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
Since we have exhaustedone of the arrays arrayAwe simply copy the remaining items from theother array arrayB intoarrayC
Merge Sort
1 13 24
26
1 2 13 15 24 26 27 38
arrayA
2 15 27 38arrayB
arrayC
Efficiency Summary
Sort Worst Case Average Case
Insertion O(n2) O(n2)
Shell O(n15) O(n125)
Selection O(n2) O(n2)
Bubble O(n2) O(n2)
Quick O(n2) O(nlog2n)
Heap O(nlog2n) O(nlog2n)
Merge O(nlog2n) O(nlog2n)
More Sorting
radix sort bucket sort in-place sorting how fast can we sort
Radix Sort
Unlike other sorting methods radix sort considers the structure of the keys
Assume keys are represented in a base M number system (M = radix) ie if M = 2 the keys are represented in binary
Sorting is done by comparing bits in the same position
Extension to keys that are alphanumeric strings
Radix Exchange Sort
Examine bits from left to right1 Sort array with respect to leftmost bit
2 Partition array
3Recursionrecursively sort top subarray ignoring leftmost bitrecursively sort bottom subarray ignoring leftmost bit
Time to sort n b-bit numbers O(b n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Merge Sort The smaller of arrayA[indexA] and
arrayB[indexB] is copied into arrayC[indexC] and the appropriate position holders are advanced
When either input list is exhausted the remainder of the other list is copied into arrayC
Merge Sort
1 13 24 26index
A
indexC
arrayA
2 15 27 38index
B
arrayB
arrayC
We compare arrayA[indexA]with arrayB[indexB] Whichever value is smaller isplaced into arrayC[indexC]
1 lt 2 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1index
C
arrayA
2 15 27 38index
B
arrayB
arrayC
2 lt 13 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
13 lt 15 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
15 lt 24 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13 15indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
24 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
indexA
1 2 13 15 24indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
26 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
1 2 13 15 24 26indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
Since we have exhaustedone of the arrays arrayAwe simply copy the remaining items from theother array arrayB intoarrayC
Merge Sort
1 13 24
26
1 2 13 15 24 26 27 38
arrayA
2 15 27 38arrayB
arrayC
Efficiency Summary
Sort Worst Case Average Case
Insertion O(n2) O(n2)
Shell O(n15) O(n125)
Selection O(n2) O(n2)
Bubble O(n2) O(n2)
Quick O(n2) O(nlog2n)
Heap O(nlog2n) O(nlog2n)
Merge O(nlog2n) O(nlog2n)
More Sorting
radix sort bucket sort in-place sorting how fast can we sort
Radix Sort
Unlike other sorting methods radix sort considers the structure of the keys
Assume keys are represented in a base M number system (M = radix) ie if M = 2 the keys are represented in binary
Sorting is done by comparing bits in the same position
Extension to keys that are alphanumeric strings
Radix Exchange Sort
Examine bits from left to right1 Sort array with respect to leftmost bit
2 Partition array
3Recursionrecursively sort top subarray ignoring leftmost bitrecursively sort bottom subarray ignoring leftmost bit
Time to sort n b-bit numbers O(b n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Merge Sort
1 13 24 26index
A
indexC
arrayA
2 15 27 38index
B
arrayB
arrayC
We compare arrayA[indexA]with arrayB[indexB] Whichever value is smaller isplaced into arrayC[indexC]
1 lt 2 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1index
C
arrayA
2 15 27 38index
B
arrayB
arrayC
2 lt 13 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
13 lt 15 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
15 lt 24 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13 15indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
24 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
indexA
1 2 13 15 24indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
26 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
1 2 13 15 24 26indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
Since we have exhaustedone of the arrays arrayAwe simply copy the remaining items from theother array arrayB intoarrayC
Merge Sort
1 13 24
26
1 2 13 15 24 26 27 38
arrayA
2 15 27 38arrayB
arrayC
Efficiency Summary
Sort Worst Case Average Case
Insertion O(n2) O(n2)
Shell O(n15) O(n125)
Selection O(n2) O(n2)
Bubble O(n2) O(n2)
Quick O(n2) O(nlog2n)
Heap O(nlog2n) O(nlog2n)
Merge O(nlog2n) O(nlog2n)
More Sorting
radix sort bucket sort in-place sorting how fast can we sort
Radix Sort
Unlike other sorting methods radix sort considers the structure of the keys
Assume keys are represented in a base M number system (M = radix) ie if M = 2 the keys are represented in binary
Sorting is done by comparing bits in the same position
Extension to keys that are alphanumeric strings
Radix Exchange Sort
Examine bits from left to right1 Sort array with respect to leftmost bit
2 Partition array
3Recursionrecursively sort top subarray ignoring leftmost bitrecursively sort bottom subarray ignoring leftmost bit
Time to sort n b-bit numbers O(b n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Merge Sort
1 13 24 26indexA
1index
C
arrayA
2 15 27 38index
B
arrayB
arrayC
2 lt 13 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
13 lt 15 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
15 lt 24 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13 15indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
24 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
indexA
1 2 13 15 24indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
26 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
1 2 13 15 24 26indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
Since we have exhaustedone of the arrays arrayAwe simply copy the remaining items from theother array arrayB intoarrayC
Merge Sort
1 13 24
26
1 2 13 15 24 26 27 38
arrayA
2 15 27 38arrayB
arrayC
Efficiency Summary
Sort Worst Case Average Case
Insertion O(n2) O(n2)
Shell O(n15) O(n125)
Selection O(n2) O(n2)
Bubble O(n2) O(n2)
Quick O(n2) O(nlog2n)
Heap O(nlog2n) O(nlog2n)
Merge O(nlog2n) O(nlog2n)
More Sorting
radix sort bucket sort in-place sorting how fast can we sort
Radix Sort
Unlike other sorting methods radix sort considers the structure of the keys
Assume keys are represented in a base M number system (M = radix) ie if M = 2 the keys are represented in binary
Sorting is done by comparing bits in the same position
Extension to keys that are alphanumeric strings
Radix Exchange Sort
Examine bits from left to right1 Sort array with respect to leftmost bit
2 Partition array
3Recursionrecursively sort top subarray ignoring leftmost bitrecursively sort bottom subarray ignoring leftmost bit
Time to sort n b-bit numbers O(b n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Merge Sort
1 13 24 26indexA
1 2index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
13 lt 15 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
15 lt 24 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13 15indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
24 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
indexA
1 2 13 15 24indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
26 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
1 2 13 15 24 26indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
Since we have exhaustedone of the arrays arrayAwe simply copy the remaining items from theother array arrayB intoarrayC
Merge Sort
1 13 24
26
1 2 13 15 24 26 27 38
arrayA
2 15 27 38arrayB
arrayC
Efficiency Summary
Sort Worst Case Average Case
Insertion O(n2) O(n2)
Shell O(n15) O(n125)
Selection O(n2) O(n2)
Bubble O(n2) O(n2)
Quick O(n2) O(nlog2n)
Heap O(nlog2n) O(nlog2n)
Merge O(nlog2n) O(nlog2n)
More Sorting
radix sort bucket sort in-place sorting how fast can we sort
Radix Sort
Unlike other sorting methods radix sort considers the structure of the keys
Assume keys are represented in a base M number system (M = radix) ie if M = 2 the keys are represented in binary
Sorting is done by comparing bits in the same position
Extension to keys that are alphanumeric strings
Radix Exchange Sort
Examine bits from left to right1 Sort array with respect to leftmost bit
2 Partition array
3Recursionrecursively sort top subarray ignoring leftmost bitrecursively sort bottom subarray ignoring leftmost bit
Time to sort n b-bit numbers O(b n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Merge Sort
1 13 24 26indexA
1 2 13index
C
arrayA
2 15 27 38indexB
arrayB
arrayC
15 lt 24 so we insert arrayB[indexB] intoarrayC[indexC]
Merge Sort
1 13 24 26indexA
1 2 13 15indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
24 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
indexA
1 2 13 15 24indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
26 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
1 2 13 15 24 26indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
Since we have exhaustedone of the arrays arrayAwe simply copy the remaining items from theother array arrayB intoarrayC
Merge Sort
1 13 24
26
1 2 13 15 24 26 27 38
arrayA
2 15 27 38arrayB
arrayC
Efficiency Summary
Sort Worst Case Average Case
Insertion O(n2) O(n2)
Shell O(n15) O(n125)
Selection O(n2) O(n2)
Bubble O(n2) O(n2)
Quick O(n2) O(nlog2n)
Heap O(nlog2n) O(nlog2n)
Merge O(nlog2n) O(nlog2n)
More Sorting
radix sort bucket sort in-place sorting how fast can we sort
Radix Sort
Unlike other sorting methods radix sort considers the structure of the keys
Assume keys are represented in a base M number system (M = radix) ie if M = 2 the keys are represented in binary
Sorting is done by comparing bits in the same position
Extension to keys that are alphanumeric strings
Radix Exchange Sort
Examine bits from left to right1 Sort array with respect to leftmost bit
2 Partition array
3Recursionrecursively sort top subarray ignoring leftmost bitrecursively sort bottom subarray ignoring leftmost bit
Time to sort n b-bit numbers O(b n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Merge Sort
1 13 24 26indexA
1 2 13 15indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
24 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
indexA
1 2 13 15 24indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
26 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
1 2 13 15 24 26indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
Since we have exhaustedone of the arrays arrayAwe simply copy the remaining items from theother array arrayB intoarrayC
Merge Sort
1 13 24
26
1 2 13 15 24 26 27 38
arrayA
2 15 27 38arrayB
arrayC
Efficiency Summary
Sort Worst Case Average Case
Insertion O(n2) O(n2)
Shell O(n15) O(n125)
Selection O(n2) O(n2)
Bubble O(n2) O(n2)
Quick O(n2) O(nlog2n)
Heap O(nlog2n) O(nlog2n)
Merge O(nlog2n) O(nlog2n)
More Sorting
radix sort bucket sort in-place sorting how fast can we sort
Radix Sort
Unlike other sorting methods radix sort considers the structure of the keys
Assume keys are represented in a base M number system (M = radix) ie if M = 2 the keys are represented in binary
Sorting is done by comparing bits in the same position
Extension to keys that are alphanumeric strings
Radix Exchange Sort
Examine bits from left to right1 Sort array with respect to leftmost bit
2 Partition array
3Recursionrecursively sort top subarray ignoring leftmost bitrecursively sort bottom subarray ignoring leftmost bit
Time to sort n b-bit numbers O(b n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Merge Sort
1 13 24
26
indexA
1 2 13 15 24indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
26 lt 27 so we insert arrayA[indexA] intoarrayC[indexC]
Merge Sort
1 13 24
26
1 2 13 15 24 26indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
Since we have exhaustedone of the arrays arrayAwe simply copy the remaining items from theother array arrayB intoarrayC
Merge Sort
1 13 24
26
1 2 13 15 24 26 27 38
arrayA
2 15 27 38arrayB
arrayC
Efficiency Summary
Sort Worst Case Average Case
Insertion O(n2) O(n2)
Shell O(n15) O(n125)
Selection O(n2) O(n2)
Bubble O(n2) O(n2)
Quick O(n2) O(nlog2n)
Heap O(nlog2n) O(nlog2n)
Merge O(nlog2n) O(nlog2n)
More Sorting
radix sort bucket sort in-place sorting how fast can we sort
Radix Sort
Unlike other sorting methods radix sort considers the structure of the keys
Assume keys are represented in a base M number system (M = radix) ie if M = 2 the keys are represented in binary
Sorting is done by comparing bits in the same position
Extension to keys that are alphanumeric strings
Radix Exchange Sort
Examine bits from left to right1 Sort array with respect to leftmost bit
2 Partition array
3Recursionrecursively sort top subarray ignoring leftmost bitrecursively sort bottom subarray ignoring leftmost bit
Time to sort n b-bit numbers O(b n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Merge Sort
1 13 24
26
1 2 13 15 24 26indexC
arrayA
2 15 27 38indexB
arrayB
arrayC
Since we have exhaustedone of the arrays arrayAwe simply copy the remaining items from theother array arrayB intoarrayC
Merge Sort
1 13 24
26
1 2 13 15 24 26 27 38
arrayA
2 15 27 38arrayB
arrayC
Efficiency Summary
Sort Worst Case Average Case
Insertion O(n2) O(n2)
Shell O(n15) O(n125)
Selection O(n2) O(n2)
Bubble O(n2) O(n2)
Quick O(n2) O(nlog2n)
Heap O(nlog2n) O(nlog2n)
Merge O(nlog2n) O(nlog2n)
More Sorting
radix sort bucket sort in-place sorting how fast can we sort
Radix Sort
Unlike other sorting methods radix sort considers the structure of the keys
Assume keys are represented in a base M number system (M = radix) ie if M = 2 the keys are represented in binary
Sorting is done by comparing bits in the same position
Extension to keys that are alphanumeric strings
Radix Exchange Sort
Examine bits from left to right1 Sort array with respect to leftmost bit
2 Partition array
3Recursionrecursively sort top subarray ignoring leftmost bitrecursively sort bottom subarray ignoring leftmost bit
Time to sort n b-bit numbers O(b n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Merge Sort
1 13 24
26
1 2 13 15 24 26 27 38
arrayA
2 15 27 38arrayB
arrayC
Efficiency Summary
Sort Worst Case Average Case
Insertion O(n2) O(n2)
Shell O(n15) O(n125)
Selection O(n2) O(n2)
Bubble O(n2) O(n2)
Quick O(n2) O(nlog2n)
Heap O(nlog2n) O(nlog2n)
Merge O(nlog2n) O(nlog2n)
More Sorting
radix sort bucket sort in-place sorting how fast can we sort
Radix Sort
Unlike other sorting methods radix sort considers the structure of the keys
Assume keys are represented in a base M number system (M = radix) ie if M = 2 the keys are represented in binary
Sorting is done by comparing bits in the same position
Extension to keys that are alphanumeric strings
Radix Exchange Sort
Examine bits from left to right1 Sort array with respect to leftmost bit
2 Partition array
3Recursionrecursively sort top subarray ignoring leftmost bitrecursively sort bottom subarray ignoring leftmost bit
Time to sort n b-bit numbers O(b n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Efficiency Summary
Sort Worst Case Average Case
Insertion O(n2) O(n2)
Shell O(n15) O(n125)
Selection O(n2) O(n2)
Bubble O(n2) O(n2)
Quick O(n2) O(nlog2n)
Heap O(nlog2n) O(nlog2n)
Merge O(nlog2n) O(nlog2n)
More Sorting
radix sort bucket sort in-place sorting how fast can we sort
Radix Sort
Unlike other sorting methods radix sort considers the structure of the keys
Assume keys are represented in a base M number system (M = radix) ie if M = 2 the keys are represented in binary
Sorting is done by comparing bits in the same position
Extension to keys that are alphanumeric strings
Radix Exchange Sort
Examine bits from left to right1 Sort array with respect to leftmost bit
2 Partition array
3Recursionrecursively sort top subarray ignoring leftmost bitrecursively sort bottom subarray ignoring leftmost bit
Time to sort n b-bit numbers O(b n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
More Sorting
radix sort bucket sort in-place sorting how fast can we sort
Radix Sort
Unlike other sorting methods radix sort considers the structure of the keys
Assume keys are represented in a base M number system (M = radix) ie if M = 2 the keys are represented in binary
Sorting is done by comparing bits in the same position
Extension to keys that are alphanumeric strings
Radix Exchange Sort
Examine bits from left to right1 Sort array with respect to leftmost bit
2 Partition array
3Recursionrecursively sort top subarray ignoring leftmost bitrecursively sort bottom subarray ignoring leftmost bit
Time to sort n b-bit numbers O(b n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Radix Sort
Unlike other sorting methods radix sort considers the structure of the keys
Assume keys are represented in a base M number system (M = radix) ie if M = 2 the keys are represented in binary
Sorting is done by comparing bits in the same position
Extension to keys that are alphanumeric strings
Radix Exchange Sort
Examine bits from left to right1 Sort array with respect to leftmost bit
2 Partition array
3Recursionrecursively sort top subarray ignoring leftmost bitrecursively sort bottom subarray ignoring leftmost bit
Time to sort n b-bit numbers O(b n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Radix Exchange Sort
Examine bits from left to right1 Sort array with respect to leftmost bit
2 Partition array
3Recursionrecursively sort top subarray ignoring leftmost bitrecursively sort bottom subarray ignoring leftmost bit
Time to sort n b-bit numbers O(b n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Radix Exchange SortHow do we do the sort from the previous page Same idea as partition in Quicksort repeat
scan top-down to find key starting with 1scan bottom-up to find key starting with 0exchange keys
until scan indices cross
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Radix Exchange Sort
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Radix Exchange Sort vs Quicksort
Similaritiesboth partition arrayboth recursively sort sub-arrays
DifferencesMethod of partitioning
radix exchange divides array based on greater than or less than 2b-1
quicksort partitions based on greater than or less than some element of the array
Time complexityRadix exchange O (bn)Quicksort average case O (n log n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Straight Radix Sort
Examines bits from right to left
for k = 0 to b-1 sort the array in a stable waylooking only at bit k
Note order of these bits after sort
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Recall ldquosort in a stable
wayrdquo
In a stable sort the initial relative order of equal keys is unchangedFor example observe the first step of the sort from the previous page
Note that the relative order of those keys ending with 0 is unchanged and the same is true for elements ending in 1
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
The Algorithm is Correct (right)
We show that any two keys are in the correct relative order at the end of the algorithm
Given two keys let k be the leftmost bit-position where they differ
At step k the two keys are put in the correct relative orderBecause of stability the successive steps do not change the relative order of the two keys
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
For InstanceConsider a sort on an array with these two keys
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Radix sorting can be applied to decimal numbers
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Straight Radix SortTime Complexity
for k = 0 to b - 1 sort the array in a stable way looking only at bit k
Suppose we can perform the stable sort above in O(n) time The total time complexity would be
O(bn)As you might have guessed we can perform a stable sort based on the keysrsquo kth digit in O(n) time
The method you ask Why itrsquos Bucket Sort of course
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Bucket Sort
BASICSn numbersEach number 1 2 3 mStableTime O (n + m)
For example m = 3 and our array is
(note that there are two ldquo2rdquos and two ldquo1rdquos)
First we create M ldquobucketsrdquo
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Bucket SortEach element of the array is put in one of the m ldquobucketsrdquo
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Bucket Sort
Now pull the elements from the buckets into the array
At last the sorted array (sorted in a stable way)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Comparison Sorting Elements are rearranged by comparing
values of keys Comparison of keys is interleaved with
rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order
Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys
Sorting an array of 3 elements a b and c produces 6 or 3 distinct orderings
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Decision TreeConsider a decision tree that sorts n items The next slide shows a decision tree for 3 items a b c
Ovals represent comparisons If lt we traverse down a left branch
otherwise a right branch
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Decision Tree on 3 Elements
a b c
bltc
bltc
altb
altc altc
a c b b a cc a b
c b a
b c a
true
true
true true
true
false
false
false
false
false
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Levels in Decision Trees Number the levels starting at zero Level i can represent 2i arrangements Our decision tree can represent 8 = 23
arrangements If we have 4 items then 4 = 24 elements
but 4 levels gives at most 24 = 16 so we need 5 levels since 25 = 32 24
If we have k arrangements to represent then we need a decision tree with at least lg k levels
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
Size of Decision TreeWhen dealing with n elements we have n
possible arrangements and need a decision tree with at least lg n levels
)lg(2
lg2
2lg
2
)2lg()1lg(lg
)1lg()2lg()2lg()1lg(lg
))1)(2)(2)(1(lg()lg(
nn
nn
n
nn
nnn
nnn
nnnn
Lower Bound Resultn = n(n-1)(n-2)hellip21
from this we getlg(n) = lg(n) + lg(n-1) + lg(n-2) +hellip+
lg2 + lg1this means
lg(n) lt nlg(n)and the two results together show
O(lg(n)) = O(nlg(n))
No general sorting algorithm can run faster than O(n lg n)
top related