part3 sorting (1)

8/13/2019 Part3 Sorting (1)

1/62

1

CSE 326: Sorting

Henry KautzAutumn Quarter 2002


2/62

2

Material to be Covered

Sorting by comparision:1. Bubble Sort2. Selection Sort3. Merge Sort4. QuickSort

Efficient list-based implementations Formal analysis

Theoretical limitations on sorting by comparison Sorting without comparing elements Sorting and the memory hierarchy


3/62

3

Bubble Sort Idea

Move smallest element in range 1,, n to position 1 by a series of swaps

Move smallest element in range 2,, n to position 2 by a series of swaps

Move smallest element in range 3,, n to position 3 by a series of swaps etc .


4/62

4

Selection Sort Idea

Rearranged version of Bubble Sort: Are first 2 elements sorted? If not, swap.

Are the first 3 elements sorted? If not, move the3rd element to the left by series of swaps.

Are the first 4 elements sorted? If not, move the

4th

element to the left by series of swaps. etc .


5/62

5

Selection Sort procedure SelectionSort (Array[1..N])

For (i=2 to N) {j = i;

while ( j > 0 && Array[j] < Array[j-1] ){

swap( Array[j], Array[j-1] )j --; }

}


6/62

6

Why Selection (or Bubble) Sort

is Slow Inversion : a pair (i,j) such that i Array[j] Array of size N can have (N 2) inversions Selection/Bubble Sort only swaps adjacent

elements Only removes 1 inversion at a time!

Worst case running time is (N 2)


7/62

7

Merge Sort

Photo from http://www.nrma.com.au/inside-nrma/m-h-m/road-rage.html

Merging Cars by key[Aggressiveness of driver].Most aggressive goes first.

MergeSort (Table [1.. n ])Split Table in half

Recursively sort each halfMerge two halves together

Merge ( T1 [1.. n ], T2 [1.. n ])i1 =1, i2 =1

While i1 < n , i2 < nIf T1 [ i1 ] < T2 [ i2 ]

Next is T1 [ i1 ]i1 ++

ElseNext is T2 [ i2 ]i2 ++

End IfEnd While


8/62

8

Merge Sort Running Time

T(1) = bT(n) = 2T(n/2) + c n for n>1

T(n) = 2T(n/2)+cnT(n) = 4T(n/4) +cn +cn substitute

T(n) = 8T(n/8)+cn+cn+cn substitute

T(n) = 2 k T(n/2 k )+kcn inductive leap

T(n) = nT(1) + cn log n where k = log n select value for k

T(n) = (n log n) simplify

Any difference best / worse case?


9/62

9

QuickSort

28

15 47

1. Pick a pivot.2. Divide list into two lists:

One less-than-or-equal-to pivot value One greater than pivot

3. Sort each sub-problem recursively4. Answer is the concatenation of the two solutions

Picture from PhotoDisc.com


10/62

10

QuickSort: Array-Based Version7 2 8 3 5 9 6Pick pivot:

Partitionwith cursors

7 2 8 3 5 9 6

< >

7 2 8 3 5 9 6

< >

2 goes to

less-than


11/62

11

QuickSort Partition (contd)

7 2 6 3 5 9 8

< >

6, 8 swapless/greater-than

7 2 6 3 5 9 83,5 less-than9 greater-than

7 2 6 3 5 9 8Partition done.


12/62

12

QuickSort Partition (contd)

9876532Recursivelysort each side.

8973625Put pivotinto final

position.


13/62

13

QuickSort Complexity

QuickSort is fast in practice, but has (N 2)worst-case complexity

Tomorrow we will see why But before then


14/62

14

List-Based Implementation

All these algorithms can be implemented usinglinked lists rather than arrays while retaining the

same asymptotic complexity Exercise:

Break into 6 groups (6 or 7 people each) Select a leader 25 minutes to sketch out an efficient implementation

Summarize on transparencies Report back at 3:00 pm.


15/62

15

Notes

Almost Java pseudo -code is fine Dont worry about iterators, hiding, etc

just directly work on ListNodes The head field can point directly to the

first node in the list, or to a dummy node, asyou prefer


16/62

16

List Class Declarations

class LinkedList {

class ListNode {

Object element;

ListNode next; }

ListNode head;

void Sort(){ . . . }

}


17/62

17

My Implementations

Probably no better (or worse) than yours Assumes no header nodes for lists

Careless about creating garbage, butasymptotically doesnt hurt

For selection sort, did the bubble-sort variation, but moving largest element to end rather thansmallest to beginning each time. Swappedelements rather than nodes themselves.


18/62

18

My QuickSort

void QuickSort(){ // sort selfif (is_empty()) return;Object val = Pop(); // choose pivot

b = new List();c = new List();Split(val, b, c); // split self into 2 lists

b.QuickSort();c.QuickSort();

c.Push(val); // insert pivot b.Append(c); // concatenate solutions

head = b.head; // set self to solution}


19/62

19

Split, Append

void Split( Object val, List b, c ){if (is_empty()) return;Object obj = Pop();if (obj


20/62

20

Last, Push, PopListNode Last(){

ListNode n = head;if (n==null) return null;

while (n.next!=null) n=n.next;return n; }

void Push(Object val){ListNode h = new ListNode(val);h.next = head;head = h; }

Object Pop(){if (head==null) error();Object val = head.element;head = head.next;return val; }


21/62

21

My Merge Sort

void MergeSort(){ // sort selfif (is_empty()) return;

b = new List();c = new List();SplitHalf(b, c); // split self into 2 lists

b.MergeSort();c.MergeSort();head = Merge(b.head,c.head);

// set self to merged solutions}


22/62

22

SplitHalf, Mergevoid SplitHalf(List b, c){

if (is_empty()) return; b.Push(Pop());SplitHalf(c, b); // alternate b,c

}

ListNode Merge( ListNode b, c ){if (b==null) return c;if (c==null) return b;if (b.element


23/62

23

My Bubble Sortvoid BubbleSort(){

int n = Length(); // length of this listfor (i=2; i


24/62

24

Lets go to the Races!
http://localhost/var/www/apps/conversion/tmp/scratch_4/sort-demos/mydemo.htmhttp://localhost/var/www/apps/conversion/tmp/scratch_4/sort-demos/mydemo.htm


25/62

25

Analyzing QuickSort

Picking pivot: constant time Partitioning: linear time

Recursion: time for sorting left partition(say of size i) + time for right (size N-i-1) +time to combine solutions

T(1) = bT(N) = T(i) + T(N-i-1) + cNwhere i is the number of elements smaller than the pivot


26/62

26

QuickSort

Worst casePivot is always smallest element, so i=0 :

T(N) = T(i) + T(N-i-1) + cN

T(N) = T(N-1) + cN

= T(N-2) + c(N-1) + cN

= T(N-k) +

= O(N 2)

1

0

( )k

i

c N i


27/62

27

Dealing with Slow QuickSorts

Randomly choose pivot Good theoretically and practically, but call to

random number generator can be expensive Pick pivot cleverly

Median -of- 3 rule takes Median(first, middle,

last element elements). Also works well.


28/62

28

QuickSort

Best CasePivot is always middle element.

T(N) = T(i) + T(N-i-1) + cN

T(N) = 2T(N/2 - 1) + cN

2 ( / 2)

4 ( / 4) (2 / 2 )

8 ( / 8) (1 1 1)

(( / ) l go og( ) l )

T N cN

T N c N N

T N cN

kT N k cN k O N N

< < < <

What is k?


29/62

29

QuickSortAverage Case

Suppose pivot is picked at random from values inthe list

All the following cases are equally likely : Pivot is smallest value in list Pivot is 2 nd smallest value in list Pivot is 3 rd smallest value in list

Pivot is largest value in list

Same is true if pivot is e.g. always first element, but the input itself is perfectly random


30/62

30

QuickSort Avg Case, cont. Expected running time = sum of

(time when partition size i) (probability partition is size i)

In either random case, all size partitions are equallylikely probability is just 1/N

0

1

0

1

( ) ( ) ( 1)

( ( )) (2 / ) ( ( ))

Solving this recursive equation (see Weiss pg 249) yiel

( ( )) (

( ( )) (1/ ) ( ( )

log )

d :

) (

s

( 1)) N

i

N

i

T N T i T N i cN

E T

E

N N E

T N N E T

T

i

E T N

E T N i cN

O N

i cN

N


31/62

31

Could We Do Better?

For any possible correct Sorting by Comparison algorithm, what is

lowest worst case time? Imagine how the comparisons thatwould be performed by the best

possible sorting algorithm form a

decision tree Worst-case running time cannot be

less than the depth of this tree!


32/62

32

Decision tree to sort list A,B,C

A < B

B < C

A < C C

< A

C < B

B < A

A < C C

< A

B < C C < B

A


33/62

33

Max depth of the decision tree

How many permutations are there of N numbers?

How many leaves does the tree have?

Whats the shallowest tree with a given number of leaves?

What is therefore the worst running time (number ofcomparisons) by the best possible sorting algorithm?


34/62

34

Max depth of the decision tree

How many permutations are there of N numbers? N!

How many leaves does the tree have? N!

Whats the shallowest tree with a given number of leaves? log(N!)

What is therefore the worst running time (number ofcomparisons) by the best possible sorting algorithm?

log(N!)


35/62

35

Stirlings approximation n

e

nnn

2!

log( !) log 2

log( 2 ) lo ( log )g

n

n

nn n

e

nn n n

e


36/62

36

Stirlings Approximation Redux

11

1

(log !) (ln !) ( ln ) ( lo

ln ! ln1 ln

g

2 ... ln

ln ln

ln ln 1

)

n n

k

n

n n

k x dx

x

n n n n n

x

n

n n n


37/62

37

Why is QuickSort Faster thanMerge Sort?

Quicksort typically performs more comparisons thanMergesort, because partitions are not always perfectly

balanced Mergesort n log n comparisons Quicksort 1.38 n log n comparisons on average

Quicksort performs many fewer copies , because onaverage half of the elements are on the correct side ofthe partition while Mergesort copies every elementwhen merging Mergesort 2n log n copies (using temp array)

n log n copies (using alternating array) Quicksort n/2 log n copies on average


38/62

38

Sorting HUGE Data Sets

US Telephone Directory: 300,000,000 records

64-bytes per record Name: 32 characters

Address: 54 characters Telephone number: 10 characters

About 2 gigabytes of data Sort this on a machine with 128 MB RAM

Other examples?


39/62

39

Merge Sort Good for Something!

Basis for most external sorting routines Can sort any number of records using a tiny

amount of main memory in extreme case, only need to keep 2 records in

memory at any one time!


40/62

40

External MergeSort Split input into two tapes (or areas of disk) Merge tapes so that each group of 2 records is

sorted

Split again Merge tapes so that each group of 4 records is

sorted Repeat until data entirely sorted

log N passes


41/62

41

Better External MergeSort

Suppose main memory can hold M records. Initially read in groups of M records and

sort them ( e.g . with QuickSort). Number of passes reduced to log(N/M)


42/62

42

Sorting by Comparison: Summary Sorting algorithms that only compare adjacent

elements are (N 2) worst case but may be(N) best case

MergeSort - (N log N) both best and worstcase

QuickSort (N 2) worst case but (N log N) best and average case

Any comparison-based sorting algorithm is(N log N) worst case External sorting: MergeSort with (log N/M)

passes

but not quite the end of the story


43/62

43

BucketSort

If all keys are 1K Have array of K buckets (linked lists)

Put keys into correct bucket of array linear time!

BucketSort is a stable sorting algorithm:

Items in input with the same key end up in thesame order as when they began

Impractical for large K


44/62

44

RadixSort Radix = The base of a

number system(Websters dictionary) alternate terminology: radix is

number of bits needed to represent

0 to base- 1; can say base 8 orradix 3

Used in 1890 U.S.census by Hollerith

Idea: BucketSort oneach digit, bottom up.


45/62


46/62

46

Inductive Proof that RadixSort Works

Keys: K -digit numbers, base B (that wasnt hard!)

Claim: after ith BucketSort, least significant i digits

are sorted. Base case: i=0. 0 digits are sorted. Inductive step: Assume for i, prove for i+1 .

Consider two numbers: X, Y. Say X i is ith digit of X: X i+1 < Yi+1 then i+1 th BucketSort will put them in order X i+1 > Yi+1 , same thing X i+1 = Yi+1 , order depends on last i digits. Induction

hypothesis says already sorted for these digits becauseBucketSort is stable


47/62

47

Running time of Radixsort

N items, K digit keys in base B How many passes?

How much work per pass?

Total time?


48/62


49/62

49

Evaluating Sorting Algorithms

What factors other than asymptoticcomplexity could affect performance?

Suppose two algorithms perform exactly thesame number of instructions. Could one be

better than the other?


50/62

50

Example Memory Hierarchy Statistics

Name Extra CPU cyclesused to access

Size

L1 (on chip)cache

0 32 KB

L2 cache 8 512 KB

RAM 35 256 MB

Hard Drive 500,000 8 GB


51/62

51

The Memory Hierarchy Exploits

Locality of Reference Idea: small amount of fast memory Keep frequently used data in the fast memory

LRU replacement policy Keep recently used data in cache To free space, remove Least Recently Used data

Often significant practical reduction in runtime byminimizing cache misses


52/62

52

Cache Details (simplified)Main Memory

Cache

Cache line

size (4 adjacentmemory cells)


53/62

53

Iterative MergeSort

Cache Size cache misses

cache hits


54/62

54

Iterative MergeSort contd

Cache Size no temporallocality!


55/62

55

Tiled MergeSort better

Cache Size


56/62

56

Tiled MergeSort contd

Cache Size


57/62

57

Additional Cache Optimizations

TBL Padding optimizes virtual memory insert a few unused cells into array so that sub-

problems fit into separate pages of memory

Translation Lookaside Buffer Multi-MergeSort merge all tiles

simultaneously, in a big (n/tilesize) multi-waymerge

Lots of tradeoffs L1, L2, TBL cache, numberof instructions


58/62

58


59/62

59


60/62

60

Other Sorting Algorithms Quicksort - Similar cache optimizations can be

performed still slightly better than the best-tunedMergesort

Radix Sort ordinary implementation makes bad useof cache: on each BucketSort Sweep through input list cache misses along the way

( bad! ) Append to output list indexed by pseudo-random digit

(ouch! )

With a lot of work, is competitive with Quicksort


61/62

61


62/62

Conclusions

Speed of cache, RAM, and externalmemory has a huge impact on sorting (and

other algorithms as well) Algorithms with same asymptoticcomplexity may be best for different kindsof memory

Tuning algorithm to improve cache performance can offer large improvements

part3 sorting (1)

Documents