Memory Management during Run Generation in External Sorting – Larson & Graefe

Need For External Merge Sort:• One of most often used operations • Used in joins• Used in duplicate elimination• User requests “sort”

It is important for the algorithm to degrade gracefully. The query optimizer does not know ahead the exact number of records (vs. amount of memory available).

External Sorting

• External N-way merge sort• (Usually) two phases: Run generation and

merge• During run generation, create one run at a

time (load-sort-store):1. Load records2. Sort array of pointers to records (quicksort)3. Scan pointer array while outputting records

Run Generation

• Problem:– Discontinuity in the cost function (thus it is

difficult to estimate cost of a sort operator) due to load-sort-store

– Note: That’s why (naively) hash join is preferred over sort-merge join

• Contribution:– Sorting algorithm with smooth cost function

(graceful degradation)

This Paper

• Idea– Similar to hybrid-hash: Keep parts of the

previous run in-memory

• Problem: Memory management during initial run generation

Assumptions

• Ascending sort order

• Sort must copy records into its own workspace

• No specialized disk page/file management

• Single thread of execution

• Variable-length records, no vertical partitioning of records possible

Assumptions (Contd.)

• Fixed amount of sort workspace (main memory) available, divided into extents

• An extent is a sequence of segments– Record and free segments, one bit indication in header– Each segment holds one record. – Segment type and length stored in segment header– Free segments store pointers for space management– Free segments never touch

• Separate output buffer (one or several pages for asynch. I/O)

Variable-Length Fields and Records:

• Human readable text data (non-numeric)• Even in numeric records, NULL values will

usually be not allocated• Any non-traditional data (pictures, sound)• Padding records is NOT a good idea!

Managing variable length records deterred researches from using this approach

Memory Management Algorithms

• First fit (find first space that matches)

• Next fit (same as above, but move around)

• Best fit (find closest, larger match)

Collapse free spaces

Move records

Run Formation

• First, do not output the run from memory until needed! That is, only output records from memory when need space for new record

• These ‘left over’ records can be used in first merge step of external sort – reduces IO

• Depends on the fan-in of input buffers. If fan-in maximal, there is little records left in memory for this optimization.

Replacement Selection

• If we keep track of the highest key of a record written to disk from current output, we can add next record to the current output provided its key is higher

• Pointer array no longer efficient – use priority heap (‘tree of losers’)

• Runs on average twice the size of the available memory

Result: steady IO exploits pre-sorting

Merging

• Always merge as many runs as possible

• Keep final run in memory to eliminate unnecessary IO

• Possibly reverse record order when writing to disk (not used)

Memory Management

• Next Fit– free segments implemented as linked list– first check the list– second consider moving records– finally throw current record out– example: inserting 120 byte record

100 free, N occupied, 60 free, 80 occupied

100 bytes 80 bytes

140 bytes60 bytes

120 bytes

60 bytes

Next Fit Structure

Record Segment Free Segment

Type Type

Length Length

Pointer to reference map (Forward Pointer)

Actual Record Backward Pointer

Memory Management (cont)

• Best Fit– unbalanced binary tree– go down the tree until right fit found or move

one to right– need to know neighboring segments for

merging

Best Fit Structure

Record Segment Free Segment

Type Type

Length Length

Actual Record (Forward Pointer)

Backward Pointer

(Parent Pointer)

Left Child Pointer

Right Child Pointer

Results

• Best fit memory management algorithm achieved 90% memory utilization

• Runs were 1.8x size of available memory

• Degrades Gracefully!