memory management during run generation in external sorting – larson & graefe
Post on 04-Jan-2016
215 Views
Preview:
TRANSCRIPT
Memory Management during Run Generation in External Sorting – Larson & Graefe
Need For External Merge Sort:• One of most often used operations • Used in joins• Used in duplicate elimination• User requests “sort”
It is important for the algorithm to degrade gracefully. The query optimizer does not know ahead the exact number of records (vs. amount of memory available).
External Sorting
• External N-way merge sort• (Usually) two phases: Run generation and
merge• During run generation, create one run at a
time (load-sort-store):1. Load records2. Sort array of pointers to records (quicksort)3. Scan pointer array while outputting records
Run Generation
• Problem:– Discontinuity in the cost function (thus it is
difficult to estimate cost of a sort operator) due to load-sort-store
– Note: That’s why (naively) hash join is preferred over sort-merge join
• Contribution:– Sorting algorithm with smooth cost function
(graceful degradation)
This Paper
• Idea– Similar to hybrid-hash: Keep parts of the
previous run in-memory
• Problem: Memory management during initial run generation
Assumptions
• Ascending sort order
• Sort must copy records into its own workspace
• No specialized disk page/file management
• Single thread of execution
• Variable-length records, no vertical partitioning of records possible
Assumptions (Contd.)
• Fixed amount of sort workspace (main memory) available, divided into extents
• An extent is a sequence of segments– Record and free segments, one bit indication in header– Each segment holds one record. – Segment type and length stored in segment header– Free segments store pointers for space management– Free segments never touch
• Separate output buffer (one or several pages for asynch. I/O)
Variable-Length Fields and Records:
• Human readable text data (non-numeric)• Even in numeric records, NULL values will
usually be not allocated• Any non-traditional data (pictures, sound)• Padding records is NOT a good idea!
Managing variable length records deterred researches from using this approach
Memory Management Algorithms
• First fit (find first space that matches)
• Next fit (same as above, but move around)
• Best fit (find closest, larger match)
Collapse free spaces
Move records
Run Formation
• First, do not output the run from memory until needed! That is, only output records from memory when need space for new record
• These ‘left over’ records can be used in first merge step of external sort – reduces IO
• Depends on the fan-in of input buffers. If fan-in maximal, there is little records left in memory for this optimization.
Replacement Selection
• If we keep track of the highest key of a record written to disk from current output, we can add next record to the current output provided its key is higher
• Pointer array no longer efficient – use priority heap (‘tree of losers’)
• Runs on average twice the size of the available memory
Result: steady IO exploits pre-sorting
Merging
• Always merge as many runs as possible
• Keep final run in memory to eliminate unnecessary IO
• Possibly reverse record order when writing to disk (not used)
Memory Management
• Next Fit– free segments implemented as linked list– first check the list– second consider moving records– finally throw current record out– example: inserting 120 byte record
100 free, N occupied, 60 free, 80 occupied
100 bytes 80 bytes
140 bytes60 bytes
120 bytes
60 bytes
Next Fit Structure
Record Segment Free Segment
Type Type
Length Length
Pointer to reference map (Forward Pointer)
Actual Record Backward Pointer
Memory Management (cont)
• Best Fit– unbalanced binary tree– go down the tree until right fit found or move
one to right– need to know neighboring segments for
merging
Best Fit Structure
Record Segment Free Segment
Type Type
Length Length
Actual Record (Forward Pointer)
Backward Pointer
(Parent Pointer)
Left Child Pointer
Right Child Pointer
Results
• Best fit memory management algorithm achieved 90% memory utilization
• Runs were 1.8x size of available memory
• Degrades Gracefully!
top related