experimental analysis of space-bounded ... - harsha...
TRANSCRIPT
Experimental Analysis of Space-Bounded Schedulers
Harsha Vardhan SimhadriCarnegie Mellon University,
Lawrence Berkeley [email protected]
Guy E. BlellochCarnegie Mellon University
Jeremy T. FinemanGeorgetown University
Phillip B. GibbonsIntel Labs Pittsburgh
Aapo KyrolaCarnegie Mellon [email protected]
ABSTRACTThe running time of nested parallel programs on sharedmemory machines depends in significant part on how wellthe scheduler mapping the program to the machine is opti-mized for the organization of caches and processors on themachine. Recent work proposed “space-bounded schedulers”for scheduling such programs on the multi-level cache hier-archies of current machines. The main benefit of this classof schedulers is that they provably preserve locality of theprogram at every level in the hierarchy, resulting (in theory)in fewer cache misses and better use of bandwidth than thepopular work-stealing scheduler. On the other hand, com-pared to work-stealing, space-bounded schedulers are infe-rior at load balancing and may have greater scheduling over-heads, raising the question as to the relative effectiveness ofthe two schedulers in practice.
In this paper, we provide the first experimental studyaimed at addressing this question. To facilitate this study,we built a flexible experimental framework with separate in-terfaces for programs and schedulers. This enables a head-to-head comparison of the relative strengths of schedulers interms of running times and cache miss counts across a rangeof benchmarks. (The framework is validated by comparisonswith the Intel R© CilkTM Plus work-stealing scheduler.) Wepresent experimental results on a 32-core Xeon R© 7560 com-paring work-stealing, hierarchy-minded work-stealing, andtwo variants of space-bounded schedulers on both divide-and-conquer micro-benchmarks and some popular algorith-mic kernels. Our results indicate that space-bounded sched-ulers reduce the number of L3 cache misses compared towork-stealing schedulers by 25–65% for most of the bench-marks, but incur up to 7% additional scheduler and load-imbalance overhead. Only for memory-intensive benchmarkscan the reduction in cache misses overcome the added over-head, resulting in up to a 25% improvement in running timefor synthetic benchmarks and about 20% improvement foralgorithmic kernels. We also quantify runtime improvements
ACM acknowledges that this contribution was authored or co-authored by an em-ployee, contractor or affiliate of the national government of United States. As such,the Government retains a nonexclusive, royalty-free right to publish or reproduce thisarticle, or to allow others to do so, for Government purposes only.SPAA’14, June 23–25, 2014, Prague, Czech Republic.Copyright is held by the owner/author(s). Publication rights licensed to ACM.ACM 978-1-4503-2821-0/14/06 ...$15.00.http://dx.doi.org/10.1145/2612669.2612678.
varying the available bandwidth per core (the “bandwidthgap”), and show up to 50% improvements in the runningtimes of kernels as this gap increases 4-fold. As part ofour study, we generalize prior definitions of space-boundedschedulers to allow for more practical variants (while stillpreserving their guarantees), and explore implementationtradeoffs.
Categories and Subject DescriptorsD.3.4 [Processors]: Runtime environments
KeywordsThread schedulers, space-bounded schedulers, work stealing,cache misses, multicores, memory bandwidth
1. INTRODUCTIONWriting nested parallel programs using fork-join primi-
tives on top of a unified memory space is an elegant andproductive way to program parallel machines. Nested par-allel programs are portable, sufficiently expressive for manyalgorithmic problems [28, 5], relatively easy to analyze [22,3], and supported by many programming languages includ-ing OpenMP [25], Cilk++ [17], Intel R© TBB [18], Java Fork-Join [21], and Microsoft TPL [23]. The unified memoryaddress space hides from programmers the complexity ofmanaging a diverse set of physical memory components likeRAM and caches. Processor cores can access memory lo-cations without explicitly specifying their physical location.Beneath this interface, however, the real cost of accessing amemory address from a core can vary widely, depending onwhere in the machine’s cache/memory hierarchy the dataresides at time of access. Runtime thread schedulers canplay a large role in determining this cost, by optimizing thetiming and placement of program tasks for effective use ofthe machine’s caches.
Machine Models and Schedulers. Robust schedulersfor mapping nested parallel programs to machines with cer-tain kinds of simple cache organizations such as single-levelshared and private caches have been proposed. They workwell both in theory [10, 11, 8] and in practice [22, 24].Among these, the work-stealing scheduler is particularly ap-pealing for private caches because of its simplicity and lowoverheads, and is widely deployed in various run-time sys-tems such as Cilk++. The PDF scheduler [8] is suited forshared caches and practical versions of this schedule havebeen studied. The cost of these schedulers in terms of cache
misses or running times can be bounded by the locality costof the programs as measured in certain abstract program-centric cost models [10, 1, 6, 7, 29].
However, modern parallel machines have multiple levelsof cache, with each cache shared amongst a subset of cores(e.g., see Fig. 1(a)). A parallel memory hierarchy (PMH) asrepresented by a tree of caches [2] (Fig. 1(b)) is a reason-ably accurate and tractable model for such machines [16,15, 14, 6]. Because previously studied schedulers for sim-ple machine models may not be optimal for these complexmachines, recent work has proposed a variety of hierarchy-aware schedulers [16, 15, 14, 27, 6] for use on such machines.For example, hierarchy-aware work-stealing schedulers suchas PWS and HWS schedulers [27] have been proposed, butno theoretical bounds are known.
To address his gap, space-bounded schedulers [15, 14] havebeen proposed and analyzed. To use space-bounded sched-ulers, the computation needs to annotate each function callwith the size of its memory footprint. The scheduler thentries to match the memory footprint of a subcomputationto a cache of appropriate size in the hierarchy and then runthe subcomputation fully on the cores associated with thatcache. Note that although space annotations are required,the computation can be oblivious to the size of the cachesand hence is portable across machines. Under certain condi-tions these schedulers can guarantee good bounds on cachemisses at every level of the hierarchy and running time interms of some intuitive program-centric metrics. Chowd-hury et al. [15] (updated as a journal article in [14]) pre-sented such schedulers with strong asymptotic bounds oncache misses and runtime for highly balanced computations.Our follow-on work [6] presented slightly generalized sched-ulers that obtain similarly strong bounds for unbalancedcomputations.
Our Results: The First Experimental Study of Space-Bounded Schedulers. While space-bounded schedulershave good theoretical guarantees on the PMH model, therehas been no experimental study to suggest that these (asymp-totic) guarantees translate into good performance on realmachines with multi-level caches. Existing analyses of theseschedulers ignore the overhead costs of the scheduler itselfand account only for the program run time. Intuitively,given the low overheads and highly-adaptive load balancingof work-stealing in practice, space-bounded schedulers wouldseem to be inferior on both accounts, but superior in termsof cache misses. This raises the question as to the relativeeffectiveness of the two types of schedulers in practice.
This paper presents the first experimental study aimed ataddressing this question through a head-to-head comparisonof work-stealing and space-bounded schedulers. To facilitatea fair comparison of the schedulers on various benchmarks,it is necessary to have a framework that provides separatemodular interfaces for writing portable nested parallel pro-grams and specifying schedulers. The framework should belight-weight, flexible, provide fine-grained timers, and en-able access to various hardware counters for cache misses,clock cycles, etc. Prior scheduler frameworks, such as theSequoia framework [16] which implements a scheduler thatclosely resembles a space-bounded scheduler, fall short ofthese goals by (i) forcing a program to specify the specificsizes of the levels of the hierarchy it is intended for, mak-ing it non-portable, and (ii) lacking the flexibility to readilysupport work-stealing or its variants.
Memory: up to 1 TB
4 of these
24 MB
32 KB
128 KB 128 KB
8 of these
P
32 KB
P
24 MB
32 KB
128 KB 128 KB
8 of these
P
32 KB
P
L3
L2
L1
Memory: Mh = ∞, Bh
Mh−1, Bh−1 Mh−1, Bh−1 Mh−1, Bh−1 Mh−1, Bh−1
M1, B1 M1, B1 M1, B1 M1, B1 M1, B1
h
fhfh−1 . . . f1
fh
f1P P P f1
P P P f1P P P f1
P P P f1P P P
Cost: Ch−1
Cost: Ch−2
(a) 32-core Xeon R© 7560 (b) PMH model of [2]
Figure 1: Memory hierarchy of a current generationarchitecture from Intel R©, plus an example abstractparallel hierarchy model. Each cache (rectangle) isshared by all cores (circles) in its subtree.
This paper describes a scheduler framework that we de-signed and implemented, which achieves these goals. Tospecify a (nested-parallel) program in the framework, theprogrammer uses a Fork-Join primitive (and a Parallel-Forbuilt on top of Fork-Join). To specify the scheduler, oneneeds to implement just three primitives describing the man-agement of tasks at Fork and Join points: add, get, anddone. Any scheduler can be described in this framework aslong as the schedule does not require the preemption of se-quential segments of the program. A simple work-stealingscheduler, for example, can be described with only 10s oflines of code in this framework. Furthermore, in this frame-work, program tasks are completely managed by the sched-ulers, allowing them full control of the execution.
The framework enables a head-to-head comparison of therelative strengths of schedulers in terms of running timesand cache miss counts across a range of benchmarks. (Theframework is validated by comparisons with the commercialCilkTM Plus work-stealing scheduler.) We present experi-mental results on a 32-core Intel R© Nehalem series Xeon R©7560 multicore with 3 levels of cache. As depicted in Fig. 1(a),each L3 cache is shared (among the 8 cores on a socket)while the L1 and L2 caches are exclusive to cores. We com-pare four schedulers—work-stealing, priority work-stealing(PWS) [27], and two variants of space-bounded schedulers—on both divide-and-conquer micro-benchmarks (scan-basedand gather-based) and popular algorithmic kernels such asquicksort, sample sort, matrix multiplication, and quad trees.
Our results indicate that space-bounded schedulers re-duce the number of L3 cache misses compared to work-stealing schedulers by 25–65% for most of the benchmarks,while incurring up to 7% additional overhead. For memory-intensive benchmarks, the reduction in cache misses over-comes the added overhead, resulting in up to a 25% im-provement in running time for synthetic benchmarks andabout 20% improvement for algorithmic kernels. To bet-ter understand how the widening gap between processingpower (cores) and memory bandwidth impacts schedulerperformance, we quantify runtime improvements over a 4-fold range in the available bandwidth per core and showfurther improvements in the running times of kernels (up to50%) as the bandwidth gap increases.
Finally, as part of our study, we generalize prior defini-tions of space-bounded schedulers to allow for more practicalvariants, and explore implementation tradeoffs, e.g., in a keyparameter of such schedulers. This is useful for engineeringspace-bounded schedulers, which were previously described
only at a high level suitable for theoretical analyses, into aform suitable for real machines.
Contributions. The contributions of this paper are:
• A modular framework for describing schedulers, ma-chines as tree of caches, and nested parallel programs(Section 3). The framework is equipped with timersand counters. Schedulers that are expected to workwell on tree of cache models such space-bounded sched-ulers and certain work-stealing schedulers are imple-mented.
• A precise definition for the class of space-bounded sched-ulers that retains the competitive cache miss boundsexpected for this class, but also allows more sched-ulers than previous definitions (which were motivatedmainly by theoretical guarantees [15, 14, 6]) (Section 4).We describe two variants, highlighting the engineeringdetails that allow for low overhead.
• The first experimental study of space-bounded sched-ulers, and the first head-to-head comparison with work-stealing schedulers (Section 5). On a common multi-core machine configuration (4 sockets, 32 cores, 3 lev-els of caches), we quantify the reduction in L3 cachemisses incurred by space-bounded schedulers relativeto both work-stealing variants on synthetic and non-synthetic benchmarks. On bandwidth-bound bench-marks, an improvement in cache misses translates toimprovement in running times, although some of theimprovement is eroded by the greater overhead of thespace-bounded scheduler.
2. DEFINITIONSWe start with a recursive definition of nested parallel com-
putation, and use it to define what constitutes a schedule.We will then define the parallel memory hierarchy (PMH)model—a machine model that reasonably accurately repre-sents shared memory parallel machines with deep memoryhierarchies. This terminology will be used later to defineschedulers for the PMH model.
Computation Model, Tasks and Strands. We con-sider computations with nested parallelism, allowing arbi-trary dynamic nesting of fork-join constructs including par-allel loops, but no other synchronizations. This correspondsto the class of algorithms with series-parallel dependencegraphs (see Fig. 2(left)).
Nested parallel computations can be decomposed into“tasks”,“parallel blocks” and “strands” recursively as follows. As abase case, a strand is a serial sequence of instructions notcontaining any parallel constructs or subtasks. A task isformed by serially composing k ≥ 1 strands interleaved with(k − 1) “parallel blocks”, denoted by t = `1; b1; . . . ; `k. Aparallel block is formed by composing in parallel one ormore tasks with a fork point before all of them and a joinpoint after (denoted by b = t1‖t2‖ . . . ‖tk). A parallel blockcan be, for example, a parallel loop or some constant num-ber of recursive calls. The top-level computation is a task.We use the notation L(t) to indicate all strands that arerecursively included in a task.
Our computation model assumes all strands share a singlememory address space. We say two strands are concurrentif they are not ordered in the dependence graph. Concurrentreads (i.e., concurrent strands reading the same memorylocation) are permitted, but not data races (i.e., concurrent
f
Task
Strand
Block
f’Parallel
Lifetime of a task.
Lifetime of a strand
the task to the left.nested immediately within
spawn at the same time
start at the same time
end
end
live
queued
executing
Figure 2: (left) Decomposing the computation:tasks, strands and parallel blocks. f, f ′ are cor-responding fork and join points. (right) Timelineof a task and its first strand, showing the differencebetween being live and execution.
strands that read or write the same location with at leastone write). For every strand `, there exists a task t(`) suchthat ` is nested immediately inside t(`). We call this thetask of strand `.
Schedule. We now define what constitutes a valid sched-ule for a nested parallel computation on a machine. Thesedefinitions will enable us to precisely define space-boundedschedulers later on. We restrict ourselves to non-preemptiveschedulers—schedulers that cannot migrate strands acrosscores once they begin executing. Both work-stealing andspace-bounded schedulers are non-premptive. We use P todenote the set of cores on the machine, and L to denotethe set of strands in the computation. A non-preemptiveschedule defines three functions for each strand `.
• Start time: start : L→ Z, where start(`) denotes thetime the first instruction of ` begins executing;
• End time: end : L → Z, where end(`) denotes the(post-facto) time the last instruction of ` finishes; and
• Location: proc : L → P , where proc(`) denotes thecore on which the strand is executed. Note that proc iswell defined because of the non-preemptive policy forstrands.
We say that a strand ` is live at any time τ with start(`) ≤τ < end(`).
A non-preemptive schedule must also obey the followingconstraints on the ordering of strands and timing:
• (ordering): For any strand `1 ordered by the fork-joindependence graph before `2: end(`1) ≤ start(`2).
• (processing time): For any strand `, end(`) = start(`)+γ〈schedule,machine〉(`). Here γ denotes the processingtime of the strand, which may vary depending on thespecifics of the machine and the history of the sched-ule. The schedule alone does not control this value.
• (non-preemptive execution): No two strands may belive on the same core at the same time, i.e., `1 6=`2, proc(`1) = proc(`2) =⇒ [start(`1), end(`1)) ∩[start(`2), end(`2)) = ∅.
We extend the same notation and terminology to tasks.The start time start(t) of a task t is a shorthand for start(t) =start(`s), where `s is the first strand in t. Similarly end(t)denotes the end time of the last strand in t. The functionproc, however, is undefined for tasks as a task’s containedstrands may execute on different cores.
When discussing specific schedulers, it is convenient toconsider the time a task or strand first becomes availableto execute. We use the term spawn time to refer to thistime, which is the instant at which the preceding fork orjoin finishes. Naturally, the spawn time is no later than thestart time, but a schedule may choose not to execute thetask or strand immediately. We say that the task or strandis queued during the time between its spawn time and starttime and live during the time between its start time and endtime. Fig. 2(right) illustrates the spawn, start and end timesof a task and its initial strand. The task and initial strandare spawned and start at the same time by definition. Thestrand is continuously executed until it ends, while a taskgoes through several phases of execution and idling beforeit ends.
Machine Model: Parallel Memory Hierarchy (PMH).Following prior work addressing multi-level parallel hierar-chies [2, 12, 4, 13, 31, 9, 15, 14, 6], we model parallel ma-chines using a tree-of-caches abstraction. For concreteness,we use a symmetric variant of the parallel memory hierar-chy (PMH) model [2] (see Fig. 1(b)), which is consistentwith many other models [4, 9, 12, 13, 15, 14]. A PMH con-sists of a height-h tree of memory units, called caches. Theleaves of the tree are at level-0 and any internal node haslevel one greater than its children. The leaves (level-0 nodes)are cores, and the level-h root corresponds to an infinitelylarge main memory. As described in [6] each level in the treeis parameterized by four parameters: the size of the cacheM i at level i, the block size Bi used to transfer to the nexthigher level, the cost of a cache miss Ci which represents thecombined costs of latency and bandwidth, and the fanout fi(number of level i− 1 caches below it).
3. EXPERIMENTAL FRAMEWORKWe implemented a C++ based framework with the fol-
lowing design objectives in which nested parallel programsand schedulers can be built for shared memory multicoremachines. The implementation, along with a few schedulersand algorithms, is available on the web page http://www.
cs.cmu.edu/~hsimhadr/sched-exp. Some of the code forthe threadpool module has been adapted from an earlierimplementation of threadpool [20].
Modularity: The framework separates the specification ofthree components—programs, schedulers, and description ofmachine parameters—for portability and fairness. The usercan choose any of the candidates from these three categories.Note, however, some schedulers may not be able to executeprograms without scheduler-specific hints (such as space an-notations).
Clean Interface: The interface for specifying the com-ponents should be clean, composable, and the specificationbuilt on the interface should be easy to reason about.
Hint Passing: While it is important to separate programand schedulers, it is useful to allow the program to passhints (extra annotations on tasks) to the scheduler to guideits decisions.
Minimal Overhead: The framework itself should be light-weight with minimal system calls, locking and code com-plexity. The control flow should pass between the func-tional modules (program, scheduler) with negligible timespent outside. The framework should avoid generating back-ground memory traffic and interrupts.
Framework
Fork
Thread
Scheduler (concurrent)
Each thread is bound to one core
Nested Parallel Program Join
run
add done get
Job
Shared structures (Queues…)
Program code Concurrent code in scheduler module
Figure 3: Interface for the program and schedulingmodules.
Timing and Measurement: It should enable fine-grainedmeasurements of the various modules. Measurements in-clude not only clock time, but also insightful hardware coun-ters such as cache and memory traffic statistics. In light ofthe earlier objective, the framework should avoid OS systemcalls for these, and should use direct assembly instructions.
3.1 InterfaceThe framework has separate interfaces for the program
and the scheduler.
Programs: Nested parallel programs, with no other syn-chronization primitives, are composed from tasks using fork
and join constructs. A parallel_for primitive built withfork and join is also provided. Tasks are implemented asinstances of classes that inherit from the Job class. Differ-ent kinds of tasks are specified as classes with a methodthat specifies the code to be executed. An instance of aclass derived from Job is a task containing a pointer to astrand nested immediately within the task. The control flowof this function is sequential with a terminal fork or join
call. (This interface could be readily extended to handlenon-nested parallel constructs such as futures [30] by addingother primitives to the interface beyond fork and join.)
The interface allows extra annotations on a task such as itssize, which is required by space-bounded schedulers. Suchtasks inherit a derived class of Job class, the extensions in thederived class specifying the annotations. For example, theclass SBJob suited for space-bounded schedulers is derivedfrom Job by adding two functions—size(uint block_size)
and strand_size(uint block_size)—that allow the anno-tations of the job size.
Scheduler: The scheduler is a concurrent module that han-dles queued and live tasks (as defined in Section 2) and isresponsible for maintaining its own queues and other inter-nal shared data structures. The module interacts with theframework that consists of a thread attached to each pro-cessing core on the machine, through an interface with threecall-back functions.
• Job* get (ThreadIdType): This is called by the frame-work on behalf of a thread attached to a core whenthe core is ready to execute a new strand, after com-pleting a previously live strand. The function maychange the internal state of the scheduler module andreturn a (possibly null) Job so that the core may im-mediately begin executing the strand. This functionspecifies proc for the strand.• void done(Job*,ThreadIdType) This is called when a
core finishes the execution of a strand. The scheduler
is allowed to update its internal state to reflect thiscompletion.
• void add(Job*,ThreadIdType): This is called when afork or join is encountered. In case of a fork, thiscall-back is invoked once for each of the newly spawnedtasks. For a join, it is invoked for the continuationtask of the join. This function decides where to en-queue the job.
Other auxiliary parameters to these call-backs have beendropped from the above description for clarity and brevity.The Job* argument passed to these functions may be in-stances of one of the derived classes of Job* that carry ad-ditional information helpful to the scheduler. Appendix Apresents an example of a work-stealing scheduler implementedin this scheduler interface.
Machine configuration: The interface for specifying ma-chine descriptions accepts a description of the cache hierar-chy: number of levels, fanout at each level, and cache andcache-line size at each level. In addition, a mapping betweenthe logical numbering of cores on the system to their left-to-right position as a leaf in the tree of caches must be specified.For example, Fig. 4 is a description of one Nehalem-EX se-ries 4-socket × 8-core machine (32 physical cores) with 3levels of caches as depicted in Fig. 1(a).
3.2 ImplementationThe runtime system initially fixes a POSIX thread to each
core. Each thread then repeatedly performs a call (get)to the scheduler module to ask for work. Once assigned atask and a specific strand inside it, the thread completes thestrand and asks for more work. Each strand either ends ina fork or a join. In either scenario, the framework invokesthe done call back. For a fork, the add call-back is invokedto let the scheduler add new tasks to its data structures.
All specifics of how the scheduler operates (e.g., how thescheduler handles work requests, whether it is distributedor centralized, internal data structures, where mutual ex-clusion occurs, etc.) are relegated to scheduler implementa-tions. Outside the scheduling modules, the runtime systemincludes no locks, synchronization, or system calls (exceptduring the initialization and cleanup of the thread pool),meeting our design objective.
3.3 MeasurementsActive time and overheads: Control flow on each threadmoves between the program and the scheduler modules. Fine-grained timers in the framework break down the executiontime into five components: (i) active time—the time spentexecuting the program, (ii) add overhead, (iii) done over-head, (iv) get overhead, and (v) empty queue overhead.While active time depends on the number of instructionsand the communication costs of the program, add, done andget overheads depend on the complexity of the scheduler,and the number of times the scheduler code is invoked byforks and joins. The empty queue overhead is the amountof time the scheduler fails to assign work to a thread (getreturns null), and reflects on the load balancing capability ofthe scheduler. In most of the results in Section 5, we usuallyreport two numbers: active time averaged over all threadsand the average overhead, which includes measures (ii)–(v).Note that while we might expect this partition of time to beindependent, it is not so in practice—the background coher-
int num_procs=32;int num_levels = 4;int fan_outs[4] = {4,8,1,1};long long int sizes[4] = {0, 3*(1<<22), 1<<18, 1<<15};int block_sizes[4] = {64,64,64,64};int map[32] = {0,4,8,12,16,20,24,28,
2,6,10,14,18,22,26,30,1,5,9,13,17,21,25,29,3,7,11,15,19,23,27,31};
Figure 4: Specification entry for a 32-core Xeon R©machine depicted in Fig. 1(a).
ence traffic generated by the scheduler’s bookkeeping mayadversely affect active time. The timers have very little over-head in practice—less than 1% in most of our experiments.
Measuring hardware counters: Modern multicores areequipped with hardware counters that can provide variousperformance statistics such as the number of cache misses atvarious levels. Such counters, however, are somewhat chal-lenging to use. Appendix B details the specific methodologywe used for the Intel R© Nehalem architecture.
4. SCHEDULERSIn this section, we will define the class of space-bounded
schedulers and describe the schedulers we compare using ourframework.
4.1 Space-bounded SchedulersSpace-bounded schedulers are designed to achieve good
cache performance on PMHs and use the sizes of tasks andstrands to choose a schedule (start , end , proc) mapping thehierarchy of tasks in a nested parallel program to the hierar-chy of caches. They have been described previously in [15]and [6]. The definitions in [15] are not complete in that theydo not specify how to handle “skip level” tasks (describedbelow). Our earlier definition [6] handled skip level tasks,but was more restrictive than necessary, ruling out practicalvariants, as discussed below. Here, we provide a broader def-inition for the class of space-bounded schedulers that allowsfor practical variants, while retaining the strong analyticalbounds on cache misses that are the hallmark of the class ofspace-bounded schedulers. Specifically, our new definitionprovides more flexibility in the scheduling of strands by (i)allowing each strand to have its own size, and (ii) accountingfor strand sizes differently from task sizes.
Informally, a space-bounded schedule satisfies two proper-ties: (i) Anchored: Each task t gets “anchored” to a smallestpossible cache that is larger than its size—strands withint can only be scheduled on cores in the tree rooted at thecache; and (ii) Bounded: At any point in time, the sumof the sizes of all tasks and strands occupying space in acache is at most the size of the cache. These two condi-tions (when formally defined) are sufficient to imply strongbounds on the number of cache misses at every level in thetree of caches. A good space-bounded scheduler would alsohandle load balancing subject to anchoring constraints toquickly complete execution.
More formally, a space-bounded scheduler is parameter-ized by a global dilation parameter 0 < σ ≤ 1 and machineparameters {Mi, Bi, Ci, fi}. We will need the following ter-minologies for the definition (which are both simplified andgeneralized from [6]).
Chowdhury et. al. [15] also suggest two other approachesto scheduling: CGC (coarse-grained contiguous) and CGC-on-SB (which combines CGC with space bounded sched-ulers). CGC is designed to keep nearby blocks of iterationsclose together so they are run on the same cache. This canbe simulated in our framework by grouping iterations recur-sively (which is what we do). This means that our approachwill not have constant critical path length for certain algo-rithms, but experimentally we are using a modest numberof cores. CGC-on-SB is primarily a mechanism for skippinglevels in the cache. Our variant of space-bounded scheduleralready allows for level skipping.
Task Size and Strand Size: The size of a task (strand) isdefined as a function of cache-line size B, independent of thescheduler. Let loc(t;B) denote the set of distinct cache linestouched by instructions within a task t. Then S(t;B) =|loc(t;B)| · B denotes the size of t. The size of a strand isdefined in the same way. While results in [6] show that itis not necessary for the analytical bounds that strands beallowed to have their own size, we found that the flexibilityit enables is an important running time optimization.1 Notethat even strands immediately nested within the same taskmay have different sizes.
Cluster: For any cache Xi, its cluster is the set of cachesand cores nested below Xi. Let P (Xi) denote the set ofcores in Xi’s cluster.
Befitting Cache: Given a particular cache hierarchy anddilation parameter σ ∈ (0, 1], we say that a level-i cachebefits a task t if σMi−1 < S(t, Bi) ≤ σMi.
Maximal Task: We say that a task t with parent task t′
is level-i maximal if and only if a level-i cache befits t butnot t′, i.e., σMi−1 < S(t, Bi) ≤ σMi < S(t′, Bi).
Anchored: A task t with strand set L(t) is said to be an-chored to level-i cache Xi (or equivalently to Xi’s cluster)if and only if (i) it is executed entirely in the cluster, i.e.,{proc(`)|` ∈ L(t)} ⊆ P (Xi), and (ii) the cache befits thetask. Anchoring prevents the migration of tasks to a differ-ent cluster or cache. The advantage of anchoring a task toa befitting cache is that once it loads its working set, it canreuse the working set without the risk of losing it from thecache. If a task is not anchored anywhere, for notationalconvenience we assume it is anchored at the root of the tree.
Cache-occupying tasks: The definition depends on whetherthe cache is inclusive or non-inclusive. For a level-i inclusivecache Xi and time τ , the set of cache-occupying tasks fora cache Xi at time τ , denoted by Ot(Xi, τ), is the unionof (a) the maximal tasks live at time τ that are anchoredto Xi, and (b) the maximal tasks live at time τ that areanchored to any cache below Xi in the hierarchy whose im-mediate parents are anchored to a cache above Xi in thehierarchy. The tasks in (b) are called “skip level” tasks. Fora non-inclusive cache, only type (a) tasks are included inOt(Xi, τ). Tasks in Ot(Xi, τ) are the tasks that consumespace in the cache at time τ . Note that we need accountonly for maximal tasks because any non-maximal task t′ isanchored to the same cache as its closest enclosing maximaltask t and loc(t′;B) ⊆ loc(t;B).
1On the other hand, it does require additional size informa-tion on programs—thus we view it as optional: Any strandwhose size is not specified is assumed by default to be thesize of its enclosing task.
Cache-occupying strands: The set of cache-occupyingstrands for a cache Xi at time τ , denoted by Ol(Xi, τ), isthe set of strands {`} such that (a) ` is live at time τ (b) `is executed below Xi, i.e., proc(`) ∈ P (Xi), and (c) `’s taskis anchored strictly above Xi.
A space-bounded scheduler for a particular cache hier-archy is a scheduler parameterized by σ ∈ (0, 1] that satisfiesthe following two properties:
• Anchored: Every subtask (recursively) of the root taskis anchored to a befitting cache.• Bounded: At every time τ , for every level-i cache Xi,
the sum of the sizes of cache-occupying tasks and strandsis at most Mi:
Σt∈Ot(Xi,τ)S(t, Bi) + Σ
`∈Ol(Xi,τ)S(`, Bi) ≤Mi.
A key property of the definition of space-bounded sched-ulers in [6] is that for any PMH and any level i, one canupper bound the number of level-i cache misses incurred byexecuting a task on the PMH by Q∗(t;σMi, Bi), where Q∗
is the cache complexity as defined in the Parallel Cache-Oblivious (PCO) model in [6]. Roughly speaking, the cachecomplexity Q∗ of a task t in terms of a cache of size M andline size B is defined as follows. Decompose the task into acollection of maximal subtasks that fit in M space, and“gluenodes” – instructions outside these subtasks. For a maximalsize M task t′, the PCO cache complexity Q∗(t′;M ;B) isdefined to be the number of distinct cache lines it accesses,counting accesses to a cache line from unordered instruc-tions multiple times. The model then pessimistically countsall memory instructions that fall outside of a maximal sub-task (i.e., glue nodes) as cache misses. The total cache com-plexity of an algorithm is the sum of the complexities ofthe maximal subtasks, and the memory accesses outside ofmaximal subtasks. Note that Q∗ is a program-centric ormachine-independent metric, capturing the inherent local-ity in a parallel algorithm [7, 29].
Theorem 1. Consider a PMH and any dilation param-eter 0 < σ ≤ 1. Let t be a task anchored to the root ofthe tree. Then the number of level-i cache misses incurredby executing t with any space-bounded scheduler is at mostQ∗(t;σMi, Bi), where Q∗ is the cache complexity as definedin the Parallel Cache-Oblivious (PCO) model in [6].
The proof is a simple adaptation of the proof in [6] toaccount for our more general definition. At a high level, theargument is that for any cache Xi, the cache-occupying tasksand strands for Xi bring in their working sets into the cacheXi exactly once because the boundedness property preventscache overflows. Thus a replacement policy that keeps theseworking sets in the cache until they are no longer needed willincur no additional misses; because the PMH model assumesan ideal replacement policy, it will perform at least as well.
In the course of our experimental study, we found that thefollowing minor modification to the boundedness property ofspace-bounded schedulers improves their performance. Namely,we introduce a new parameter µ ∈ (0, 1] (µ = 0.2 in our ex-periments) and modify the boundedness property to be suchthat at every time τ , for every level-i cache Xi:
Σt∈Ot(Xi,τ)S(t, Bi)+Σ
`∈Ol(Xi,τ)min{µMi, S(`, Bi)} ≤Mi,
The minimum term with µMi is to allow several large strandsto be explored simultaneously without their space measure
taking too much of the space bound. This helps the sched-uler to quickly traverse the higher levels of recursion in theDAG and reveal parallelism so that the scheduler can achievebetter load balance.
Given this modified boundedness condition, it is easy toshow that the bound in Theorem 1 becomesQ∗(t;µσMi, Bi).
Note that while setting σ to 1 yields the best bounds oncache misses, it also makes load balancing harder. As we willsee later, a lower value for σ like 0.5 allows greater schedulingflexibility.
4.2 Schedulers implementedSpace-bounded schedulers: SB and SB-D. We imple-mented a space-bounded scheduler by constructing a treeof caches based on the specification of the target machine.Each cache is assigned one logical queue, a counter to keeptrack of “occupied” space and a lock to protect updates tothe counter and queue. Cores can be considered to be leavesof the tree; when a scheduler call-back is issued to a thread,that thread can modify an internal node of the tree aftergathering all locks on the path to the node from the coreit is mapped onto. This scheduler accepts Jobs which areannotated with task and strand sizes. When a new Job isspawned at a fork, the add call-back enqueues it at the clus-ter where its parent was anchored. For a new Job spawnedat a join, add enqueues it at the cluster where the Job thatcalled the corresponding fork of this join was anchored.
A basic version of such a scheduler would implement logi-cal queues at each cache as one queue. However, this presentstwo problems: (i) It is difficult to separate tasks in queuesby the level of cache that befits it, and (ii) a single queuemight be a contention hotspot. To solve problem (i), behindeach logical queue, we use separate “buckets” for each levelof cache below to hold tasks that befit those levels. Coreslooking for a task at a cache go through these buckets fromthe top (heaviest tasks) to bottom. We refer to this variantas the SB scheduler. To solve problem (ii) involving queue-ing hotspots, we replace the top bucket with a distributedqueue—one queue for each child cache—like in the work-stealing scheduler. We refer to the SB scheduler with thismodification as the SB-D scheduler.
Work-Stealing scheduler: WS. A basic work-stealingscheduler based on Cilk++ [11] is implemented and is re-ferred to as the WS scheduler. Since the Cilk++ runtimesystem is built around work-stealing and deeply integratedand optimized exclusively for it, we focus our comparison onthe WS implementation in our framework for fairness andto allow us to implement variants. The head-to-head micro-benchmark study in Section 5 between our WS implemen-tation and CilkTM Plus (the commercial version of Cilk++)suggests that, for these benchmarks, WS well-representsthe performance of Cilk++’s work-stealing. We associatea double-ended queue (dequeue) of ready tasks with eachcore. The function add enqueues new tasks (or strands)spawned on a core to the bottom of its dequeue. When inneed of work, the core uses get to remove a task from thebottom of its dequeue. If its dequeue is empty, it choosesanother dequeue uniformly at random, and steals the workby removing a task from the top of that core’s dequeue. Theonly contention in this type of scheduler is on the distributeddequeues—there is no other centralized data structure.
To implement the dequeues, we employed a simple two-locks-per-dequeue approach, one associated with the owning
core, and the second associated with all cores currently at-tempting to steal. Remote cores need to obtain the secondlock before they attempt to lock the first. Contention isthus minimized for the common case where the core needsto obtain only the first lock before it asks for work from itsown dequeue.
Priority Work-Stealing scheduler: PWS. Unlike in thebasic WS scheduler, cores in the PWS scheduler [27] choosevictims of their steals according to the “closeness” of the vic-tims in the socket layout. Dequeues at cores that are closerin the cache hierarchy are chosen with a higher probabil-ity than those that are farther away to improve schedul-ing locality while retaining the load balancing properties ofWS scheduler. On our 4 socket machines, we set the prob-ability of an intra-socket steal to be 10 times that of aninter-socket steal.
5. EXPERIMENTSThe goal of our experimental study is to compare the per-
formance of the four schedulers on a range of benchmarks,varying the available memory bandwidth. Our primary met-rics are runtime and L3 (last level) cache misses. We havefound that the cache misses on other levels do not vary sig-nificantly among the schedulers (within 5%). We will alsovalidate our work-stealing implementation via a compari-son with the commercial CilkTM Plus scheduler. Finally,we will quantify the overheads for space-bounded schedulersand study the performance impact of the key parameter σfor space-bounded schedulers.
5.1 BenchmarksWe use seven benchmarks in our study. The first two
are synthetic micro-benchmarks that mimic the behavior ofmemory-intensive divide-and-conquer algorithms. Becauseof their simplicity, we use these benchmarks to closely ana-lyze the behavior of the schedulers under various conditionsand verify that we get the expected cache behavior on areal machine. The remaining five benchmarks are a set ofpopular algorithmic kernels.
Recursive repeated map (RRM): This benchmark takestwo n-length arrays A and B and a point-wise map functionthat maps elements of A to B. In our experiments eachelement of the arrays is a double and the function simplyadds one. RRM first does a parallel point-wise map fromA to B, and repeats the same operation multiple times. Itthen divides A and B into two by some ratio (e.g., 50/50)and recursively calls the same operation on each of the twoparts. The base case of the recursion is set to some constantat which point the recursion terminates. The input parame-ters are the size of the arrays n, number of repeats r, the cutratio f , and the base-case size. We set r = 3 (the numberof repeats in quicksort), and f = 50% in the experimentsunless mentioned otherwise. RRM is a memory intensivebenchmark because there is very little work done per mem-ory operation. However, once a recursive call fits in a cache(i.e., the cache size is at least 16n bytes for subproblem sizen), all remaining accesses are cache hits.
Recursive repeated gather (RRG): This benchmark issimilar to RRM but instead of doing a simple map it doesa gather. In particular at any given level of the recursion ittakes three n-length arrays A, B and I and for each locationi sets B[i] = A[I[i] mod n]. The values in I are random in-
tegers. As with RRM after repeating r times it splits thearrays in two and repeats on each part. RRG is even morememory intensive than RRM because its accesses are ran-dom instead of linear. Again, however, once a recursive callfits in a cache all remaining accesses are cache hits.
Quicksort: This is a parallel quicksort algorithm that bothparallelizes the partitioning and the recursive calls, using amedian-of-3 pivot selection strategy. It switches to a versionwhich parallelizes only the recursive calls for n < 128K anda serial version for n < 16K. These parameters worked wellfor all the schedulers. We note that our quicksort is about2x faster than the Cilk code found in the Cilk+ guide [22].This is because it does the partitioning in parallel. It is alsothe case that the divide does not exactly partition the dataevenly, because it depends on how well the pivot divides thedata. For an input of size n the program has cache com-plexity Q∗(n;M,B) = O(dn/Belog2(n/M)) and therefore isreasonably memory intensive.
Samplesort: This is cache-optimal parallel Sample Sort al-gorithm described in [9]. The algorithm splits the input ofsize n into
√n subarrays, recursively sorts each subarray,
“block transposes” them into√n buckets and recursively
sorts these buckets. For this algorithm, Q∗(n;M,B) =O(dn/Be log2+(M/B) n/B) making it relatively cache friendly,and optimally cache oblivious even.
Aware samplesort: This is a variant of sample-sort that isaware of the cache sizes. In particular it moves elements intobuckets that fit into the L3 cache and then runs quicksort onthe buckets. This is the fastest sort we implemented and isin fact faster than any other sort we found for this machine.In particular it is about 10% faster than the PBBS samplesort [28].
Quad-tree: This generates a quad tree for a set of n pointsin two dimensions. This is implemented by recursively par-titioning the points into four sets along the mid line of eachof two dimensions. When the number of points is less than16K we revert to a sequential algorithm.
Matrix multiplication: This benchmark is an 8-way re-cursive matrix multiplication. To allow for an in-place im-plementation, four of the recursive calls are invoked in par-allel followed by the other four. Matrix multiplication hascache complexity Q∗(n;M,B) = (dn2/Be × dn/
√Me). The
ratio of instructions to cache misses is therefore very high,about B
√M , making this is a very compute-intensive bench-
mark. We switch to serial Intel R© Math Kernel Library’scblas_dgemm matrix multiplication for sizes of ≤ 128× 128to make use of the floating point SIMD operations.
5.2 Experimental SetupThe benchmarks were run on a 4-socket 32-core Xeon R©
7560 machine (Nehalem-EX architecture), as described inFig. 1(a) and Fig. 4, using each of the four schedulers. Eachcore uses 2-way hyperthreading. The last level cache on eachsocket is a 24MB L3 cache that is shared by eight cores. Eachof the four sockets on the machine has memory links to dis-tinct DRAM modules. The sockets are connected with theIntel R© QPI interconnect. Memory requests from a socket toa DRAM module connected to another socket pass throughthe QPI, the remote socket, and the remote memory link. Toprevent excessive TLB cache misses, we use Linux hugepagesof size 2MB to pre-allocate the space required by the algo-rithms. We configured the system to have a pool of 10, 000
huge pages by setting vm.nr_hugepages to that value us-ing sysctl. We used the hugectl tool to execute memoryallocations with hugepages.
Monitoring L3 cache. We focus on L3 cache misses, themost expensive cache level before DRAM on our machine.While we will report runtime subdivided into applicationcode and scheduler code, such partitioning was not possi-ble for L3 misses because of software limitations. Even if itwere possible to count them separately, it would be difficultto interpret the results because of the non-trivial interfer-ence between the data cached by the program and by thescheduler. Further details can be found in Appendix B.
Controlling bandwidth. As part of our study, we willquantify runtime improvements varying the available mem-ory bandwidth per core (the “bandwidth gap”). We controlthe memory bandwidth available to the program as follows.Because the QPI has high bandwidth, if we treat the RAMas a single functional module, the bandwidth between theL3 and the RAM depends on the number of memory linksused, which in turn depends on the mapping of pages to theDRAM modules. If all the pages used by a program aremapped to DRAM modules connected to one socket, theprogram effectively utilizes one-fourth of the memory band-width. On the other hand, an even distribution of pages toDRAM modules across the sockets provides the full band-width to the program. The numactl tool can be used tocontrol this mapping.
Code. We use the same code for the applications/algo-rithms for all benchmarks, except for the CilkPlus code,which we kept as close as possible. The code always includesthe space annotations, but those annotations are ignored bythe schedulers that do not need them. The code was com-piled with a CilkPlus fork of gcc 4.8.0 compiler.
5.3 ResultsWe use σ = 0.5 and µ = 0.2 in the SB and SB-D sched-
ulers, after some experimentation with the parameters, un-less otherwise noted. All numbers reported in this paper arethe average of at least 10 runs with the smallest and largestreadings across runs removed. The results are not sensitiveto this particular choice of reporting.
Synthetic benchmarks. Fig. 5 and Fig. 6 show the num-ber of L3 cache misses of RRM and RRG, respectively, alongwith their active times and scheduling overheads on 64 hy-perthreads at different bandwidth values. In addition to thefour schedulers discussed in Section 4.2 (WS, PWS, SB andSB-D), we include the CilkPlus work-stealing scheduler inthese plots to validate our WS implementation. We couldnot separate overhead from time in CilkPlus because it doesnot supply such information.
These plots show that the space-bounded schedulers in-cur roughly 42–44% fewer L3 cache misses than the work-stealing schedulers. As expected, the number of L3 missesdoes not significantly depend on the available bandwidth.On the other hand, the active time is most influenced by thenumber of instructions in the benchmark (constant acrossschedulers) and the costs incurred by L3 misses. The ex-tent to which improvements in L3 misses translates to animprovement in active time depends on the memory band-width given to the program. When the bandwidth is low(25%), the active times are almost directly proportional to
0
10
20
30
40
50
60
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
Cilk Plus
WS PWS SB SB-‐D Cilk Plus
WS PWS SB SB-‐D Cilk Plus
WS PWS SB SB-‐D Cilk Plus
WS PWS SB SB-‐D
100% b/w 75% b/w 50% b/w 25% b/w
Ac>ve Time (seconds) Overhead (seconds) L3 Cache misses (millions)
Figure 5: RRM on 10 million double elements, vary-ing the memory bandwidth. Left axis is runningtime in seconds. Right axis is L3 misses in millions.
0
10
20
30
40
50
60
70
80
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Cilk Plus
WS PWS SB SB-‐D Cilk Plus
WS PWS SB SB-‐D Cilk Plus
WS PWS SB SB-‐D Cilk Plus
WS PWS SB SB-‐D
100% b/w 75% b/w 50% b/w 25% b/w
Ac?ve Time (seconds) Overhead (seconds) L3 Cache misses (millions)
Figure 6: RRG on 10 million double elements, vary-ing the memory bandwidth. Left axis is runningtime in seconds. Right axis is L3 misses in millions.
the number of L3 cache misses, while at full bandwidth, theactive times are far less sensitive to misses.
The differences in L3 cache costs of space-bounded andwork-stealing schedulers roughly corresponds to the differ-ence between the cache complexity of the program with acache of size σM3 (M3 being the size of L3) and a cache ofsize M3/16 (because eight cores with sixteen hyperthreadsshare an L3 cache). In other words, space-bounded sched-ulers share the cache constructively while work-stealing sched-ulers effectively split the cache between the cores. To seethis, consider our RRM benchmark. Each Map operationthat does not fit in L3 touches 2 × 107 × 8 = 160 millionbytes of data, and RRM has to unfold four levels of recur-sion before it fits in σM3 = 0.5× 24MB= 12MB space withspace-bounded schedulers. Therefore, since the cache linesize B3 = 64 bytes, space-bounded schedulers incur about(160×106×3×4)/64 = 30×106 cache misses, which matchesclosely with the results in Fig. 5. On the other hand, thenumber of cache misses of the WS scheduler (55 million)corresponds to unfolding about 7 levels of recursions, threemore than with space-bounded schedulers. Loosely speak-ing, this means that the recursion has to unravel to one-sixteenth the size of L3 before work-stealing schedulers startpreserving locality.
To support this observation, we ran the RRM and RRGbenchmarks varying the number of cores per socket; the re-sults are in Fig. 7. The number of L3 cache misses whenusing the SB and SB-D schedulers do not change with thenumber of cores, because cores constructively share the L3cache independent of their number. However, when usingthe WS and PWS schedulers, the number of L3 misses is
0
10
20
30
40
50
60
70
80
WS PWS SB SB-‐D WS PWS SB SB-‐D WS PWS SB SB-‐D WS PWS SB SB-‐D WS PWS SB SB-‐D
4 X 1 4 X 2 4 x 4 4 x 8 4x8x2(HT)
RRG (L3 misses in millions) RRM (L3 misses in millions)
Figure 7: L3 cache misses for RRM and RRG, vary-ing the number of cores used per socket.
0
100
200
300
400
500
600
0
0.5
1
1.5
2
2.5
WS PWS SB SB-‐D WS PWS SB SB-‐D WS PWS SB SB-‐D WS PWS SB SB-‐D WS PWS SB SB-‐D
Quicksort (100M double) Samplesort (100M double) Aware SampleSort (100M double)
Quad-‐Tree (100M double) MatMul (5.12K X 5.12K double)
AcHve Time (seconds) Overhead (seconds) L3 Cache misses (millions)
Figure 8: Active times, overheads, and L3 cachemisses for the 5 benchmark algorithms at full band-width.
0
100
200
300
400
500
600
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
WS PWS SB SB-‐D WS PWS SB SB-‐D WS PWS SB SB-‐D WS PWS SB SB-‐D WS PWS SB SB-‐D
Quicksort (100M double) Samplesort (100M double) Aware SampleSort (100M double)
Quad-‐Tree (100M double) MatMul (5.12K X 5.12K double)
AcHve Time (seconds) Overhead (seconds) L3 Cache misses (millions)
Figure 9: Active times, overheads, and L3 cachemisses for the 5 benchmark algorithms at 25% band-width.
highly dependent on the number of cores: when fewer coresare active on each socket, there is lesser contention for spacein the shared L3 cache. Thus, again the experimental resultscoincide with the theoretical analysis.
These experiments indicate that the advantages of space-bounded schedulers over work-stealing schedulers improveas (i) the number of cores per socket goes up, and (ii) thebandwidth per core goes down. At 8 cores there is a 30–35%reduction in L3 cache misses. At 64 cores we would expect(by the analysis) over a 60% reduction.
Algorithms. Fig. 8 and Fig. 9 show the active times,scheduling overheads, and L3 cache misses of the five algo-rithmic kernels at 100% and 25% bandwidth, respectively,with 64 hyperthreads. These plots show that the space-bounded schedulers incur significantly fewer L3 cache misseson 4 of the 5 benchmarks, with up to 65% on matrix multi-ply. The cache-oblivious sample sort is the sole benchmark
0
20
40
60
80
100
SB SB-‐D SB SB-‐D SB SB-‐D SB SB-‐D
σ = 0.5 σ = 0.7 σ = 0.9 σ = 1.0
Empty Queue Time (milliseconds)
Figure 10: Empty queue times for Quad-tree.
with little difference in L3 cache misses across schedulers.Given the problem size n = 108 doubles and sample sort’s√n-way recursion, all subtasks after one level of recursion
are much smaller than the L3 cache. Thus, all four sched-ulers avoid overflowing the cache. Because of their addedoverhead relative to work-stealing, the space-bounded sched-ulers are 7% slower for this benchmark.
As with the synthetic benchmarks, the sensitivity of activetime to L3 cache misses depends on whether the algorithmis memory-intensive enough to stress the bandwidth. MatrixMultiplication, although benefitting from space-bounded sched-ulers in terms of cache misses, shows no significant improve-ment in active time at full bandwidth because it is very com-pute intensive. However, when the machine bandwidth isreduced to 25%, Matrix Multiplication is bandwidth boundand the space-bounded schedulers are about 50% faster thanwork-stealing. The other three benchmarks—Quicksort, AwareSamplesort and Quad-tree—are memory intensive and seeimprovements of up to 25% in running time at full band-width (Fig. 8). At 25% bandwidth, the improvement is evenmore significant and up to 40% (Fig. 9).
Load balance and the dilation parameter σ. Thechoice of σ, determining which tasks are maximal, is animportant parameter affecting the performance of space-bounded schedulers, especially their ability to load balance.If σ were set to 1, it is likely that one task that is aboutthe size of the cache gets anchored to the cache, leaving lit-tle room for other tasks or strands. This adversely affectsload balance, and we would expect to see greater emptyqueue times. On the other hand, if σ were set to a lowervalue like 0.5, then each cache can allow more than one taskor strand to be simultaneously anchored, leading to betterload balance. Fig. 10 gives an example algorithm (Quad-tree) demonstrating this. If σ is too low (closer to 0), thenthe number of levels of recursion until the space-boundedschedulers preserve locality would increase, resulting in lesseffective use of shared caches.
Comparison of scheduler variants. Looking at the re-sults, we find that while PWS can reduce the number ofcache misses by up to 10% compared to standard WS, ithas negligible impact on running times for the benchmarksstudied. Similarly, while SB-D is designed to remove a se-rial bottleneck in the SB scheduler, the runtime (and cachemiss) performance of the two are nearly identical. This isbecause our benchmarks call the scheduler sufficiently infre-quently so that the performance difference of each invocationis not noticeable in the overall running time.
6. CONCLUSIONWe developed a framework for comparing schedulers, and
deployed it on a 32-core machine with 3 levels of caches.We used it to compare four schedulers, two each of work-
stealing and space-bounded types. As predicted by the-ory, we did notice that space-bounded schedulers demon-strate some, or even significant, improvement over work-stealing schedulers in terms of cache miss counts on sharedcaches for most benchmarks. In memory-intensive bench-marks with low instruction count to cache miss count ratios,an improvement in L3 miss count because of space-boundedschedulers can improve running time, despite their addedoverheads. On the other hand, for compute-intensive bench-marks or benchmarks with highly optimized cache complex-ities, work-stealing schedulers are slightly faster, because oftheir low scheduling overhead. Improving the overhead ofspace-bounded schedulers further could make the case forspace-bounded schedulers stronger and is an important di-rection for future work.
Our experiments were run on an Intel R© multicore with(only) 8 cores per socket, 32 cores total, and one level ofshared cache (the L3). The experiments make it clear thatas the core count per socket goes up (as is expected with eachnew generation), the advantages of space-bounded sched-ulers should increase due to the increased benefit of avoidingcache conflicts among the many unrelated threads sharingthe limited on-chip cache capacity. As the core count persocket goes up, the available bandwidth per core decreases,again increasing space-bounded schedulers’ advantage. Wealso anticipate a greater advantage for space-bounded sched-ulers over work-stealing schedulers when more cache lev-els are shared and when the caches are shared amongst agreater number of cores. Such studies are left to futurework, when such multicores become available. On the otherhand, compute-intensive benchmarks will likely continue tobenefit from the lower scheduling overheads of work-stealingschedulers, for the next few generations of multicores, if notlonger.
AcknowledgementsThis work is supported in part by the National Science Foun-dation under grant numbers CCF-1018188, CCF-1314633,CCF-1314590, the Intel Science and Technology Center forCloud Computing (ISTC-CC), the U.S. Department of En-ergy, Office of Science, Office of Advanced Scientific Com-puting Research, Applied Mathematics and Computer Sci-ence programs under contract No. DE-AC02-05CH11231through the Dynamic Exascale Global Address Space (DE-GAS) programming environments project, and a FacebookGraduate Fellowship.
7. REFERENCES
[1] U. A. Acar, G. E. Blelloch, and R. D. Blumofe. Thedata locality of work stealing. Theory Comp. Sys.,35(3), 2002.
[2] B. Alpern, L. Carter, and J. Ferrante. Modelingparallel computers as memory hierarchies. InProgramming Models for Massively ParallelComputers, 1993.
[3] G. E. Blelloch. Programming parallel algorithms.Communications of the ACM, 39(3), 1996.
[4] G. E. Blelloch, R. A. Chowdhury, P. B. Gibbons,V. Ramachandran, S. Chen, and M. Kozuch. Provablygood multicore cache performance fordivide-and-conquer algorithms. In SODA, 2008.
[5] G. E. Blelloch, J. T. Fineman, P. B. Gibbons, andJ. Shun. Internally deterministic parallel algorithmscan be fast. In PPoPP, 2012.
[6] G. E. Blelloch, J. T. Fineman, P. B. Gibbons, andH. V. Simhadri. Scheduling irregular parallelcomputations on hierarchical caches. In SPAA, 2011.
[7] G. E. Blelloch, J. T. Fineman, P. B. Gibbons, andH. V. Simhadri. Program-centric cost models forlocality. In MSPC, 2013.
[8] G. E. Blelloch, P. B. Gibbons, and Y. Matias.Provably efficient scheduling for languages withfine-grained parallelism. JACM, 46(2), 1999.
[9] G. E. Blelloch, P. B. Gibbons, and H. V. Simhadri.Low-depth cache oblivious algorithms. In SPAA, 2010.
[10] R. D. Blumofe, M. Frigo, C. F. Joerg, C. E. Leiserson,and K. H. Randall. An analysis of dag-consistentdistributed shared-memory algorithms. In SPAA,1996.
[11] R. D. Blumofe and C. E. Leiserson. Schedulingmultithreaded computations by work stealing. JACM,46(5), 1999.
[12] R. A. Chowdhury and V. Ramachandran. Thecache-oblivious gaussian elimination paradigm:theoretical framework, parallelization andexperimental evaluation. In SPAA, 2007.
[13] R. A. Chowdhury and V. Ramachandran.Cache-efficient dynamic programming algorithms formulticores. In SPAA, 2008.
[14] R. A. Chowdhury, V. Ramachandran, F. Silvestri, andB. Blakeley. Oblivious algorithms for multicores andnetworks of processors. Journal of Parallel andDistributed Computing, 73(7):911 – 925, 2013. BestPapers of IPDPS 2010, 2011 and 2012.
[15] R. A. Chowdhury, F. Silvestri, B. Blakeley, andV. Ramachandran. Oblivious algorithms for multicoresand network of processors. In IPDPS, 2010.
[16] K. Fatahalian, D. R. Horn, T. J. Knight, L. Leem,M. Houston, J. Y. Park, M. Erez, M. Ren, A. Aiken,W. J. Dally, et al. Sequoia: Programming the memoryhierarchy. In Supercomputing, 2006.
[17] Intel. Intel Cilk++ SDK programmer’s guide.https://www.clear.rice.edu/comp422/resources/
Intel_Cilk++_Programmers_Guide.pdf, 2009.
[18] Intel. Intel Thread Building Blocks reference manual.http://software.intel.com/sites/products/
documentation/doclib/tbb_sa/help/index.htm\
#reference/reference.htm, 2013. Version 4.1.
[19] Intel. Performance counter monitor (PCM).http://www.intel.com/software/pcm, 2013. Version2.4.
[20] R. Kriemann. Implementation and usage of a threadpool based on posix threads. www.hlnum.org/english/projects/tools/threadpool/doc.html, 2004.
[21] D. Lea. A java fork/join framework. In ACM JavaGrande, 2000.
[22] C. Leiserson. The Cilk++ concurrency platform. J.Supercomputing, 51, 2010.
[23] Microsoft. Task Parallel Library. http://msdn.microsoft.com/en-us/library/dd460717.aspx, 2013..NET version 4.5.
[24] G. J. Narlikar. Scheduling threads for low spacerequirement and good locality. In SPAA, 1999.
[25] OpenMP Architecture Review Board. OpenMP API.http://www.openmp.org/mp-documents/spec30.pdf,May 2008. v 3.0.
[26] Perfmon2. libpfm.http://perfmon2.sourceforge.net/, 2012.
[27] J.-N. Quintin and F. Wagner. Hierarchicalwork-stealing. In EuroPar, 2010.
[28] J. Shun, G. E. Blelloch, J. T. Fineman, P. B. Gibbons,A. Kyrola, H. V. Simhadri, and K. Tangwongsan.Brief announcement: the problem based benchmarksuite. In SPAA, 2012.
[29] H. V. Simhadri. Program-Centric Cost Models forLocality and Parallelism. PhD thesis, CMU, 2013.
[30] D. Spoonhower, G. E. Blelloch, P. B. Gibbons, andR. Harper. Beyond nested parallelism: tight boundson work-stealing overheads for parallel futures. InSPAA, 2009.
[31] L. G. Valiant. A bridging model for multi-corecomputing. In ESA, 2008.
APPENDIXA. WORK STEALING SCHEDULER
Fig. 11 provides an example of a work-stealing schedulerimplemented using the scheduler interface presented in Sec-tion 3.1. The Job* argument passed to the add and done
functions may be instances of one of the derived classes ofJob* that carry additional information helpful to the sched-uler.
B. MEASURING HARDWARE COUNTERSMulticore processors based on newer architectures like
Intel R©Nehalem-EX and Sandybridge contain numerous func-tional components such as cores (which includes the CPUand lower level caches), DRAM controllers, bridges to theinter-socket interconnect (QPI) and higher level cache units(L3). Each component is provided with a performance mon-itoring unit (PMU)—a collection of hardware registers thatcan track statistics of events relevant to the component.
For instance, while the core PMU on Xeon R© 7500 series(our experimental setup, see Fig. 1(a)) is capable of provid-ing statistics such as the number of instructions, L1 and L2cache hit/miss statistics, and traffic going in and out, it isunable to monitor L3 cache misses (which constitute a sig-nificant portion of active time). This is because L3 cache isa separate unit with its own PMU(s). In fact, each Xeon R©7560 die has eight L3 cache banks on a bus that also con-nects DRAM and QPI controllers (see Fig. 12). Each L3bank is connected to a core via buffered queues. The ad-dress space is hashed onto the L3 banks so that a uniquebank is responsible for each address. To collect L3 statisticssuch as L3 misses, we monitor PMUs (called C-Boxes onNehalem-EX) on all L3 banks and aggregate the numbers inour results.
Software access to core PMUs on most Intel R© architec-tures is well supported by several tools including the Linuxkernel, the Linux perf tool, and higher level APIs such aslibpfm [26]. We use the libpfm library to provide fine-grainedaccess to the core PMU. However, access to uncore PMUs—complex architecture-specific components like the C-Box—is
void WS_Scheduler::add (Job *job, int thread_id) {_local_lock[thread_id].lock();_job_queues[thread_id].push_back(job);_local_lock[thread_id].unlock();
}intWS_Scheduler::steal_choice (int thread_id) {
return (int)((((double)rand())/((double)RAND_MAX))*_num_threads);
}Job* WS_Scheduler::get (int thread_id) {
_local_lock[thread_id].lock();if (_job_queues[thread_id].size() > 0) {Job * ret = _job_queues[thread_id].back();_job_queues[thread_id].pop_back();_local_lock[thread_id].unlock();return ret;
} else {_local_lock[thread_id].unlock();int choice = steal_choice(thread_id);_steal_lock[choice].lock();_local_lock[choice].lock();if (_job_queues[choice].size() > 0) {Job * ret = _job_queues[choice].front();_job_queues[choice].erase(_job_queues[choice].begin());++_num_steals[thread_id];_local_lock[choice].unlock();_steal_lock[choice].unlock();return ret;
}_local_lock[choice].unlock();_steal_lock[choice].unlock();
}return NULL;
}void WS_Scheduler::done (Job *job, int thread_id,
bool deactivate) {}
Figure 11: WS scheduler implemented in schedulerinterface
not supported by most tools. Newer Linux kernels (3.7+)are incrementally adding software interfaces to these PMUsat the time of this writing, but we are only able to makeprogram-wide measurements using this interface rather thanfine-grained measurements. For accessing uncore counters,we adapt the Intel R© PCM 2.4 tool [19].
L3
DRAM Banks
core
DRAM ctrler
QPI
core
core
core
core
core
core
core
DRAM ctrler
to QPI
to QPI
to QPI
to QPI
L3
L3
L3
L3
L3
L3
L3
C-‐box
Figure 12: Layout of 8 cores and L3 cache banks ona bidirectional ring in Xeon R© 7560. Each L3 bankhosts a performance monitoring unit called C-boxthat measures traffic into and out of the L3 bank.
To count L3 cache misses, the uncore counters in theC-boxes were programmed using the Intel R© PCM tool tocount misses that occur due to any reason (LLC_MISSES -
event code: 0x14, umask: 0b111) and L3 cache fills inany coherence state (LLC_S_FILLS - event code: 0x16,
umask: 0b1111). Both the numbers concur up to three sig-nificant digits in most cases. Therefore, only the L3 cachemiss numbers are reported in this paper.