the challenges facing libraries and imperative languages from massively parallel architectures
DESCRIPTION
ACCU Conference 2008 and London.TRANSCRIPT
The University of Hertfordshire
The Challenges facing Libraries and Imperative Languages from
Massively Parallel Architectures
Building Futures in Computer Scienceempowering people through technology
Jason McGuinessComputer ScienceUniversity of HertfordshireUK
Colin EganComputer ScienceUniversity of HertfordshireUK
• Parallel processing– Pipeline processors, MII architectures, Multiprocessors
• Processing In Memory (PIM)– Cellular Architectures: Cyclops/DIMES and picoChip
• Code-generation issues arising from massive parallelism• Possible solutions to this issue:
– Use the compiler or some libraries• An example implementation of a library, and the issues• Questions?
– Ask as we go along, but we’ll also leave time for questions at then end of this presentation
Presentation Structure
Parallel Processing
• How can parallel processing be achieved?– By exploiting:
• Instruction Level Parallelism (ILP)• Thread Level Parallelism (TLP)• Multi-processing• Data Level Parallelism (DLP)• Simultaneous Multi-Processing (SMP)• Concurrent processing• etcetera
• Exploits ILP by overlapping instructions in different stages of execution:– ILP is the amount of operations in a computer program that
can be performed on at the same time (simultaneously)
• Improves overall program execution time by increasing throughput:– It does not improve individual instruction execution time
Pipelining
• A simple 5-stage pipeline
Pipelining
InstructionCache
Access;
Increment Program Counter;
Branch Prediction
InstructionDecode;
Register operand set up
DataCacheAccess
Write Back
Execution in ALU
• Pipelining introduces hazards which can severely impact on processor performance:– Data (RAW, WAW and WAR)– Control (conditional branch instructions)– Structural (hardware contention)
• To overcome such hazards complex hardware (dynamic scheduling) or complex software (static scheduling) or a combination of both is required
Pipelining hazards
• Data hazards– Read After Write (RAW dependency)
• The result of an operation is required as an operand of a subsequent operation and that result is not yet available in the register file
add $1,$2,$3 //$1=$2+$3addi $4,$1,8 //$4=$1+8 data dependency ($1 is unavailable)sub $5,$2,$1 //$5=$2-$1 data dependency ($1 is unavailable)ld $6,8($1) //$6=Mem[$1+8]... $1 may be availableadd $7,$6,$8 //load-use dependency...data dependency ($6 is
unavailable)
Pipelining hazardsData, Control, Structural
• Data hazards– Write After Write (WAW data dependency)
sub $1,$2,$3 addi $1,$4,8 //the addi cannot complete ahead of the sub
– Write After Read (WAR data dependency)sub $2,$1,$3 addi $1,$4,8 //the addi cannot complete ahead of the sub
Pipelining hazardsData, Control, Structural
• Data hazards– Read After Write (RAW data dependency) is a
problem in single instruction issue architectures and Multiple Instruction Issue (MII) architectures
– Write After Write (WAW data dependency) and Write After Read (WAR data dependency) are a problem MII architectures, not single instruction issue architectures
Pipelining hazardsData, Control, Structural
• Data hazards– Read After Write (RAW data dependency) are True Data
Dependencies• RAW are avoided:
– in single instruction issue architectures by forwarding (bypassing),
– in MII:
» Superscalar (dynamic) by result broadcast
» VLIW (static) by aggressive instruction (percolation) scheduling
– Write After Write (WAW data dependency) and Write After Read (WAR data dependency) are Artificial Data Dependencies
• WAW/WAR are eliminated by Register Renaming
Pipelining hazardsData, Control, Structural
• Control hazards– Are caused by (conditional) branch instructions– The only instruction which can alter the smooth flow of
instruction execution– The next instruction to be executed is not known until the
condition evaluation has been determined and in the case that the condition is to be taken then the address of the next instruction to be executed needs to be computed:
• branch resolutionbeq $1,$2,label //if $1==$2 then pc=label; else pc=pc+4;bnq $1,$2,label //if $1!=$2 then pc=label; else pc=pc+4;
Pipelining hazardsData, Control, Structural
• Control hazards– Are a major problem:
• Between 1 in 5 and 1 in 8 instructions from the dynamic stream of a general purpose program are conditional branch instructions
• Approximately 66% of conditional branches are taken (target path)• Are a particular problem in Superscalar Architectures where 100s of
instructions may be in flight
– The impact can be reduced by:• Branch prediction (dynamic and static)• Branch removal (scheduling - static)• Delay region scheduling (static)• Multithreading (dynamic and static)
Pipelining hazardsData, Control, Structural
• Structural hazards– More than one instruction attempting to access the
same piece of hardware at the same time
– These can be overcome by increasing the amount of hardware:
• For example, use the Harvard Architecture (separate the I-cache from the D-cache)
Pipelining hazardsData, Control, Structural
• MII– A processor that is capable of fetching and issuing
more than one instruction during each processor cycle– A program is executed in parallel, but the processor
maintains the outward appearance of sequential execution
– The program binary must therefore be regarded as a specification of what was done, not how it was done
– Minimise program execution time by:• by reducing instruction latencies• by exploiting additional ILP
Multiple Instruction Issue
• Superscalar Processor:– An MII processor where the number of instructions issued
in each clock cycle is determined dynamically by hardware at run-time
• Instruction issue may be in-order or out-of-order (Tomasulo or equivalent).
• VLIW Processor (Very Long Instruction Word):– An MII processor where the number of operations
(instructions) issued in each clock cycle is fixed and where the operations are selected statically by the compiler at compile time
Multiple Instruction Issue
• Fetch and Decode multiple instructions in parallel
• Dynamically predict conditional branches
• Initiate instructions for parallel execution based, whenever possible, on the availability of operands
• Frequently issue/execute instructions out-of-order
• Re-sequence instructions into the original program order after completion
Superscalars
Superscalars
1. Instruction fetch strategies Multiple instruction fetch Dynamic branch prediction
2. Detecting True Data Dependencies between registers3. Initiating or issuing multiple instructions in parallel4. Resources for parallel execution of instructions:
Multiple pipelined functional units Efficient memory hierarchies
5. Communication of data values through memory: Loads and stores (RISC) Allowing for dynamic, unpredictable behaviour of memory hierarchies
6. Committing the process state in the correct order: Maintaining the appearance of sequential instruction execution
Superscalars
• Since there is very little ILP within individual basic blocks (i.e. between branches), branch prediction is essential for high performance
• It is important to maintain an instruction window or window of execution from which ILP can be extracted
Superscalars
• The problems include:– Conditional branches– Data path complexity– The memory wall (widening gap between CPU and
memory access times)– Limited available parallelism within a finite sized
instruction window
• Is the superscalar approach at a point of diminishing returns?
• Attempts to reduce hardware complexity by extracting ILP at compile time instead of run time
• The processor issues a single very long instruction during each machine cycle
• Each very long instruction consists of a large number of independent operations packed into a single instruction word by the compiler
• Each operation requires a small, statically predictable number of cycles to execute
• Each operation is itself pipelined• Execute instructions in-order
VLIWs
• Problems:– Code expansion:
• The need to pad out instruction groups with NOPs can double the code size
– Compatibility:• Code must be recompiled (scheduled) for every new instance
of a particular VLIW architecture
VLIWs
• To sustain multiple instruction fetch, MII architectures require a complex memory hierarchy:– Caches
• l1, l2, stream buffers, non-blocking caches– Virtual Memory
• TLB
– Caches suffer from:• Compulsory misses• Capacity misses• Collisions
Problems of MII
• Compulsory misses:– The first time a processor address is requested it will not be
in cache memory and must be fetched from a slower level of the memory hierarchy:
• Hopefully main (physical) memory• If not from Virtual Memory• If not from secondary storage
– This can result in long delay (latency) due to large access time(s)
Memory Problems
• Capacity misses:– There are more cache block requests than the size of the
cache
• Collisions:– The processor makes a request to the same block but for
different instructions/data
• For both:– Blocks therefore have to be replaced– But a block that has been replaced might be referenced
again resulting in yet more replacements
Memory Problems
• A thread can be considered to be a ‘light weight process’– Where a thread consists of a short sequence of code, with
its own: • registers, data, state and so on• but shares process space
• TLP is exploited by simultaneously executing different threads on different processors:– TLP is therefore exploited by multiprocessors
Thread Level Parallelism
Multiprocessors
• Should be:– Easily scalable– Fault tolerant– Achieve higher performance than a uni-processor
• But …– How many processors can we connect?– How do parallel processors share data?– How are parallel processors co-ordinated?
Multiprocessors
• Shared Memory Processors (SMP)– All processors share a single global memory address space
– Communication is through shared variables in memory
– Synchronisation is via locks (hardware) or semaphores (software)
Multiprocessors
• Uniform Memory Access (UMA)– All memory accesses take the same time– Do not scale well
• Non-uniform Memory Access (NUMA)– Each processor has a private (local) memory– Global memory access time can vary from processor to
processor– Present more programming challenges– Are easier to scale
Multiprocessors
• NUMA– Communication and synchronization are achieved
through message passing:• Processors could then, for example, communicate over
an interconnection network• Processors use send and receive primitives
Multiprocessors
• The difficulty is in writing effective parallel programs:– Parallel programs are inherently harder to develop– Programmers need to understand the underlying hardware– Programs tend not to be portable– Amdahl’s law; a very small part of a program that is inherently
sequential can severely limit the attainable speedup
• “It remains to be seen how many important applications will run on multiprocessors via parallel processing.”
• “The difficulty has been that too few important application programs have been written to complete tasks sooner on multiprocessors.”
Multiprocessors
• Multiprocessors suffer the same memory problems as uni-processors and in addition:– The problem of maintaining memory coherence
between the processors
• A read operation must return the value of the latest write operation
• But in multiprocessors each processor will (probably) have its own private cache memory– There is no guarantee of data consistency between
private (local) cache memory and shared (global) memory
Cache Coherence
• Today’s processors are 10 times faster than processors of only 5 years ago
• But today’s processors do not perform at 1/10th of the time of
processors from 5 years ago
• CPU speeds double approximately every eighteen months (Moore's Law), while main memory speeds double only about every ten years– This causes a bottle-neck in the memory subsystem, which impacts on
overall system performance– The “memory wall” or the “Von Neumann bottleneck”
The memory wall
• In today’s processors RAM roughly holds 1% of the data stored on a hard disk drive
• And hard disk drives supply data nearly a thousand times slower than a processor can use it
• This means that for many data requests, a processor is idle while it waits waiting for data to be found and transferred
The memory wall
• IBM’s vice president Mark Dean has said:– "What's needed today is not so much the ability to
process each piece of data a great deal, it's the ability to swiftly sort through a huge amount of data."
The memory wall
• Mark Dean again:– "The key idea is that instead of focusing on
processor micro-architecture and structure, as in the past, we optimize the memory system's latency and throughput — how fast we can access, search and move data …"
• Leads to the concept of Processing In Memory (PIM)
The memory wall
• The idea of PIM is to overcome the bottleneck between the processor and main memory by combining a processor and memory on a single chip
• The benefits of a PIM architecture are:– Reduced memory latency– Increases memory bandwidth– Simplifies the memory hierarchy– Provides multi-processor scaling capabilities:
• Cellular architectures
– Avoids the Von Neumann bottleneck
Processing in memory
• This means that:– Much of the expensive memory hierarchy can be
dispensed with– CPU cores can be replaced with simpler designs– Less power is used by PIM– Less silicon space is used by PIM
Processing in memory
• But …– Processor speed is reduced– The amount of available memory is reduced
• However, PIM is easily scaled:– Multiple PIM chips connected together forming a
network of PIM cells
– Such scaled architectures are called Cellular architectures
Processing in memory
• Cellular architectures consist of a high number of cells (PIM units):– With tens of thousands up to one million processors
– Each cell (PIM) is small enough to achieve extremely large-scale parallel operations
– To minimise communication time between cells, each cell is only connected to its neighbours
Cellular architectures
• Cellular architectures are fault tolerant:– With so many cells, it is inevitable that some processors
will fail– Cellular architecture simply re-route instructions and data
around failed cells
• Cellular architectures are ranked highly as today’s Supercomputers:– IBM BlueGene takes the top slots in the Top 500 list
Cellular architectures
• Cellular architectures are threaded:– Each thread unit:
• Is independent of all other thread units• Serves as a single in-order issue processor• Shares computationally expensive hardware such as floating-point
units
– There can be a large number of thread units:• 1,000s if not 100,000s of thousands• Therefore they are massively parallel architectures
Cellular architectures
• Cellular architectures are NUMA– Have irregular memory access:
• Some memory is very close to the thread units and is extremely fast
• Some is off-chip and slow
• Cellular architectures, therefore, use caches and have a memory hierarchy
Cellular architectures
• In Cellular architectures multiple thread units perform memory accesses independently
• This means that the memory subsystem of Cellular architectures do in fact require some form of memory access model that permits memory accesses to be effectively served
Cellular architectures
• Uses of Cellular architectures:– Games machines (simple Cellular architecture)– Bioinformatics (protein folding)– Imaging
• Satellite• Medical• Etcetera
– Research– Etcetera
Cellular architectures
• Examples:– BlueGene Project:
• Cyclops (IBM) – next generation from BlueGene/P, called BlueGene/C
– DIMES – a prototype implementation
– Gilgamesh (NASA)– Shamrock (Notre Dame)– picoChip (Bath, UK)
Cellular architectures
• Developed by IBM at the Tom Watson Research Center
• Also called BlueGene/C in comparison with the earlier version of BlueGene/L and BlueGene/P
IBM Cyclops or BlueGene/C
• The idea of Cyclops is to provide around one million processors:
– Where each processor can perform a billion operations per second
– Which means that Cyclops will be capable of one petaflop of computations per second (a thousand trillion calculations per second)
Cyclops
CyclopsProcessor
ChipBoard
Crossbar Network
TU TU TU…
SP SP SP
FPU SR
TU TU TU…
SP SP SP
FPU SR
TU TU TU…
SP SP SP
FPU SR
…M
EM
OR
YB
AN
K
ME
MO
RY
BA
NK
ME
MO
RY
BA
NK
ME
MO
RY
BA
NK
ME
MO
RY
BA
NK
ME
MO
RY
BA
NK
ME
MO
RY
BA
NK
ME
MO
RY
BA
NK
…
6 * 4 GB/sec
4 GB/sec
50 MB/sec
1 Gbits/sec
Off-
Chi
p M
emor
y
OtherChips via 3D mesh
Off-
Chi
p M
emor
yO
ff-C
hip
Mem
ory
Off-
Chi
p M
emor
y
IDEHDD
4 GB/sec
6 * 4 GB/sec
SP SP SP SP SP SP SP SP
• DIMES:– Is the first hardware implementation of a Cellular
architecture– Is a simplified ‘cut-down’ version of Cyclops– Is hardware validation tool for Cellular architectures– Emulates Cellular architectures, in particular Cyclops,
cycle-by-cycle – Is implemented on at least one FPGA– Has been evaluated by Jason
DIMES
• The DIMES implementation that Jason evaluated:– Supports a P-thread programming model– Is a dual processor where each processor has four thread
units– Has 4K of scratch-pad (local) memory per thread unit– Has two banks of 64K global shared memory– Has different memory models:
• Scratch pad memory obeys the program consistency model for all of the eight thread units
• Global memory obeys the sequential consistency model for all of the eight thread units
– Is called DIMES/P2
DIMES
DIMES/P2
• Jason’s concerns were:– How to manage a potentially large number of
threads
– How to exploit parallelism from the input source code in these threads
– How to manage memory consistency
DIMES
• Jason tested his concerns by using an “embarrassingly parallel program which generated Mandelbrot sets”
• Jason’s approach was to distribute the work-load between threads and he also implemented a work-stealing algorithm to balance loads between threads:– When a thread completed its ‘work-load’, rather than
remain idle that thread would ‘steal-work’ from another ‘busy’ thread
– This meant that he maximised parallelism and improved thread performance and hence overall program execution time
DIMES
DIMESShortly after program start Shortly before work stealing
Just after work stealing More work stealing
picoChip• picoChip are based in Bath
– “… is dedicated to providing fast, flexible wireless solutions for next generation telecommunications systems.”
• picoArrayTM
– Is a tiled architecture– 308 heterogeneous processors connected together,– The interconnects consist of bus switches joined
by picoBusTM
– Each processor is connected to the picoBusTM above and below it
picoChip
• picoArrayTM
–
picoChip and Parallelism
• The parallelism provided by picoChip is synchronous – This avoids many of the issues raised by the other
architectures that expose asynchronous parallelism
– But it is at the cost of the flexibility that asynchronous parallelism provides
Issues arising ...
• Such massively parallel hardware creates a serious problem for the programmer:– Because it is computer science folklore that writing
parallel programs is very hard
• Writing fast and correct massively parallel programs is a very major concern
Abstraction of the Parallelism
• This may be done in various ways:– For example within the compiler:
• Using trace scheduling, list-based scheduling or other data-flow based means amongst others
– Using language features:• HPF, UPC or the additions to C++ in the IBM Visual
Age compiler and Microsoft's additions
– Most commonly using libraries:• For example: Posix threads, Win32 threads, OpenMP,
boost.thread, home-grown wrappers
Parallelism using Libraries
• Using libraries has a major advantage over implementing parallelism within the language:– It does not require the design of a new language,
nor learning a new language– Novel languages are traditionally seen as a burden
and often hinder adoption of new systems due to the volume of existing source-code
– But libraries are especially prone to mis-use and are traditionally hard to use
Issues of Libraries: part I• The model exposed is very diverse:
– Message passing, e.g. OpenMP– Exposes loop-level parallelism (e.g. “forall ...”
constructs) exposes very limited parallelism in general-purpose code
– Is very low-level, e.g. the primitives exposed in Posix Threads are extremely basic: mutexes, condition variables, and basic thread operations
• The libraries require experience to use well, or even correctly
Part II: Non-composability of atomic operations!
• The fact that atomic operations do not compose is a major concern when using libraries!
• The composition of thread-safe data structures does not guarantee thread-safety:– At each combination of the data structures, more locks are
required to ensure correct operation!– This implies more and more layers of locks, that are
slow...
Parallelism in the Compiler
• Given the concerns with regards to libraries, what about parallelising compilers?
• The fact is that auto parallelising compilers exist, e.g. list-based scheduling implemented in the EARTH-C (circa 1999) has been proven to be optimal for that architecture
• Data-flow compilers have existed for years– Why aren't they used?
Industrial Parallelising Compilers
• Microsoft is introducing OpenMP-based constructs into their compiler, e.g. “forall”
• IBM Visual Age has similar functionality
• Java has a thread library
• C++0x: much work has been done regarding standardisation with respect to multiple threads
C++ as an example
• The uses of C++ makes it an interesting target for parallelisation:– Although imperative, so arguably flawed for
implementing parallelism– It has great market penetration, therefore there is
much demand for using parallelism– Commonly used in high-performance, multi-
threaded systems– General-purpose nature and quality libraries are
increasing the appeal to super-computers
Parallelism support in C++
• Libraries exist beyond the usual C libraries:– boost.thread – exists now, requires standards-
compliant compilers– C++0x: details of the threading support are
becoming apparent that appear to include:• Atomic operations (memory consistency), exceptions,
machine-model underpins approach• Threading models: thread-as-a-class, lock objects• Thread pools – coming later – probably• More details on the web, or at the ACCU - in flux
Experience using C++• Recall DIMES:
– Prototype of massively parallel hardware– Posix-threads style library implementing threads– C++ thread-as-a-class wrapper implemented
• Summary of experience:– Hardly object-orientated: no separation in design of the
Mandlebrot application with the work-stealing algorithm and thread pool
– The software insufficiently separated the details of the hardware features from the design
Further experiences using C++
• From this work and other experiences, I developed a more interesting thread library:– Traits abstract underlying OS thread API from
wrapper library– Therefore has hardware abstractions too– Provision of higher-level threading models:
• Primarily based on futures and thread pools
– Use of thread pools and futures creates a singly rooted-tree of threads:• Trivially deadlock free – a holy grail!
C++0x threads now? No!• Included in libjmmcg:
– Relies upon non-standard behaviour and broken optimisers! For example:• Problems with global code-motion moving apparently const-objects past
locks• Exception stack is unique to a thread, not global to program and currently
unspecified• Implementation of std::list
– DSEL has syntax limitations due to design of C++– Doesn't use boost ...– Has example code and test cases– Isn't complete! (e.g. Posix & sequential specialisations
incomplete, some inefficiencies)– But get it from libjmmcg.sourceforge.net, under LGPL
Trivial example usagestruct res_t { int i; };
struct work_type {typedef res_t result_type;void process(result_type &) {}
};
pool_type pool(2);
async_work work(creator_t::it(work_type(1)));
execution_context context(pool<<joinable()<<time_critical()<<work);
pool.erase(context);
context->i;
• The devil is in the omitted details: the typedefs for:– pool_type, async_work, execution_context, joinable, time_critical• The library requires that the work to be mutated has the
items in italics defined
Explanation of the example
• The concept is:– that asynchronous work (async_work) that should be
mutated (process) to the specified output type (result_type) is transferred into a thread pool (pool_type) of some kind
• This transfer (<<) may, optionally (joinable), return a future (execution_context)– Which can be used to communicate (->) the result of the
mutation, executed at kernel priority (time_critical), back to the caller
• The future also allows exceptions to be propagated
More details regarding the example
• The thread pool (pool_type) has many traits:– Master-slave or work-stealing– The thread API (Win32, Posix or sequential)– The thread API costs in very rough terms– Implies a work schedule that is a fifo baker's ticket schedule,
implementation of GSS(k) is in progress
• The library effectively implements a software simulation of data-flow
• Wrapping a function call, plus parameters, in a class converts Kevlin's threading model to this one
Time for controversy....What faces programmers...
• Large-scale parallelism is here, now:– Blade frames at work:
• 4 cores x 4 CPUs x 20 frames per rack = 320 thread units, in a NUMA architecture
• The hardware is expensive!• But so is the software ...
– It must be programmed, economically– The programs must be maintained ...
• Or it will be an expensive failure?
Talk summary• In this talk we have looked at parallel processing and
justified the reasons for Processing In Memory (PIM) and therefore cellular architectures
• We have briefly looked at two example architectures:
– Cyclops and picoChip
• Jason has worked on DIMES, the first implementation of a (cut-down) version of a cellular architecture
• The issues of programming for these massively parallel architectures has been described
• We focussed on the future of threading in C++
The University of Hertfordshire
The Challenges facing Libraries and Imperative Languages from
Massively Parallel Architectures
Questions?
Building Futures in Computer Scienceempowering people through technology
Jason McGuinessComputer ScienceUniversity of HertfordshireUK
Colin EganComputer ScienceUniversity of HertfordshireUK