cmpe 49b sp. top. in cmpe: multi-core programming

SWE 594 1

CMPE 49B Spec. Topics in CMPE: Multi-core Programming

picture of ASCI WHITE,

the most powerfulcomputer in the world in 2001

SWE 594 2

Von Neumann Architecture

CPU RAM Device Device

• sequential computer

BUS

SWE 594 3

Memory Hierarchy

Registers

Cache

Real Memory

Disk

CD

Fast

Slow

SWE 594 4

History of Computer Architecture

• 4 Generations (identified by logic technology)

1. Tubes

2. Transistors

3. Integrated Circuits

4. VLSI (very large scale integration)

SWE 594 5

PERFORMANCE TRENDS

SWE 594 6

PERFORMANCE TRENDS

• Traditional mainframe/supercomputer performance 25% increase per year

• But … microprocessor performance 50% increase per year since mid 80’s.

SWE 594 7

Moore’s Law

• “Transistor density doubles every 18 months”

• Moore is co-founder of Intel.

• 60 % increase per year

• Exponential growth

• PC costs decline.

• PCs are building bricks of all future systems.

SWE 594 8

VLSI Generation

SWE 594 9

Bit Level Parallelism(upto mid 80’s)

• 4 bit microprocessors replaced by 8 bit, 16 bit, 32 bit etc.

• doubling the width of the datapath reduces the number of cycles required to perform a full 32-bit operation

• mid 80’s reap benefits of this kind of parallelism (full 32-bit word operations combined with the use of caches)

SWE 594 10

Instruction Level Parallelism(mid 80’s to mid 90’s)

• Basic steps in instruction processing (instruction decode, integer arithmetic, address calculations, could be performed in a single cycle)

• Pipelined instruction processing

• Reduced instruction set (RISC)

• Superscalar execution

• Branch prediction

SWE 594 11

Thread/Process Level Parallelism(mid 90’s to present)

• On average control transfers occur roughly once in five instructions, so exploiting instruction level parallelism at a larger scale is not possible

• Use multiple independent “threads” or processes

• Concurrently running threads, processes

SWE 594 12

Evolution of the Infrastructure

• Electronic Accounting Machine Era: 1930-1950

• General Purpose Mainframe and Minicomputer Era: 1959-Present

• Personal Computer Era: 1981 – Present

• Client/Server Era: 1983 – Present

• Enterprise Internet Computing Era: 1992- Present

SWE 594 13

Sequential vs Parallel Processing

• physical limits reached

• easy to program

• expensive supercomputers

• “raw” power unlimited

• more memory, multiple cache

• made up of COTS, so cheap

• difficult to program

SWE 594 14

What is Multi-Core Programming ?

• Answer: It is basically parallel programming on a single computer box (e.g. a desktop, a notebook, a blade)

SWE 594 15

Amdahl’s Law• The serial percentage of a program is fixed. So speed-up obtained by

employing parallel processing is bounded.

• Lead to pessimism in in the parallel processing community and prevented development of parallel machines for a long time.

Speedup = 1

s + 1-s

P

• In the limit:

Spedup = 1/s s

SWE 594 16

Gustafson’s Law• Serial percentage is dependent on the number of

processors/input.

• Broke/disproved Amdahl’s law.

• Demonstrated achieving more than 1000 fold speedup using 1024 processors.

• Justified parallel processing

SWE 594 17

Grand Challenge Applications

• Important scientific & engineering problems identified by U.S. High Performance Computing & Communications Program (’92)

SWE 594 18

Flynn’s Taxonomy

• classifies computer architectures according to:

1. Number of instruction streams it can process at a time

2. Number of data elements on which it can operate simultaneously

Data Streams

Single Multiple

Single

Multiple

Instruction Streams

SISD SIMD

MIMDMISD

SWE 594 19

SPMD Model (Single Program Multiple Data)

• Each processor executes the same program asynchronously

• Synchronization takes place only when processors need to exchange data

• SPMD is extension of SIMD (relax synchronized instruction execution)

• SPMD is restriction of MIMD (use only one source/object)

SWE 594 20

Parallel Processing Terminology• Embarassingly Parallel:

-applications which are trivial to parallelize

-large amounts of independent computation

-Little communication

•Data Parallelism:

-model of parallel computing in which a single operation can be applied to all data elements simultaneously

-amenable to SIMD or SPMD style of computation

•Control Parallelism:

-many different operations may be executed concurrently

-require MIMD/SPMD style of computation

SWE 594 21

Parallel Processing Terminology• Scalability:

- If the size of problem is increased, number of processors that can be effectively used can be increased (i.e. there is no limit on parallelism).

- Cost of scalable algorithm grows slowly as input size and the number of processors are increased.

- Data parallel algorithms are more scalable than control parallel alorithms

• Granularity:

- fine grain machines: employ massive number of weak processors each with small memory

- coarse grain machines: smaller number of powerful processors each with large amounts of memory

SWE 594 22

Shared Memory Machines

Shared Address Space

process(thread)

process(thread)

process(thread)

process(thread)

process(thread)

•Memory is globally shared, therefore processes (threads) see single address space

•Coordination of accesses to locations done by use of locks provided by thread libraries

•Example Machines: Sequent, Alliant, SUN Ultra, Dual/Quad Board Pentium PC

•Example Thread Libraries: POSIX threads, Linux threads.

SWE 594 23

Shared Memory Machines• can be classified as:

-UMA: uniform memory access

-NUMA: nonuniform memory access

based on the amount of time a processor takes to access local and global memory.

Inter-connectionnetwork/or BUS

Inter-connection

network

Inter-connection

network

P

P

..

P

M

M

..

M

P

M

P

M

..

P

M

P

M

P

M

..

P

M

M

M

M

..

M

(a)(b) (c)

SWE 594 24

Distributed Memory Machines

Network

process

process

process

process

processM

M

M

M

M

•Each processor has its own local memory (not directly accessible by others)

•Processors communicate by passing messages to each other

•Example Machines: IBM SP2, Intel Paragon, COWs (cluster of workstations)

•Example Message Passing Libraries: PVM, MPI

SWE 594 25

Beowulf Clusters

•Use COTS, ordinary PCs and networking equipment

•Has the best price/performance ratio

PC cluster

SWE 594 26

Multi-Core Computing

• A multi-core microprocessor is one which combines two or more

independent processors into a single package, often a single integrated circuit.

• A dual-core device contains only two independent microprocessors.

SWE 594 27

Comparison of Different Architectures

CPU State

CacheExecution

unit

Single Core Architecture

SWE 594 28


CPU State

CacheExecution

unit

Multiprocessor

CPU State

CacheExecution

unit

SWE 594 29


CPU State

CacheExecution

unit

Hyper-Threading Technology

CPU State

SWE 594 30


CPU State

CacheExecution

unit

Multi-Core Architecture

CPU State

CacheExecution

unit

SWE 594 31


CPU State

Executionunit

Multi-Core Architecture with Shared Cache

CPU State

Cache

Executionunit

SWE 594 32


Multi-Core with Hyper-Threading Technology

CPU State

CacheExecution

unit

CPU State CPU State

CacheExecution

unit

CPU State

SWE 594 33

SWE 594 34

Rank Site System Processors Rmax Rpeak

1

DOE/NNSA/LLNL eServer Blue Gene Solution

131072 280600 367000United States IBM

2

IBM Thomas J. Watson Research Center eServer Blue Gene Solution


3

DOE/NNSA/LLNL eServer pSeries p5 575 1.9 GHz


4

NASA/Ames Research Center/NAS SGI Altix 1.5 GHz, Voltaire Infiniband

10160 51870 60960United States SGI

5

Commissariat a l'Energie Atomique (CEA)NovaScale 5160, Itanium2 1.6 GHz, Quadrics

8704 42900 55705.6France Bull SA

6

Sandia National Laboratories PowerEdge 1850, 3.6 GHz, Infiniband

9024 38270 64972.8United States Dell

7

GSIC Center, Tokyo Institute of TechnologySun Fire X64 Cluster, Opteron 2.4/2.6 GHz, Infiniband

10368 38180 49868.8Japan NEC/Sun

8

Forschungszentrum Juelich (FZJ) eServer Blue Gene Solution

16384 37330 45875Germany IBM

9

Sandia National Laboratories Red Storm Cray XT3, 2.0 GHz

10880 36190 43520United States Cray Inc.

10

The Earth Simulator Center Earth-Simulator

5120 35860 40960Japan NEC

Top 10 Most Powerful Computers in the World (as of 6/2006)

SWE 594 35

Most Powerful Computers in the World (as of 11/2007)

SWE 594 36

Top 500 Lists

• http://www.top500.org/list/2007/11

• http://www.top500.org/list/2007/06

• ……..

SWE 594 37

Application Areas in Top 500 List

SWE 594 38

Top 500 Statistics

• http://www.top500.org/stats

SWE 594 39

Grid Computing• provide access to computing power and various resources

just like accessing electrical power from electrical grid

• Allows coupling of geographically distributed resources

• Provide inexpensive access to resources irrespective of their physical location or access point

• Internet & dedicated networks can be used to interconnect distributed computational resources and present them as a single unified resource

• Resources: supercomputers, clusters, storage systems, data resources, special devices

SWE 594 40

Grid Computing

• the GRID is, in effect, a set of software tools, which when combined with hardware, would let users tap processing power off the Internet as easily as the electrical power can be drawn from the electricty grid.

• Examples of Grids:

-TeraGrid (USA) : http://www.teragrid.org

-EGEE Grid (Europe) : http://www.eu-egee.org/

- TR-Grid (Turkey) : http://www.grid.org.tr/

- Sun Grid Compute Utility (Commercial, pay-per-use) http://www.network.com/

SWE 594

GRID COMPUTING

Power Grid Compute Grid

SWE 594 42

ArcheologyAstronomyAstrophysicsCivil ProtectionComp. ChemistryEarth SciencesFinanceFusionGeophysicsHigh Energy PhysicsLife SciencesMultimediaMaterial Sciences…

>250 sites48 countries>50,000 CPUs>20 PetaBytes>10,000 users>150 VOs>150,000 jobs/day

SWE 594 43

Cloud Computing

•Style of computing in which IT-related capabilities are provided “as a service”,allowing users to access technology-enabled services from the Internet ("in the cloud") without knowledge of, expertise with, or control over the technology infrastructure that supports them.

•General concept that incorporates software as a service (SaaS), Web 2.0 and other recent, well-known technology trends, in which the common theme is reliance on the Internet for satisfying the computing needs of the users.

SWE 594

Cloud Computing

• Virtualisation provides separation between infrastructure and user runtime environment

• Users specify virtual images as their deployment building blocks

• Pay-as-you-go allows users to use the service when they want and only pay for what they use

• Elasticity of the cloud allows users to start simple and explore more complex deployment over time

• Simple interface allows easy integration with existing systems

44

SWE 594

Cloud computing is about much more than technological capabilities.

Technology is the mechanism, but, as in any shift in business, the driver is economics.

Nicholas Carr,The author of “The Big Switch”

SWE 594

We want to pay only for what we useAnd we want to control it accurately.

Better Economics

SWE 594

Facing New Challenges

• Complexity of modern IT infrastructures: physical servers, virtual machines, clusters, Grids, geographical distribution

• Cost of electricity• Credit crunch• Further pressures to reduce costs• Openness to the acceptable security concept

SWE 594

SWE 594

The Grid/Cloud

Advantages

• Lower cost

• Access to larger infrastructure

– Faster calculations

– More storage

• Speed

– Faster calculations

– Easier provisioning

Disadvantages

• Very complicated

• Security

• Lack of confidence

– Trust

– Compatibility

SWE 594

Issue Classic Grid Computing Cloud computing

Why we need it?(The Problem)

To enable the R&D community to achieve its research goals in reasonable time.Computation over large data sets, or of paralleizable compute-intensive applications.

Reduce IT costs.On-demand scalability for all applications, including research, development and business applications.

Main Target Market

First - AcademiaSecond – certain industries

Mainly Industry

Business Model – Where the money comes from?

AcademiaSponsor-based (Mainly government money).

Industry paysInternal Implementations.

Hosted by commercial companies, paid-for by users. Based on the economies of scale and expertise. Only pay for what you need, when you need it: (On- Demand + Pay per Use).

Grid and Clouds

SWE 594

Example Cloud: Amazon Web Services

• EC2 (Elastic Computing Cloud) is the computing service of Amazon

– Based on hardware virtualisation

– Users request virtual machine instances, pointing to an image (public or private) stored in S3

– Users have full control over each instance (e.g. access as root, if required)

– Requests can be issued via SOAP and REST

51

SWE 594


•S3 (Simple Storage Service) is a service for storing and accessing data on the Amazon cloud– From a user’s point-of-view, S3 is independent

from the other Amazon services– Data is built in a hierarchical fashion, grouped in

buckets (i.e. containers) and objects– Data is accessible via various protocols

•Elastic Block Store– Locally mounted storage– Highly available

52

SWE 594


• Other AWS services:

– SQS (Simple Queue Service)

– SimpleDB

– Billing services: DevPay

– Elastic IP (Static IPs for Dynamic Cloud Computing)

– Multiple Locations

53

SWE 594


• Pricing information

http://aws.amazon.com/ec2/

54

SWE 594

“By 2012 ,80 percent of Fortune 1000 companies

will pay for some cloud computing service ,

And

30 percent of them will pay for

cloud computing infrastructure.”

Gartner, 2008

Cloud Market

SWE 594

EC2 – “Google of the Clouds”

According to Vogels (Amazon CTO), 370,000 developers have registered for Amazon Web Services since their start in 2002, and the company now spends more bandwidth on the developers than it does on e-commerce. http://www.theregister.co.uk/2008/06/26/amazon_trumpets_web_services/

In the last two months of 2007 usage of Amazon Web Services grew by 40%

$131 million revenues in Q1 from AWS 60,000 customersThe majority of usage comes from banks, pharmaceuticals and other large corporations

SWE 594

• - CIOs -> Do more with Less (Energy costs / Recession will boost it)

- Lower cost for Scalability

- Enterprise IT budget - Spending 80% on MAINTENANCE

- In average, we utilize only 15% of our computing resources capacity

- Peak Times economy

- The Enterprise IT is not its core business

- Psychology of Internet/Cloud trust (SalesForce, Gmail, Internet banking, etc.)

- Ideal for Developers

Why Now? (Economy)

SWE 594 58

Models of Parallel Computers

1. Message Passing Model

- Distributed memory

- Multicomputer

2. Shared Memory Model

- Multiprocessor

- Multi-core

3. Theoretical Model

- PRAM

• New architectures: combination of 1 and 2.

SWE 594 59

Theoretical PRAM Model

• Used by parallel algorithm designers

• Algorithm designers do not want to worry about low level details: They want to concentrate on algorithmic details

• Extends classic RAM model

• Consist of :

– Control unit (common clock), synchronous

– Global shared memory

– Unbounded set of processors, each with its private own memory

SWE 594 60

Theoretical PRAM Model

• Some characteristics

– Each processor has a unique identifier, mypid=0,1,2,…

– All processors operate synhronously under the control of a common clock

– In each unit of time, each procesor is allowed to execute an instruction or stay idle

SWE 594 61

Various PRAM Models

weakest

strongest

EREW (exlusive read / exclusive write)

CREW (concurrent read / exclusive write)

CRCW (concurrent read / concurrent write)

Common (must write the same value)

Arbitrary (one processor is chosen arbitrarily)

Priority (processor with the lowest index writes)

(how write conflicts to the same memory location are handled)

SWE 594 62

Algorithmic Performance Parameters

• Notation

Input size

Time Complexity of the best sequential algorithm

Number of processors

Time complexity of the parallel algorithm when run on P processors

Time complexity of the parallel algorithm when run on 1 processors

SWE 594 63


• Speed-Up

• Efficiency

SWE 594 64


• Work = Processors X Time

– Informally: How much time a parallel algorithm will take to simulate on a serial machine

– Formally:

SWE 594 65


• Work Efficient:

– Informally: a work efficient parallel algorithm does no more work than the best serial algorithm

– Formally: a work efficient algorithm satisfies:

SWE 594 66


• Scalability:

– Informally, scalability implies that if the size of the problem is increased, the number of processors effectively used can be increased (i.e. there is no limit on parallelism)

– Formally, scalability means:

SWE 594 67


• Some remarks:

– Cost of scalable algorithm grows slowly as input size and the number of procesors are increased

– Level of ‘control parallelism’ is usually a constant independent of problem size

– Level of ‘data parallelism’ is an increasing function of problem size

– Data parallel algorithms are more scalable than control parallel algorithms

SWE 594 68

Goals in Designing Parallel Algorithms

• Scalability:

– Algorithm cost grows slowly, preferably in a polylogarithmic manner

• Work Efficient:

– We do not want to waste CPU cycles

– May be an important point when we are worried about power consumption or ‘money’ paid for CPU usage

SWE 594

•Array of N numbers can be summed in log(N) steps using

Summing N numbers in Parallel

x1 x2 x3 x4 x5 x6 x7 x8

x1+x2 x2 x3+x4 x4 x5+x6 x6 x7+x8 x8

x1+..+x4 x2 x3+x4 x4 x5+..+x8 x6 x7+x8 x8

x1+..+x8 x2 x3+x4 x4 x5+..+x8 x6 x7+x8 x8

step 1

step 2

step 3

result

N/2 processors

SWE 594

Prefix Summing N numbers in Parallel

x1 x2 x3 x4 x5 x6 x7 x8

x1+x2 x2+x3 x3+x4 x4+x5 x5+x6 x6+x7 x7+x8 x8

x1+..+x4 x2+..+x4 x3+..+x6 x4+..+x7 x5+..+x8 x6+..+x8 x7+x8 x8


step 1

step 2

step 3

•Computing partial sums of an array of N numbers can be done in log(N) steps using N processors

SWE 594

Prefix Paradigm for Parallel Algorithm Design

•Prefix computation forms a paradigm for parallel algorithm development, just like other well known paradigms such as:

– divide and conquer, dynamic programming, etc.

•Prefix Paradigm: – If possible, transform your problem to prefix type computation– Apply the efficient logarithmic prefix computation

•Examples of Problems solved by Prefix Paradigm:

– Solving linear recurrence equations– Tridiagonal Solver– Problems on trees– Adaptive triangular mesh refinement

SWE 594

Solving Linear Recurrence Equations

• Given the linear recurrence equation:

• we can rewrite it as:

• if we expand it, we get the solution in terms of partial products of coefficients and the initial values z1 and z0 :

• use prefix to compute partial products

z a z b zi i i i i 1 2

z

z

a b z

zi

i

i i i

i

1

1

21 0

z

z

a b a b a b a b z

zi

i

i i i i i i

1

1 1 2 2 2 2 1

01 0 1 0 1 0 1 0...

SWE 594

Pointer Jumping Technique

•A linked list of N numbers can be prefix-summed in log(N)steps using N processors

step 1

step 3

x1 x2 x3 x4 x5 x6 x7 x8

x1+..+x4 x2+..+x5 x3+..+x6 x4+..+x7 x5+..+x8 x6+x7 x7+x8 x8

step 2

x1+.x2 x2+x3 x3+x4 x4+x5 x5+x6 x6+x7 x7+x8 x8


SWE 594

Euler Tour Technique

b d

a

c

f ge

h i

Tree Problems:

•Preorder numbering•Postorder numbering•Number of Descendants•Level of each node

•To solve such problems, first transform the tree by linearizing it into a linked-list and then apply the prefix computation

SWE 594

Computing Level of Each Node by Euler Tour Technique

b d

a

c

f ge

h i

1

-1

1

-11

-1 1

1

-11

-1

-1

1 -11

-1 weight assignment:

1 -1

level(v) = pw(<v,parent(v)>)level(root) = 0

w(<u,v>)

pw(<u,v>)

igba d a c a g h g b f b e b a1-1 -1 1 -1 -1 -1 1 -1 1 1 -1 1 -1 1 1

1212123232101010

initial weights:

prefix:

SWE 594

Computing Number of Descendants by Euler Tour Technique

b d

a

c

f ge

h i

0

1

0

10

1 0

0

10

1

1

0 10

1 weight assignment:

0 1

# of descendants(v) = pw(<parent(v),v>) - pw(<v,parent(v)>) # of descendants(root) = n

w(<u,v>)

pw(<u,v>)

igba d a c a g h g b f b e b a01 1 0 1 1 1 0 1 0 0 1 0 1 0 0

0011222334566778

initial weights:

prefix:

SWE 594

Preorder Numbering by Euler Tour Technique

b d

a

c

f ge

h i

1

0

1

01

0 1

1

01

0

0

1 01


1 0

preorder(v) = 1 + pw(<v,parent(v)>)preorder(root) = 1

w(<u,v>)

pw(<u,v>)


1223345566667788

initial weights:

prefix:

1

2

34

5

6 7

8 9

SWE 594

Postorder Numbering by Euler Tour Technique

b d

a

c

f ge

h i

0

1

0

10

1 0

0

10

1

1

0 10


0 1

postorder(v) = pw(<parent(v),v>)

postorder(root) = n

w(<u,v>)

pw(<u,v>)


0011222334566778

initial weights:

prefix:

9

6

1 25

3 4

78

cmpe 49b sp. top. in cmpe: multi-core programming

Documents