april 26, 20051 cse8380 parallel and distributed processing presentation hong yue department of...
TRANSCRIPT
April 26, 2005 1
CSE8380 Parallel and Distributed CSE8380 Parallel and Distributed ProcessingProcessing
PresentationPresentation
CSE8380 Parallel and Distributed CSE8380 Parallel and Distributed ProcessingProcessing
PresentationPresentation Hong YueHong Yue
Department of Computer Science & EngineeringDepartment of Computer Science & EngineeringSouthern Methodist UniversitySouthern Methodist University
April 26, 2005 2
Parallel Processing Parallel Processing MultianalysisMultianalysis
--- --- Compare Parallel Processing with Compare Parallel Processing with
Sequential ProcessingSequential Processing
Parallel Processing Parallel Processing MultianalysisMultianalysis
--- --- Compare Parallel Processing with Compare Parallel Processing with
Sequential ProcessingSequential Processing
April 26, 2005 3
Why did I select this topic?Why did I select this topic?Why did I select this topic?Why did I select this topic?
April 26, 2005 4
Outline Definition Characteristics of Parallel Processing and
Sequential Processing Implementation of Parallel Processing and
Sequential Processing Performance of Parallel Processing and
Sequential Processing Parallel Processing Evaluation Major Application of parallel processing
April 26, 2005 5
Definition Parallel Processing Definition
Parallel Processing refers to the simultaneous use of multiple processors to execute the same task in order to obtain faster results. These processors either communicate each other to solve a problem or work completely independent, under the control of another processor which divides the problem into a number of parts to other processors and collects results from them.
April 26, 2005 6
Definition .2
Sequential Processing Definition
Sequential processing refers to a computer architecture in which a single processor carries out a single task by series of operations in sequence. It is also called serial processing.
April 26, 2005 7
Characteristics of Parallel Processing and Sequential
Processing
Characteristics of Parallel Processing
● Each processor can perform tasks
concurrently.
● Tasks may need to be synchronized.
● Processors usually share resources, such as data, disks, and other devices.
April 26, 2005 8
Characteristics of Parallel Processing and Sequential
Processing .2
Characteristics of Sequential Processing
● Only one single processor performs task.
● The single processor performs a single
task.
● Task is executed in sequence.
April 26, 2005 9
Implementation of parallel processing and sequential
processing
Executing single task
In sequential processing, the task is executed as a single large task.
In parallel processing, the task is divided into multiple smaller tasks, and each component task is executed on a separate processor.
April 26, 2005 10
Implementation of parallel processing and sequential
processing.2
Total Elapsed Time
Processor 1
Task (runtime)
Figure 1 Sequential Processing of a Large Task
April 26, 2005 11
Implementation of parallel processing and sequential
processing .3
Total Elapsed Time
Processor 1
Processor 2
Processor 3
Processor 4
Processor 5
Processor 6
Processor 7
Figure 2 Parallel Processing: Executing Component Tasks in Parallel
Component task (runtime)
April 26, 2005 12
Implementation of parallel processing and sequential
processing.4
Executing multiple independent task
● In sequential processing, independent tasks compete for a single resource. Only task 1 runs without having to wait. Task 2 must wait until task 1 has completed; task 3 must wait until tasks 1 and 2 have completed, and so on.
April 26, 2005 13
Implementation of parallel processing and sequential
processing .5
Executing multiple independent task
● By contrast, in parallel processing, for example, a parallel server on a symmetric multiprocessor, more CPU power is assigned to the tasks. Each independent task executes immediately on its own processor: no wait time is involved.
April 26, 2005 14
Implementation of parallel processing and sequential
processing .6
Total Elapsed Time
Processor 1
Processor 2
Processor 3
Processor 4
Processor 5
Processor 6
Processor 7
Figure 3 Sequential Processing of Multiple Independent Tasks
Task (runtime)
Wait
April 26, 2005 15
Implementation of parallel processing and sequential
processing .7
Total Elapsed Time
Figure 4 Parallel Processing: Executing Independent Tasks in Parallel
Task (runtime)
Processor 1
Processor 2
Processor 3
Processor 4
Processor 5
Processor 6
Processor 7
April 26, 2005 16
Performance of parallel processing and sequential
processing
Sequential Processing Performance
● Take long time to execute task. ● Can’t handle too large task. ● Can’t handle large loads well. ● Return is diminishing. ● More increasingly expensive to make a
single processor faster.
April 26, 2005 17
Performance of parallel processing and sequential
processing .2
Solution:
using parallel processing - use lots of relatively fast, cheap processors in parallel.
April 26, 2005 18
Performance of parallel processing and sequential
processing .3
Parallel Processing Performance ● Cheaper, in terms of price and performance. ● Faster than equivalently expensive uniprocessor
machines. ● Scalable. The performance of a particular program
may be improved by execution on a large machine.
April 26, 2005 19
Performance of parallel processing and sequential
processing .4
Parallel Processing Performance
● Reliable. In theory if processors fail we can simply use others.
● Can handle bigger problems.
● Communicate with each other readily, important in calculations.
April 26, 2005 20
Parallel Processing Evaluation
Several ways to evaluate the parallel processing performance:
● Scale-up ● Speedup ● Efficiency ● Overall solution time ● Price/performance
April 26, 2005 21
Parallel Processing Evaluation .2
Scale-up
Scale-up is enhanced throughput, refers to the ability of a system n times larger to perform an n times larger job, in the same time period as the original system. With added hardware, a formula for scale-up holds the time constant, and measures the increased size of the job which can be done.
April 26, 2005 22
Parallel Processing Evaluation .3
Sequential System:
Hardware
Time
100% Task
Parallel System:
Hardware
HardwareTime
Time
200% Task
Figure 5 Scale-up
April 26, 2005 23
Parallel Processing Evaluation .4
Scale-up measurement formula:
Scale-up =
Transaction volume of multiprocessors
Transaction volume of uniprocessor
April 26, 2005 24
Parallel Processing Evaluation .5
For example, if the uniprocessor system can process 100 transactions in a given amount of time, and the parallel system can process 200 transactions in this amount of time, then the value of scale-up would be equal to 200/100 = 2.
Value 2 indicates the ideal of linear scale-up: when twice as much, hardware can process twice the data volume in the same amount of time.
April 26, 2005 25
Parallel Processing Evaluation .6
Speedup
Speedup, the improved response time, defined as the time it takes a program to execute in sequential (with one processor) divided by the time it takes to execute in parallel (with many processors). It can be achieved by two ways: breaking up a large task into many small fragments and reducing wait time.
April 26, 2005 26
Parallel Processing Evaluation .7
Sequential System:
Hardware
Time
100% Task
Parallel System:
Hardware
Hardware
Time
Time
50% Task
50% Task
Figure 6 Speedup
April 26, 2005 27
Parallel Processing Evaluation .8
Speedup measurement formula:
Elapsed time of a uniprocessorSpeedup = Elapsed time of the multiprocessors
April 26, 2005 28
Parallel Processing Evaluation .9
For example, if the uniprocessor took 40 seconds to perform a task, and two parallel systems took 20 seconds, then the value of speedup = 40 / 20 = 2.
Value 2 indicates the ideal of linear speedup: when twice as much, hardware can perform the same task in half the time.
April 26, 2005 29
Parallel Processing Evaluation .10
Workload Scale-up Speedup
OLTP Yes No
DSS Yes Yes
Batch (Mixed) Yes Possible
Parallel Query Yes Yes
Table 1 Scale-up and Speedup for Different Types of Workload
April 26, 2005 31
Parallel Processing Evaluation .12
Amdahl’s Law Amdahl's Law is a law governing the
speedup of using parallel processors on a problem, versus using only one sequential processor. Amdahl’s law attempts to give a maximum bound for speedup from the nature of the algorithm:
April 26, 2005 32
Parallel Processing Evaluation .13
Amdahl’s Law
S: purely sequential part P: parallel partS + P = 1 (for simplicity)
Maximum speedup =
S + P
S +
P
n
=
1
S +
P
n
April 26, 2005 34
Parallel Processing Evaluation .15
Gustafson’s Law If the size of a problem is scaled up as the
number of processors increases, speedup very close to the ideal speedup is possible.
That is, a problem size is virtually never
independent of the number of processors.
April 26, 2005 35
Parallel Processing Evaluation .16
Gustafson’s Law
Maximum speedup =
S + (P * n)
S + P
= n + (1 - n) * S
April 26, 2005 36
Parallel Processing Evaluation .17
Efficiency
The relative efficiency can be a useful measure as to what percentage of a processor’s time is being spent in useful computation.
Efficiency =
Speedup * 100
Number of processors
April 26, 2005 37
Parallel Processing Evaluation .18
Figure 9 Optimum efficiency & actual efficiency
April 26, 2005 38
Parallel Processing Evaluation .19
Figure 10 Optimum number of processors in actual speedup
April 26, 2005 39
Parallel Processing Evaluation .20
Problems in Parallel Processing
Parallel processing is like a dog’s walking on its hind legs. It is not done well, but you are surprised to find it done at all.
----Steve Fiddes (University of Bristol)
April 26, 2005 40
Parallel Processing Evaluation .21
Problems in Parallel Processing
● Its software is heavily platform-dependent and has to be written for a specific machine.
● It also requires a different, more difficult
method of programming, since the software needs to appropriately, through algorithms, divide the work across each processor.
April 26, 2005 41
Parallel Processing Evaluation .22
Problems in Parallel Processing
● There isn't a wide array of shrink-wrapped software ready for use with parallel machines.
● Parallelization is problem-dependent and cannot be automated.
● Speedup is not guaranteed.
April 26, 2005 42
Parallel Processing Evaluation .23
Solution 1:
● Decide which architecture is most appropriate for a given application.
The characteristics of application should drive decision as to how it should be parallelized; the form of the parallelization should then determine what kind of underlying system, both hardware and software, is best suited to running your parallelized application.
April 26, 2005 44
Major Applications of parallel processing
Clustering
● Clustering is a form of parallel processing that takes a group of workstations connected together in a local-area network and applies middleware to make them act like a parallel machine.
April 26, 2005 45
Major Applications of parallel processing .2
Clustering
Clustering is a form of parallel processing that takes a group of workstations connected together in a local-area network and applies middleware to make them act like a parallel machine.
April 26, 2005 46
Major Applications of parallel processing .3
Clustering ● Parallel processing using Linux Clusters can yield
supercomputer performance for some programs that perform complex computations or operate on large data sets. And it can accomplish this task by using cheap hardware.
● Clustering can be used at night when networks are idle, it is an inexpensive alternative to parallel-processing machines.
April 26, 2005 47
Major Applications of parallel processing .4
Clustering can work with two separate but similar implementations:
● A Parallel Virtual Machine (PVM), is an environment that allows messages to pass between computers as it would in an actual parallel machine.
● A Message-Passing Interface (MPI), allows programmers to create message-passing parallel applications, using parallel input/output functions and dynamic process management.
April 26, 2005 48
Reference
Andrew Boucher, “Parallel Machines”
Stephane vialle, “Past and Future Parallelism Challenges to Encompass sequential Processor evolution”