sreerag parallel programming
TRANSCRIPT
1
PRINCIPLES OF PROGRAMMING LANGUAGES
MODULE 5
PARALLEL PROGRAMMING
Presented by : Sreerag Gopinath P.C,
Semester VIII,
Computer Science & Engineering,
SJCET, Palai.
2
CONTENTS
11.2 PARALLEL PROGRAMMING
11.2.1 CONCURRENT EXECUTION
11.2.2 GUARDED COMMANDS
11.2.3 ADA OVERVIEW
11.2.4 TASKS
11.2.5 SYNCHRONIZATION OF TASKS
Reference:
Programming Languages – Design and Implementation, Fourth Edition
3
MOTIVATION FOR PARALLEL PROGRAMMING
Moore’s law: “Capacity and number of transistors in a chip (Computational power) for a given cost doubles ever 18 months”
Moore’s law in action:
Courtesy: http://www.engr.udayton.edu/faculty/jloomis/ece446/notes/intro/moore.html
4
PARALLEL PROGRAMMING – AN INTRODUCTION
• Parallel / Concurrent Programs : No single execution sequence as in Sequential programs. Several subprograms execute concurrently.
• Computer systems with increased capability – Multiprocessor systems, Distributed / Parallel computer system.
• Advantages: 1) May make programs simpler to design.
2) Programs run faster than a matching sequential program.
5
• Our concern – concurrent execution within a single program.
• Major stumbling block – lack of programming language constructs for building such systems.
C - fork()
Ada - tasks, concurrent execution
OUR CONCERN
6
4 STEPS IN CREATING A PARALLEL PROGRAM
P0
Tasks Processes Processors
P1
P2 P3
p0 p1
p2 p3
p0 p1
p2 p3
Partitioning
Sequentialcomputation
Parallelprogram
Assignment
Decomposition
Mapping
Orchestration
• Decomposition of computation in tasks• Assignment of tasks to processes• Orchestration of data access, communication, synch.• Mapping processes to processors
7
PRINCIPLES OF PARALLEL PROGRAMMING LANGUAGES
1. Variable definitions : mutable - values changed during program execution
definitional - assigned a value only once
2. Parallel composition : parallel statements like and
3. Program Structure : transformational - transform input data into output value
reactive - pgm reacts to external stimuli (events)
4. Communication : shared memory - common data objects
messages - individual copy of data obj. & pass val
5. Synchronization : pgms must be able to order execution of various threads of control - determinism / non-determinism
8
CONCURRENT EXECUTION
• Mechanism : Construct that allows parallel execution - the and statement,
statement1 and statement2 and … and statementn
• Concurrent task OS specification
call ReadProcess and
call WriteProcess and
call ExecuteUserProgram;
• Correct Data Handling
x := 1;
x := 2 and y := x+x; (*)
print (y);
9
IMPLEMENTATION OF THE ‘and’ CONSTRUCT
1. Execution in sequence (no assumption on order of execution)
while MoreToDo do
MoreToDo := false;
call ReadProcess;
call WriteProcess;
call ExecuteUserProgram
end
2. Parallel execution primitives of the underlying OS
fork ReadProcess;
fork WriteProcess;
fork ExecuteUserProgram;
wait /* for all 3 to terminate*/
10
GUARDED COMMANDS• Dijkstra - True non determinacy (1970) , guarded command
• Nondeterministic execution – alternative execution paths are possible
• Guards - If B is a guard (condition) and S is a command (statement), guarded command
B S
• Guarded if statement – If Bi is a set of conditions and Si is a set of statements,
if B1 S1 || B2 S2 || … || Bn Sn fi
• Guarded Repetition statement
do B1 S1 || B2 S2 || … || Bn Sn od
• No true implementation of guarded commands as defined by Dijkstra in common PLs.
11
ADA OVERVIEW
• General purpose language, although originally designed for military applications.
• Block structure & data type mechanism similar to Pascal.
• Extensions for real-time & distributed applications.
• More secure form of data encapsulation than Pascal.
• Recent versions – ability to develop objects & provide for method inheritance.
• Major features - tasks & concurrent execution,
real time task control,
exception handling,
abstract data types.
12
ADA – A BRIEF LANGUAGE OVERVIEW• Supports constrn. of large programs by teams of programmers.
• Program designed as a collection of packages, rep. an abstract data type or data objects.
• Program consists of a single procedure that serves as a main program.
• Broad range of built in data types – integers, reals, enumerations, Booleans, arrays, records, character strings, pointers.
• Sequence control within subprograms – expressions & stmt level ctrl structures similar to Pascal.
- concurrently executing tasks controlled by a time clock & other scheduling
mechanisms.
• Exception Handling – extensive set of features.
• Data-control structures – static block structure organization as in Pascal.
• Nonlocal references to type names, subprogram names, identifiers in package
• Central stack for each separate task.
• Heap storage area for programmer constructed data objects.
13
TASKS
• Task : Each subprogram that can execute concurrently with other subprograms is called a task (or sometimes a process )
• A task is dependent of the task that initiated it.
THE FORK JOIN MODEL
14
TASK MANAGEMENT
• Task definition defines how the task synchronizes & communicates with others
• Task definition in Ada
task Name is
- Declarations for synchronization and communication
end;
task body Name is
- Usual local declarations as found in any subprogram
begin – Sequence of statements
end;
• Initiating execution of a task ( PL/I)
call B (parameters) task;
Ada – No explicit call statement needed.
- Concurrent execution on entry into larger program structure
15
TASK MANAGEMENT (Contd..)
Multiple simultaneous activations of same task
PL/I : Repeated execution of call
Ada :
task type Terminal is
- Rest of definition in the same form as above
end;
…
A: Terminal;
B,C: Terminal;
Alternative
type TaskPtr is access Terminal; - Defines pointer type
NewTerm: TaskPtr := new Terminal; - Declares pointer variable
16
SYNCHRONIZATION OF TASKS• Synchronization is needed for tasks running asynchronously to coordinate their
activities.
• Consider, task A --- read in a batch of data
task B --- process each batch of data input from device
• SYNCHRONIZATION
1. Task B doesn’t start processing data before taskA has finished reading these in.
2. Task A doesn’t overwrite data taskB is still processing
• SYNCHRONIZATION MECHANISMS
1. Interrupts
2. Semaphores
3. Messages
4. Guarded Commands
5. Rendezvous
17
INTERRUPTS
• Common mechanism in computer hardware.
• Task A sends event to task B :
Interrupt control transfer resume execution.
• Disadvantages as a sync. Mechanism in high level languages-
1. Confusing program structure – separate interrupt handling code
2. Waiting for an interrupt - Busy waiting loop ---does nothing else
3. Data shared between task body and interrupt has to be protected.
• High-level languages usually provide other synchronization mechanisms.
18
SEMAPHORES
• Consists of two parts –
1. An integer counter - count number of signals sent but not received.
2. A queue of tasks - waiting for signals to be sent.
• Two primitive operations on a semaphore object P:
1. signal(P) - Tests value of counter in P.
If zero, remove first task in task queue, and resume its execution.
If not zero or if queue empty, increment counter by one.
(indicates signal sent but not received)
2. wait(P) - Tests value of counter in P.
If nonzero, decrement counter by one (indicates signal received)
If zero, insert task at the end of task queue, and suspend task.
19
SEMAPHORES (Contd..)
Synchronization Problem
2 binary semaphores - StartB - used by Task A to signal input complete
StartA – used by Task B to signal processing complete
task A;
begin
- input first data set
loop
signal(StartB) – Invoke task B
- Input next data set
wait(StartA) – Wait until taskB
finishes with data
endloop;
end A;
task B;
begin
loop
wait(StartB) – Wait for task A to read data
-Process data
signal(StartA) – Tell Task A to
continue
endloop;
end A;
20
SEMAPHORES (Contd..)
Disadvantages for use in high level programming of tasks
1. A task can wait for only one semaphore at a time.
2. Deadlocks may occur if a task fails to signal at the appropriate point.
3. Increasingly difficult to understand, debug and verify.
4. Semantics of signal and wait imply shared memory, not necessarily so in multiprocessor systems and computer networks.
In essence, semaphore is a relatively low-level synchronization construct that is adequate primarily in simple situations.
21
MESSAGES
• A message is a transfer of information from one task to another.
• The task remains free to continue executing when not synchronising.
• A message is placed into the pipe (or message queue) by a send command.
• A message is accepted by a waiting task using a receive command.
task Producer;
begin
loop - while more to read
- Read new data;
send( Consumer, data)
endloop;
end Producer
task Consumer;
begin
loop - while more to process
receive (Producer, data)
- Process new data
endloop;
end Consumer
PRODUCER CONSUMER PROBLEM
22
GUARDED COMMANDS
• Adds nondeterminacy into programming.
• Good model for task synchronization.
• The guarded if command in Ada is termed a select statement with the general form
select
when condition1 ==> statement1
or when condition2 ==> statement2
…
or when conditionn ==> statementn
else statementn+1 - optional else clause
end select;
23
RENDEZVOUS
• When two tasks synchronize their actions for a brief period, that synchronization is termed a rendezvous.
• Similar to message, but requires a synchronization action with each message.
A B A B
24
RENDEZVOUS (Contd…)
• A rendezvous point in B is called an entry ( in this example, DataReady)
• When Task B is ready to begin processing a new batch of data, it must execute an accept statement:
accept DataReady do
- statements to copy new data from A into local data area of B
end;
• When Task A has completed input of a batch of data, it must execute the
entry call: DataReady.
• The Rendezvous:
Task B reaches accept stmt wait until Task A --- entry call: DataReady
Task A reaches DataReady wait until Task B reaches accept stmt.
A continues to wait while B executes all stmts contained within the do…end
of the accept stmt, and both A and B continue their separate executions.
25
RENDEZVOUS (Contd…)
select
when Device1Status = ON ==> accept Ready1 do … end;
or when Device2Status = ON ==> accept Ready2 do … end;
or when Device3Status = connected ==> accept Ready3
do … end;
else … - No device is ready; do something else
end select;
Conditional rendezvous on the status of each device
26
TASKS & REAL TIME PROCESSING
• A program that must interact with i/o devices or other tasks within some fixed time period is said to be operating in real time.
• In real time computer systems, h/w failure of an i/o device leads to a task’s being abruptly terminated. If other tasks wait on such a failed task, the entire systems of tasks may deadlock & cause the system to crash.
• Real-time processing requires that the language hold some explicit notion of time.
• Ada - package called Calendar that includes a type Time and a function clock.
A task waiting for a rendezvous may watch the clock, as in
select DataReady;
or delay 0.5; - Wait at most 0.5 seconds
end select;
27
TASKS & SHARED DATA
• Tasks sharing data present special problems due to concurrent execution involved.
• There are two issues:
1. Storage Management Single Stack
Multiple stacks
Single Heap
2. Mutual Exclusion Critical Regions
Monitors
Message Passing
28
STORAGE MANAGEMENT IN TASKS
stack
heap
Single Stack Multiple Stacks Single Heap
stack
stack
stack
heap
stack1
stack2
stack3
act rec
act rec
act rec
act rec
act rec
act rec
act rec
29
STORAGE MANAGEMENT IN TASKS (Contd…)
1. Single stack - * C, Pascal
* If Stack & heap meet, no more space---program terminates
* Efficient use of space
2. Multiple stacks - * Used when there is enough memory
* If any stack overlaps next memory segment, program terminates
* Effective solution with today’s modern virtual memory systems
3. Single Heap - * Systems with limited memory
* Used for early PL/I compilers
* High overhead
* Memory fragmentation can be a problem in unique sized ARs.
30
CACTUS STACK MODEL OF MULTIPLE TASKS
Task 3
Task 1
Task 2
Task 4
31
MUTUAL EXCLUSION
• If TaskA and TaskB each have access to a single data object X, then A and B must synchronize their access to X so that TaskA is not in the process of assigning a new value to X while TaskB is simultaneously referencing that value or assigning a different value.
• To ensure that two tasks do not simultaneously attempt to access and update a shared data object, one task must be able to gain exclusive access to the data object while it manipulates it.
32
CRITICAL REGIONS• A Critical region is a sequence of program statements within a task where the task is operating on some data object shared with other tasks.
• If a critical region in Task A is manipulating data object X, then mutual exclusion requires that no other task be simultaneously executing a critical region that also manipulates X.
• During execution of Task A, A must wait until any other task has completed a critical region that manipulates X.
• As Task A begins its critical region, all other tasks must be locked out so that they cannot enter their critical regions (for variable X) until A has completed its critical region.
• Critical regions may be implemented in tasks by associating a semaphore with each shared data object.
33
MONITORS
• A monitor is a shared data object together with the set of operations that may manipulate it.
• Similar to a data object defined by an abstract data type.
• To enforce mutual exclusion, it is only necessary to require that at most one of the operations defined for the data object may be executing at any given time.
• Mutual exclusion and encapsulation constraints require a monitor to be represented as a task.
34
MONITORS (Contd…)
task TableManager is
entry EnterNewItem(…);
entry FindItem(…);
end;
task body TableManager is
BigTable: array(…) of
procedure Enter(…) is
- Statements to enter item in BigTable
end Enter;
function Find(…) returns … is
- Statements to find item in BigTable
end Find;
begin
- Statements to initialise BigTable
loop – Loop forever to process entry requests
select
accept EnterNewItem(…) do
- Call Enter to enter received in BigTable
end;
or accept FindItem(…) do
- Call Find to look up received item in BigTable
end;
end select;
end loop;
end TableManager;
35
MESSAGE PASSING
• The idea is to prohibit shared data objects and provide only the sharing of data values through passing the values as messages.
• Mutual exclusion comes naturally because each data object is owned by exactly one task, and no other task may access the data object directly.
Copy of data values
Local copyData object
Processing
Task A Task B
Processed Copy
36
THANK YOU !!!