tutorial: parallel & distributed simulation systems: from chandy/misra to the high level...

tutorial:

Parallel & Distributed Simulation Systems:From Chandy/Misra to the High Level

Architecture and Beyond

Richard M. Fujimoto

College of Computing

Georgia Institute of Technology

Atlanta, GA 30332-0280

[email protected]

References

R. Fujimoto, Parallel and Distributed Simulation Systems, Wiley Interscience, 2000.

(see also http://www.cc.gatech.edu/classes/AY2000/cs4230_spring)

HLA:

F. Kuhl, R. Weatherly, J. Dahmann, Creating Computer Simulation Systems: An Introduction to the High Level Architecture for Simulation, Prentice Hall, 1999.

(http://hla.dmso.mil)

Part I:Introduction

Part II:Time Management

Part III:Distributed Virtual Environments

Outline

Parallel and Distributed Simulation

Distributed simulation involves the execution of a single simulation program on a collection of loosely coupled processors (e.g., PCs interconnected by a LAN or WAN).

Replicated trials involves the execution of several, independent simulations concurrently on different processors

Parallel simulation involves the execution of a single simulation program on a collection of tightly coupled processors (e.g., a shared memory multiprocessor).

M M M

P P P P

simulationmodel

parallelprocessor

Reasons to Use Parallel / Distributed SimulationEnable the execution of time consuming simulations that could

not otherwise be performed (e.g., simulation of the Internet)• Reduce model execution time (proportional to # processors)• Ability to run larger models (more memory)

Enable simulation to be used as a forecasting tool in time critical decision making processes (e.g., air traffic control)

• Initialize simulation to current system state• Faster than real time execution for what-if experimentation• Simulation results may be needed in seconds

Create distributed virtual environments, possibly including users at distant geographical locations (e.g., training, entertainment)

• Real-time execution capability• Scalable performance for many users & simulated entities

Geographically Distributed Users/Resources

WAN interconnect

LAN interconnect

Server architecture

Cluster of workstationson LAN

Distributed architecture

Distributed computers

Geographically distributed users and/or resources are sometime needed• Interactive games over the Internet• Specialized hardware or databases

Stand-Alone vs. Federated Simulation Systems

Federated Simulation Systems

Run Time Infrastructure-RTI(simulation backplane)

• federation set up & tear down• synchronization, message ordering• data distribution

Simulator 1Simulator 2

Simulator 3

Interconnect autonomous, heterogeneous simulators• interface to RTI software

Stand-AloneSimulation System

Homogeneous programming environment• simulation language

Parallel simulation environment

Process 1

Process 2

Process 3

Process 4

Distributed Virtual Environments (DVEs)

• Networked interactive, immersive environments

• Scalable, real-time performance

• Create virtual worlds that appear realistic

• Typical applications– Training– Entertainment– Social interaction

Principal Application DomainsParallel Discrete Event

Simulation (PDES)• Discrete event simulation to

analyze systems• Fast model execution (as-

fast-as-possible)• Produce same results as a

sequential execution• Typical applications

– Telecommunication networks– Computer systems– Transportation systems– Military strategy and tactics

Chandy/Misra/Bryantalgorithm

Time Warp algorithm

early experimental data

second generation algorithms

making it fast andeasy to use

High Performance Computing Community

1975 1980 1985 1990 1995 2000

Historical Perspective

SIMulator NETworking (SIMNET)(1983-1990)

Distributed Interactive Simulation (DIS)Aggregate Level Simulation Protocol (ALSP)

(1990 - 1997ish)

High Level Architecture(1996 - today)

Defense Community

Adventure(Xerox PARC)

Dungeons and DragonsBoard Games

Multi-User Dungeon (MUD)Games

Internet & Gaming Community

Multi-User Video Games


• Parallel discrete event simulation• Conservative synchronization• Optimistic synchronization• Time Management in the High Level Architecture

Time

• physical time: time in the physical system– Noon, December 31, 1999 to noon January 1, 2000

• simulation time: representation of physical time within the simulation– floating point values in interval [0.0, 24.0]

• wallclock time: time during the execution of the simulation, usually output from a hardware clock (e.g., GPS)– 9:00 to 9:15 AM on September 10, 1999

physical system

• physical system: the actual or imagined system being modeled• simulation: a system that emulates the behavior of a physical system

simulation

main(){ ...double clock;...

}

Paced vs. Unpaced ExecutionModes of execution• As-fast-as-possible execution (unpaced): no fixed

relationship necessarily exists between advances in simulation time and advances in wallclock time

• Real-time execution (paced): each advance in simulation time is paced to occur in synchrony with an equivalent advance in wallclock time

• Scaled real-time execution (paced): each advance in simulation time is paced to occur in synchrony with S * an equivalent advance in wallclock time (e.g., 2x wallclock time)

Here, focus on as-fast-as-possible; execution can be paced to run in real-time (or scaled real-time) by inserting delays

Discrete Event Simulation FundamentalsDiscrete event simulation: computer model for a system where changes in the state of the system occur at discrete points in simulation time.

Fundamental concepts:• system state (state variables)• state transitions (events)• simulation time: totally ordered set of values representing time in the

system being modeled (physical system)• simulator maintains a simulation time clock

A discrete event simulation computation can be viewed as a sequence of event computations

Each event computation contains a (simulation time) time stamp indicating when that event occurs in the physical system.

Each event computation may:(1) modify state variables, and/or(2) schedule new events into the simulated future.

A Simple DES Example

H0 H1

Time Event

0 H0 Send Pkt

100 Done

Time Event

100 Done6 H0 Retrans

1 H1 Recv Pkt

Time Event


2 H0 Recv Ack

Time Event


1 H1 Send Ack

Time Event

100 Done

2 H0 Send Pkt

• Simulator maintains event list• Events processed in simulation time order• Processing events may generate new events• Complete when event list is empty (or some other termination condition)

Parallel Discrete Event SimulationA parallel discrete event simulation program can be viewed as a collection of sequential discrete event simulation programs executing on different processors that communicate by sending time stamped messages to each other

“Sending a message” is synonymous with “scheduling an event”

Parallel Discrete Event Simulation Example

logical process

LP (Subnet 1)

LP(Subnet 3)

LP(Subnet 2)

time stamped event (message)

all interactions between LPs must be via messages (no shared state)

Recv Pkt@15

Simulation

physical process interactions among physical processes

Physical system

The “Rub”Golden rule for each logical process:

“Thou shalt process incoming messages in time stamp order!!!” (local causality constraint)

Parallel Discrete Event Simulation Example

logical process

LP (Subnet 1)

LP(Subnet 3)

LP(Subnet 2)

time stamped event (message)

all interactions between LPs must be via messages (no shared state)

Recv Pkt@15

Simulation

physical process interactions among physical processes

Physical system

Safe to process???

The Synchronization ProblemLocal causality constraint: Events within each logical process must be

processed in time stamp order

Observation: Adherence to the local causality constraint is sufficient to ensure that the parallel simulation will produce exactly the same results as the corresponding sequential simulation*

Synchronization (Time Management) Algorithms• Conservative synchronization: avoid violating the local causality

constraint (wait until it’s safe)– 1st generation: null messages (Chandy/Misra/Bryant)– 2nd generation: time stamp of next event

• Optimistic synchronization: allow violations of local causality to occur, but detect them at runtime and recover using a rollback mechanism– Time Warp (Jefferson)– approaches limiting amount of optimistic execution

* provided events with the same time stamp are processed in the same order as in the sequential execution

Chandy/Misra/Bryant “Null Message” AlgorithmAssumptions• logical processes (LPs) exchanging time stamped events (messages)• static network topology, no dynamic creation of LPs• messages sent on each link are sent in time stamp order• network provides reliable delivery, preserves order

Observation: The above assumptions imply the time stamp of the last message received on a link is a lower bound on the time stamp (LBTS) of subsequent messages received on that link

Goal: Ensure LP processes events in time stamp order

9 8 2

45

H3logical

process one FIFOqueue per

incoming link

H3H2

H1

A Simple Conservative Algorithm

Observation: Algorithm A is prone to deadlock!

Algorithm A (executed by each LP):

Goal: Ensure events are processed in time stamp order:

WHILE (simulation is not over)

wait until each FIFO contains at least one message

remove smallest time stamped event from its FIFO

process that event

END-LOOP

9 8 2

45

H3logical

process

H1

H2• wait until a message is received from H2

• process time stamp 2 event



Deadlock Example

A cycle of LPs forms where each is waiting on the next LP in the cycle.

No LP can advance; the simulation is deadlocked.

9 8

H3(waitingon H1)

7H1(waitingon H2)

H2(waitingon H3)

1510

Deadlock Avoidance Using Null MessagesBreak deadlock by having each LP send “null” messages indicating a lower bound on the time stamp of future messages it could send.

9 8

H3(waitingon H1)

7H1(waitingon H2)

H2(waitingon H3)

1510

Assume minimum delay between hosts is 3 units of simulation time

• H3 initially at time 5

8

• H3 sends null message to H2 with time stamp 8

11

• H2 sends null message to H1 with time stamp 11• H1 may now process message with time stamp 7

Deadlock Avoidance Using Null MessagesNull Message Algorithm (executed by each LP):

Goal: Ensure events are processed in time stamp order and avoid deadlock

WHILE (simulation is not over)

wait until each FIFO contains at least one message

remove smallest time stamped event from its FIFO

process that event

send null messages to neighboring LPs with time stamp indicating a lower bound on future messages sent to that LP

(current time plus lookahead)

END-LOOP

The null message algorithm relies on a “lookahead” (minimum delay).

Lookahead Creep

9 8

H3(waitingon H1)

7H1(waitingon H2)

H2(waitingon H3)

1510

Assume minimum delay between hosts is 3 units of time

• H3 initially at time 5

0.5

5.5

• H3 sends null message, time stamp 5.5;

6.5


6.0

H2 sends null message, time stamp 6.0

7.0

H3 send null message, time stamp 7.0

7.5


If lookahead is small, there may be many null messages!

H1 can process time stamp 7 message

Five null messages to process a single event

Preventing Lookahead Creep: Next Event Time Information

9 8

H3(waitingon H1)

7H1(waitingon H2)

H2(waitingon H3)

1510

Observation: If all LPs are blocked, they can immediately advance to the time of the minimum time stamp event in the system

Lower Bound on Time Stamp

Given a snapshot of the computation, LBTS is the minimum among• Time stamp of any transient messages (sent, but not yet received)• Unblocked LPs: Current simulation time + lookahead• Blocked LPs: Time of next event + lookahead

No null messages, assume any LP can send messages to any other LP

When a LP blocks, compute lower bound on time stamp (LBTS) of messages it may later receive; those with time stamp < LBTS safe to process

5 6 7 8 9 10 11 12 13 14 15 16

98

1510

7

H1

H2

H3

H3@6(not blocked)

[email protected](blocked)

H1@5

transient

Simulation timeLBTS = ?LBTS = min (6, 10, 7) (assume zero lookahead)

Lower Bound on Time Stamp (LBTS)

cutmessage

LP4

wallclock time

Past

Future

LP3

LP2

LP1

cut point: an instant dividing process’s computation into past and future

cut: set of cut points, one per process

cut message: a message that was sent in the past, and received in the future

consistent cut: cut + all cut messages

cut value: minimum among (1) local minimum at each cut point and (2) time stamp of cut messages; non-cut messages can be ignored

It can be shown LBTS = cut value

LBTS can be computed asynchonously using a distributed snapshot algorithm (Mattern)

A Simple LBTS Algorithm

cutmessage

LP4

wallclock time

Past

Future

LP3

LP2

LP1

Initiator broadcasts start LBTS computation message to all LPs

Each LP sets cut point, reports local minimum back to initiator

Account for transient (cut) messages• Identify transient messages, include time stamp in minimum computation

– Color each LP (color changes with each cut point); message color = color of sender– An incoming message is transient if message color equals previous color of receiver– Report time stamp of transient to initiator when one is received

• Detecting when all transients have been received– For each color, LPi keeps counter of messages sent (Sendi) and received (Receivei)

– At cut point, send counters to initiator: # transients = (Sendi – Receivei)

– Initiator detects termination (all transients received), broadcasts global minimum

Another LBTS Algorithm

cutmessage

LP4

wallclock time

Past

Future

LP3

LP2

LP1

An LP initiates an LBTS computation when it blocks

Initiator: broadcasts start LBTS message to all LPs

LPi places cut point, report local minimum and (Sendi – Receivei) back to initiator

Initiator: After all reports received

if ( (Sendi – Receivei) = 0) LBTS = global minimum, broadcast LBTS value

Else Repeat broadcast/reply, but do not establish a new cut

Synchronous AlgorithmsBarrier Synchronization: when a process reaches a barrier synchronization, it must block until all other processors have also reached the barrier.

- barrier -

- barrier -- barrier -

wait waitwait

- barrier -

process 1 process 2 process 3 process 4

wallclocktime

Synchronous algorithmDO WHILE (unprocessed events remain)

barrier synchronization; flush all messages from the network

LBTS = min (Ni + LAi); Ni = time of next event in LPi; LAi = lookahead of LPi

S = set of events with time stamp ≤ LBTS

process events in S

endDO

Variations proposed by Lubachevsky, Ayani, Chandy/Sherman, Nicol

all i

Topology Information

Global LBTS algorithm is overly conservative: does not exploit topology information

• Lookahead = minimum flight time to another airport• Can the two events be processed concurrently?

– Yes because the event @ 10:00 cannot affect the event @ 10:45• Simple global LBTS algorithm:

– LBTS = 10:30 (10:00 + 0:30)– Cannot process event @ 10:45 until next LBTS computation

6:00

2:004:00

0:30

10:00

10:45

ORD

JFKLAX

SAN

Distance Between LPs

• Associate a lookahead with each link: LAB is the lookahead on the link from LPA to LPB

– Any message sent on the link from LPA to LPB must have a time stamp of TA + LAB where TA is the current simulation time of LPA

• A path from LPA to LPZ is defined as a sequence of LPs: LPA, LPB, …, LPY, LPZ

• The lookahead of a path is the sum of the lookaheads of the links along the path

• DAB, the minimum distance from LPA to LPB is the minimum lookahead over all paths from LPA to LPB

• The distance from LPA to LPB is the minimum amount of simulated time that must elapse for an event in LPA to affect LPB

Distance Between Processes

The distance from LPA to LPB is the minimum amount of simulated time that must elapse for an event in LPA to affect LPB

LPA

LPB

LPC

LPD

LPA

4

4

3

5

LPB

3

5

6

4

LPC

1

3

4

2

LPD

3

1

2

4

min (1+2, 3+1)

Distance Matrix:D [i,j] = minimum distance from LPi to LPj

LPA LPB

LPC LPD

13 14

2

2

3

4

11

13 15

• An event in LPY with time stamp TY depends on an event in LPX with time stamp TX if TX + D[X,Y] < TY

• Above, the time stamp 15 event depends on the time stamp 11 event, the time stamp 13 event does not.

Computing LBTSAssuming all LPs are blocked and there are no transient messages:

LBTSi=min(Nj+Dji) (all j) where Ni = time of next event in LPi

LPA

LPB

LPC

LPD

LPA

4

4

3

5

LPB

3

5

6

4

LPC

1

3

4

2

LPD

3

1

2

4

min (1+2, 3+1)

Distance Matrix:D [i,j] = minimum distance from LPi to LPj

LBTSA = 15 [min (11+4, 13+5)]

LBTSB = 14 [min (11+3, 13+4)]

LBTSC = 12 [min (11+1, 13+2)]

LBTSD = 14 [min (11+3, 13+4)]

Need to know time of next event of every other LP

Distance matrix must be recomputed if lookahead changes

LPA LPB

LPC LPD

13 14

2

2

3

4

11

13 15

Example

Using distance information:

• DSAN,JFK = 6:30

• LBTSJFK = 16:30 (10:00 + 6:30)

• Event @ 10:45 can be processed this iteration• Concurrent processing of events at times 10:00 and 10:45

6:00

2:004:00

0:30

10:00

10:45

ORD

JFKLAX

SAN

Lookahead

LP A

LP B

LP C

LP D

Logical Time

problem: limited concurrencyeach LP must process events in time stamp order

TA

possible messageOK to process

event

not OK to process yet

without lookahead

TA+LA

possible message

OK to process

with lookahead

Each LP A using declares a lookahead value LA; the time stamp of any event generated by the LP must be ≥ TA+ LA

• Used in virtually all conservative synchronization protocols• Relies on model properties (e.g., minimum transmission delay)

Lookahead is necessary to allow concurrent processing of events with different time stamps (unless optimistic event processing is used)

Speedup of Central Server Queueing Model Simulation

0

1

2

3

4

1 2 4 8 16 32 64 128 256

Number of Jobs in Network

Speedup

Optimized(deterministic servicetime)Optimized (exponentialservice time)

Classical (deterministicservice time)

Classical (exponentialservice time)

Deadlock Detection and Recovery Algorithm(5 processors)

Exploiting lookahead is essential to obtain good performance

Summary: Conservative Synchronization

• Each LP must process events in time stamp order• Must compute lower bound on time stamp (LBTS) of

future messages an LP may receive to determine which events are safe to process

• 1st generation algorithms: LBTS computation based on current simulation time of LPs and lookahead– Null messages– Prone to lookahead creep

• 2nd generation algorithms: also consider time of next event to avoid lookahead creep

• Other information, e.g., LP topology, can be exploited• Lookahead is crucial to achieving concurrent processing

of events, good performance

Conservative Algorithms

Con:• Cannot fully exploit available parallelism in the simulation

because they must protect against a “worst case scenario”• Lookahead is essential to achieve good performance• Writing simulation programs to have good lookahead can be

very difficult or impossible, and can lead to code that is difficult to maintain

Pro:• Good performance reported for many applications containing

good lookahead (queueing networks, communication networks, wargaming)

• Relatively easy to implement• Well suited for “federating” autonomous simulations, provided

there is good lookahead

Time Warp Algorithm (Jefferson)Assumptions• logical processes (LPs) exchanging time stamped events (messages)• dynamic network topology, dynamic creation of LPs OK• messages sent on each link need not be sent in time stamp order• network provides reliable delivery, but need not preserve order

Basic idea:• process events w/o worrying about messages that will arrive later• detect out of order execution, recover using rollback

process all available events (2, 4, 5, 8, 9) in time stamp order

9 8 2

45

H3logical

processH3H2

H1

41

Input Queue(event list)

18

straggler message arrives in the past, causing rollback

12 21 35

processed event

unprocessed event

Time Warp (Jefferson)

Adding rollback:

• a message arriving in the LP’s past initiates rollback

• to roll back an event computation we must undo:

– changes to state variables performed by the event;

– message sends

Each LP: process events in time stamp order, like a sequential simulator, except: (1) do NOT discard processed events and (2) add a rollback mechanism

State Queue

solution: checkpoint state or use incremental state saving (state queue)

snapshot of LP state

1212Output Queue(anti-messages) 19 42

solution: anti-messages and message annihilation (output queue)

anti-message

Anti-Messages

• Used to cancel a previously sent message• Each positive message sent by an LP has a corresponding anti-

message• Anti-message is identical to positive message, except for a sign bit• When an anti-message and its matching positive message meet in

the same queue, the two annihilate each other (analogous to matter and anti-matter)

• To undo the effects of a previously sent (positive) message, the LP need only send the corresponding anti-message

• Message send: in addition to sending the message, leave a copy of the corresponding anti-message in a data structure in the sending LP called the output queue.

42positive message

anti-message 42

1212

41


Output Queue(anti-messages)

19 42

18

1. straggler message arrives in the past, causing rollbackBEFORE

12 21 35 processed event

unprocessed event


anti-message

State Queue

Rollback: Receiving a Straggler Message

1212

21 35 41

19

18


Output Queue(anti-messages)

AFTER

5. resume execution by processing event at time 18

12

State Queue

2. roll back events at times 21 and 35 2(a) restore state of LP to that prior to processing time stamp 21 event

2(b) send anti-message

Case II: corresponding message has already been processed

– roll back to time prior to processing message (secondary rollback)

– annihilate message/anti-message pair

55

33 57

27 42 45

processed event

unprocessed event


anti-message

may cause “cascaded” rollbacks; recursively applying eliminates all effects of error

Processing Incoming Anti-Messages

Case I: corresponding message has not yet been processed

– annihilate message/anti-message pair

421. anti-message arrives

Case III: corresponding message has not yet been received

– queue anti-message

– annihilate message/anti-message pair when message is received

2. roll back events (time stamp 42 and 45) 2(a) restore state

2(b) send anti-message

3. Annihilate message and anti-message

Global Virtual Time and Fossil CollectionA mechanism is needed to:• reclaim memory resources (e.g., old state and events)• perform irrevocable operations (e.g., I/O)

Observation: A lower bound on the time stamp of any rollback that can occur in the future is needed.

Global Virtual Time (GVT) is defined as the minimum time stamp of any unprocessed (or partially processed) message or anti-message in the system. GVT provides a lower bound on the time stamp of any future rollback.

• storage for events and state vectors older than GVT (except one state vector) can be reclaimed

• I/O operations with time stamp less than GVT can be performed.• GVT algorithms are similar to LBTS algorithms in conservative

synchronization

Observation: The computation corresponding to GVT will not be rolled back, guaranteeing forward progress.

Time Warp and Chandy/Misra Performance

0

1

2

3

4

5

6

7

8

0 16 32 48 64Message Density

(messages per logical process)

Speedup

Time Warp (64 logicalprocesses)Time Warp (16 logicalprocesses)Deadlock Avoidance(64 logical processes)Deadlock Avoidance(16 logical processes)Deadlock Recovery (64logical processes)Deadlock Recovery (64logical processes)

• eight processors

• closed queueing network, hypercube topology

• high priority jobs preempt service from low priority jobs (1% high priority)

• exponential service time (poor lookahead)

Other Optimistic AlgorithmsPrincipal goal: avoid excessive optimistic execution

A variety of protocols have been proposed, among them:• window-based approaches

– only execute events in a moving window (simulated time, memory)• risk-free execution

– only send messages when they are guaranteed to be correct• add optimism to conservative protocols

– specify “optimistic” values for lookahead• introduce additional rollbacks

– triggered stochastically or by running out of memory• hybrid approaches

– mix conservative and optimistic LPs• scheduling-based

– discriminate against LPs rolling back too much• adaptive protocols

– dynamically adjust protocol during execution as workload changes

Summary of Optimistic Algorithms

Con:• Memory requirements may be large• Implementation is generally more complex and difficult to

debug than conservative mechanisms– System calls, memory allocation ...– Must be able to recover from exceptions

• Use in federated simulations requires adding rollback capability to existing simulations

Pro:• Good performance reported for a variety of applications

(queueing communication networks, combat models, transportation systems)

• Good transparency: offers the best hope for “general purpose” parallel simulation software (more resilient to poor lookahead than conservative methods)

High Level Architecture (HLA)• based on a composable “system of systems” approach

– no single simulation can satisfy all user needs– support interoperability and reuse among DoD simulations

• federations of simulations (federates)– pure software simulations– human-in-the-loop simulations (virtual simulators)– live components (e.g., instrumented weapon systems)

• mandated as the standard reference architecture for all M&S in the U.S. Department of Defense (September 1996)

The HLA consists of• rules that simulations (federates) must follow to achieve proper

interaction during a federation execution• Object Model Template (OMT) defines the format for specifying the set

of common objects used by a federation (federation object model), their attributes, and relationships among them

• Interface Specification (IFSpec) provides interface to the Run-Time Infrastructure (RTI), that ties together federates during model execution

HLA Federation

Simulation(federate)

Runtime Infrastructure(RTI)Services to create and manage the execution of the federation• Federation setup / tear down• Transmitting data among federates• Synchronization (time management)

InterfaceSpecification

InterfaceSpecification

Federation



Interconnecting autonomous simulators

Interface Specification

Category Functionality

Federation ManagementCreate and delete federation executionsjoin and resign federation executionscontrol checkpoint, pause, resume, restart

Declaration ManagementEstablish intent to publish and subscribe to object attributes and interactions

Object Management

Create and delete object instancesControl attribute and interaction publicationCreate and delete object reflections

Ownership Management Transfer ownership of object attributes

Time ManagementCoordinate the advance of logical time and its relationship to real time

Data Distribution Management Supports efficient routing of data

A Typical Federation Execution

initialize federation• Create Federation Execution (Federation Mgt)• Join Federation Execution (Federation Mgt)

declare objects of common interest among federates• Publish Object Class (Declaration Mgt)• Subscribe Object Class Attribute (Declaration Mgt)

exchange information• Update/Reflect Attribute Values (Object Mgt)• Send/Receive Interaction (Object Mgt)• Time Advance Request, Time Advance Grant (Time Mgt)• Request Attribute Ownership Assumption (Ownership Mgt)• Modify Region (Data Distribution Mgt)

terminate execution• Resign Federation Execution (Federation Mgt)• Destroy Federation Execution (Federation Mgt)

HLA Message Ordering Services

typical applications training, T&E analysis

Latency

reproduce before and after relationships?all federates see same ordering of events?

execution repeatable?

Property

low

no

no

no

ReceiveOrder (RO)

higher

yes

yes

yes

Time StampOrder (TSO)

The HLA provides two types of message ordering:

• receive order (unordered): messages passed to federate in an arbitrary order

• time stamp order (TSO): sender assigns a time stamp to message; successive messages passed to each federate have non-decreasing time stamps

• receive order minimizes latency, does not prevent temporal anomalies• TSO prevents temporal anomalies, but has somewhat higher latency

Advancing Logical Time

HLA TM services define a protocol for federates to advance logical time; logical time only advances when that federate explicitly requests an advance

• Time Advance Request: time stepped federates• Next Event Request: event stepped federates• Time Advance Grant: RTI invokes to acknowledge logical time advances

If the logical time of a federate is T, the RTI guarantees no more TSO messages will be passed to the federate with time stamp < T

Federates responsible for pacing logical time advances with wallclock time in real-time executions

federate

RTI

Time Advance Requestor

Next Event RequestTime Advance Grant

HLA Time Management Services

federate• local time and event management• mechanism to pace execution with

wallclock time (if necessary)• federate specific techniques (e.g.,

compensation for message latencies)

wallclock time(synchronized with

other processors)

logical time

FIFOqueue

FIFOqueue

timestamp

orderedqueue

timestamp

orderedqueue

Runtime Infrastructure (RTI)

state updatesand interactions logical time advances

receiveorder

messages

time stamporder

messages

eventordering

timesynchronized

delivery

Synchronizing Message Delivery

Goal: process all events (local and incoming messages) in time stamp order; To support this, RTI will

• Deliver messages in time stamp order (TSO)• Synchronize delivery with simulation time advances

RTI

federate

TSOmessages

TSOmessages

localevents

localevents

logicaltimecurrent

time

nextlocalevent

nextTSO

message

T

Federate: next local event has time stamp T

• If no TSO messages w/ time stamp < T, advance to T, process local event

• If there is a TSO message w/ time stamp T’ ≤ T, advance to T’ and process TSO message

T’

Typical execution sequences

RTI

Wall clocktimeRTI delivers events

Federate

NER: Next Event RequestTAG: Time Advance GrantRAV: Reflect Attribute Values

Federate calls in blackRTI callbacks in red

Next Event Request (NER)

• Federate invokes Next Event Request (T) to request its logical time be advanced to time stamp of next TSO message, or T, which ever is smaller

• If next TSO message has time stamp T’ ≤ T

– RTI delivers next TSO message, and all others with time stamp T’

– RTI issues Time Advance Grant (T’)

• Else

– RTI advances federate’s time to T, invokes Time Advance Grant (T)

NER(T)

RAV (T’)

RAV (T’)

TAG(T’)

NER(T)

TAG(T)

Federate RTI

no TSO events

Logical timeT+LT1. Current time is T, lookahead L2. Request lookahead decrease by ∆L to L’

Lookahead in the HLA• Each federate must declare a non-negative lookahead value

• Any TSO sent by a federate must have time stamp at least the federate’s current time plus its lookahead

• Lookahead can change during the execution (Modify Lookahead)

– increases take effect immediately

– decreased do not take effect until the federate advances its logical time

Logical timeT+LT+ ∆T

L- ∆T∆T

3. Advance ∆T, lookahead, decreases ∆T

Logical timeT+L

L’∆L

T+∆L 4. After advancing ∆L, lookahead is L’

Federate0

network

TSO queue13118

Federate1

Current Time =8Lookahead = 2

Federate2

Current Time =9Lookahead = 8

Minimum Next Event Time and LBTS

• LBTSi: Lower Bound on Time Stamp of TSO messages that could later be placed into the TSO queue for federate i

– TSO messages w/ TS ≤LBTSi eligible for delivery

– RTI ensures logical time of federate i never exceeds LBTSi

LBTS0=10

MNET0=8

• MNETi: Minimum Next Event Time is a lower bound on the time stamp of any message that could later be delivered to federate i.

– Minimum of LBTSi and minimum time stamp of messages in TSO queue

Wallclock time

Vehicle

Observer

Sample execution sequence: NER: Next Event RequestUAV: Update Attribute Values)TAG: Time Advance Grant (callback)

Event Retraction

Previously sent events can be “unsent” via the Retract service– Update Attribute Values and Send Interaction return a “handle” to

the scheduled event– Handle can be used to Retract (unschedule) the event– Can only retract event if its time stamp > current time + lookahead– Retracted event never delivered to destination (unless Flush

Queue used)

1. Vehicle schedules position update at time 100, ready to advance to time 100

1. NER (100)

1. Handle=UAV (100)

2. Receive Interaction (90)2. TAG (90)

2. receives interaction (break down event) invalidating position update at time 100

3. Retract(Handle)

3. Vehicle retracts update scheduled for time 100

Mechanisms to ensure events are processed in time stamp order:

• conservative: block to avoid out of order event processing

• optimistic: detect out-of-order event processing, recover (e.g., Time Warp)

Optimistic Time Management

Primitives for optimistic time management

• Optimistic event processing– Deliver (and process) events without time stamp order delivery guarantee– HLA: Flush Queue Request

• Rollback– Deliver message w/ time stamp T, other computations already performed at times >

T– Must roll back (undo) computations at logical times > T– HLA: (local) rollback mechanism must be implemented within the federate

• Anti-messages & secondary rollbacks– Anti-message: message sent to cancel (undo) a previously sent message– Causes rollback at destination if cancelled message already processed– HLA: Retract service; deliver retract request if cancelled message already delivered

• Global Virtual Time– Lower bound on future rollback to commit I/O operations, reclaim memory– HLA: Query Next Event Time service gives current value of GVT

Optimistic Time Management in the HLA

Flush Queue Request: similar to NER except

(1) deliver all messages in RTI’s local message queues,

(2) need not wait for other federates before issuing a Time Advance Grant

HLA Support for Optimistic Federates• federations may include conservative and/or optimistic federates• federates not aware of local time management mechanism of other

federates (optimistic or conservative)• optimistic events (events that may be later canceled) will not be delivered

to conservative federates that cannot roll back• optimistic events can be delivered to other optimistic federates• individual federates may be sequential or parallel simulations

Summary: HLA Time Management

Functionality:

• allows federates with different time management requirements (and local TM mechanisms) to be combined within a single federation execution

– DIS-style training simulations

– simulations with hard real-time constraints

– event-driven simulations

– time-stepped simulations

– optimistic simulations

HLA Time Management services:

• Event order

– receive order delivery

– time stamp order delivery

• Logical time advance mechanisms

– TAR/TARA: unconditional time advance

– NER/NERA: advance depending on message time stamps


• Introduction• Dead reckoning• Data distribution

Example: Distributed Interactive Simulation (DIS)

• developed in U.S. Department of Defense, initially for training• distributed virtual environments widely used in DoD; growing use in other

areas (entertainment, emergency planning, air traffic control)

“The primary mission of DIS is to define an infrastructure for linking simulations of various types at multiple locations to create realistic, complex, virtual ‘worlds’ for the simulation of highly interactive activities” [DIS Vision, 1994].

ImageGenerator Other Vehicle

State Table

networkinterface

control/display

interface

own vehicledynamics

soundgenerator

visual displaysystem

terraindatabase

network

controlsand panels

A Typical DVE Node Simulator

Reproduced from Miller, Thorpe (1995), “SIMNET: The Advent of Simulator Networking,” Proceedings of the IEEE,83(8): 1114-1123.

Execute every 1/30th of a second:• receive incoming messages & user inputs, update state of remote vehicles• update local display• for each local vehicle

• compute (integrate) new state over current time period• send messages (e.g., broadcast) indicating new state


State Table

networkinterface

control/display

interface

own vehicledynamics

soundgenerator

visual display

terraindatabase


State Table

networkinterface

control/display

interface

own vehicledynamics

soundgenerator

visual display

terraindatabase

Controls/panels

Controls/panels

Typical Sequence

1

1. Detect trigger press

2

2. Audio “fire” sound

3

3. Display muzzel flash

4

4

4. Send fire PDU

5

5. Display muzzel flash

6

6. Compute trajectory, display tracer

7

7. Display shell impact

8

8 8. Send detonation PDU9

9. Display shell impact

10

10. Compute damage

11

11. Send Entity state PDU indicating damage

DIS Design Principles

• Autonomy of simulation nodes– simulations broadcast events of interest to other simulations; need not determine

which others need information– receivers determine if information is relevant to it, and model local effects of new

information– simulations may join or leave exercises in progress

• Transmission of “ground truth” information– each simulation transmits absolute truth about state of its objects– receiver is responsible for appropriately “degrading” information (e.g., due to

environment, sensor characteristics)• Transmission of state change information only

– if behavior “stays the same” (e.g., straight and level flight), state updates drop to a predetermined rate (e.g., every five seconds)

• “Dead Reckoning” algorithms– extrapolate current position of moving objects based on last reported position

• Simulation time constraints– many simulations are human-in-the-loop– humans cannot distinguish temporal difference < 100 milliseconds– places constraints on communication latency of simulation platform

Distributed Simulation Example

• Virtual environment simulation containing two moving vehicles

• One vehicle per federate (simulator)• Each vehicle simulator must track location of other vehicle

and produce local display (as seen from the local vehicle)• Approach 1: Every 1/60th of a second:

– Each vehicle sends a message to other vehicle indicating its current position

– Each vehicle receives message from other vehicle, updates its local display

Limitations

• Position information corresponds to location when the message was sent; doesn’t take into account delays in sending message over the network

• Requires generating many messages if there are many vehicles

Dead Reckoning

• Send position update messages less frequently• local dead reckoning model predicts the position of remote

entities between updates

1000 1050

1000

(1050,1000)

last reported state:position = (1000,1000)traveling east @ 50 feet/sec

predictedposition

one second later

• When are updates sent?• How does the DRM predict vehicle position?

ImageGenerator

Dead reckoningmodel

visual displaysystem

terraindatabase

“infrequent” positionupdate messages

get positionat frame rate

DRMaircraft 1 simulator for

aircraft 2

sample DRM atframe rate

display

HighFidelityModel

aircraft 1

DRMaircraft 1

DRMaircraft 2

over thresholdor timeout

closeenough?timeout?

entity stateupdate PDU

simulator foraircraft 1

Re-synchronizing the DRM

When are position update messages generated?

• Compare DRM position with exact position, and generate an update message if error is too large

• Generate updates at some minimum rate, e.g., 5 seconds (heart beats)

Dead Reckoning Models

• P(t) = precise position of entity at time t

• Position update messages: P(t1), P(t2), P(t3) …

• v(ti), a(ti) = ith velocity, acceleration update

• DRM: estimate D(t), position at time t

– ti = time of last update preceding t

– ∆t = ti - t

• Zeroth order DRM:

– D(t) = P(ti)

• First order DRM:

– D(t) = P(ti) + v(ti)*∆t

• Second order DRM:

– D(t) = P(ti) + v(ti)*∆t + 0.5*a(ti)*(∆t)2

t2

generate stateupdate message

true position

DRM estimate oftrue position

state update

display update

message E

t1

DRM Example

Potential problems:• Discontinuity may occur when position update arrives;

may produce “jumps” in display• Does not take into account message latency

B CA

DRM estimates position

D

receivemessage

just beforescreen update

Time Compensation

Taking into account message latency• Add time stamp to message when update is generated

(sender time stamp)• Dead reckon based on message time stamp

update with timecompensation

D E

t2

t1

true position


state update

display update

message

A B C

Smoothing

Reduce discontinuities after updates occur• “phase in” position updates• After update arrives

– Use DRM to project next k positions– Interpolate position of next update

interpolatedposition

D

extrapolated positionused for smoothing

E

t2

t1

true position


state update

display update

message

Accuracy is reduced to create a more natural display

A B C

Data Distribution ManagementA federate sending a message (e.g., updating its current location) cannot

be expected to explicitly indicate which other federates should receive the message

Basic Problem: which federates receive messages?• broadcast update to all federates (SimNet, early versions of DIS)

– does not scale to large numbers of federates• grid sectors

– OK for distribution based on spacial proximity• routing spaces (STOW-RTI, HLA)

– generalization of grid sector idea

Content-based addressing: in general, federates must specify:• Name space: vocabulary used to specify what data is of interest and

to describe the data contained in each message (HLA: routing space)• Interest expressions: indicate what information a federate wishes to

receive (subset of name space; HLA: subscription region)• Data description expression: characterizes data contained within

each message (subset of name space; HLA: publication region)

HLA Routing Spaces

HLA Data Distribution Management• N dimensional normalized routing space• interest expressions are regions in routing space (S1=[0.1,0.5],

[0.2,0.5])• each update associated with an update region in routing space• a federate receives a message if its subscription region overlaps with

the update region

• Federate 1 (sensor): subscribe to S1• Federate 2 (sensor): subscribe to S2• Federate 3 (target): update region U

update messages by target are sent to federate 1, but not to federate 2

S2

S1U

1.0

1.00.0

0.0 0.5

0.5

S1

U

S2

1.0

1.00.0

0.0 0.5

0.5

Implementation: Grid Based

• subscription region: Join each group overlapping subscription region• attribute update: send Update to each group overlapping update region• need additional filtering to avoid unwanted messages, duplicates

U publishes to 12, 13S1 subscribes to 6,7,8,11,12,13S2 subscribes to 11,12,16,17

1 2 3 4 5

6 7 8 9 10

11 12 13 14 15

16 17 18 19 20

21 22 23 24 25

multicast group,id=15

partition routing space into grid cells, map each cell to a multicast group

Changing a Subscription Region

1 2 3 4 5 6 7 8

9 10 11 12 13 14 15 16

17 18 19 20 21 22 23 24

25 26 27 28 29 30 31 32

33 34 35 36 37 38 39 40

existing subscription region

Change subscription region:• issue Leave operations for (cells in old region - cells in new region)• issue Join operations for (cells in new region - cells in old region)

Observation: each processor can autonomously join/leave groups whenever its subscription region changes w/o communication.

new region

Leave group

Join group

no operations issued 2 3 4

10 11 12 13

18 19 20 21

27 28 29

Another Approach: Region-Based Implementation

• Associate a multicast group with each update region• Membership: all subscription regions overlapping with update region• Changing subscription (update) region

– Must match new subscription (update) region against all existing update (subscription) regions to determine new membership

• Multicast group associated with U• Membership of U: S1• Modify U to U’:

– determine subscription regions overlapping with U’

– Modify membership accordingly

S2

S1U

1.0

1.00.0

0.0 0.5

0.5

• No duplicate or extra messages• Change subscription: typically requires interprocessor communication• Can use grids (in addition to regions) to reduce matching cost

Distributed Virtual Environments: Summary• Perhaps the most dominant application of distributed

simulation technology to date– Human in the loop: training, interactive, multi-player video games– Hardware in the loop

• Managing interprocessor communication is the key– Dead reckoning techniques– Data distribution management

• Real-time execution essential• Many other issues

– Terrain databases, consistent, dynamic terrain– Real-time modeling and display of physical phenomena– Synchronization of hardware clocks– Human factors

Future ResearchDirections

The good news...

Parallel and distributed simulation technology is seeing widespread use in real-world applications and systems

• HLA, standardization (OMG, IEEE 1516)• Other defense systems• Interactive games (no time management yet, though!)

Interoperability and Reuse

“critical capability that underpin the IMTR vision [is]... total, seamless model interoperability: Future models and simulations will be transparently compatible, able to plug-and-play …”

- Integrated Manufacturing Technology Roadmap

“Sub-objective 1-1: Establish a common high level simulation architecture to facilitate interoperability [and] reuse of M&S components.”

- U.S. DoD Modeling and Simulation Master Plan

Reuse the principal reason distributed simulation technology is seeing widespread use (scalability still important!)

• Enterprise Management• Circuit design, embedded systems• Transportation• Telecommunications

Research Challenges• Shallow (communication) vs. deep (semantics)

interoperability: How should I develop simulations today, knowing they may be used tomorrow for entirely different purposes?

• Multi-resolution modelling• Smart, adaptable interfaces?• Time management: A lot of work completed already, but

still not a solved problem!– Conservative: zero lookahead simulations will continue to be the

common case– Optimistic: automated optimistic-ization of existing simulations

(e.g., state saving)– Challenge basic assumptions used in the past: New problems &

solutions, not new solutions to the old problems

• Relationship of interoperability and reuse in simulation to other domains (e.g., e-commerce)?

Real-Time Systems

Interactive distributed simulations for immersive virtual environments represents fertile new ground for distributed simulation research

The distinction between the virtual and real world is disappearing

Research Challenges• Real-time time management

– Time managed, real-time distributed simulations?– Predictable performance– Performance tunable (e.g., synchronization overheads)

– Mixed time management (e.g., federations with both analytic and

real-time training simulations) • Distributed simulation over WANs, e.g., the Internet

– Server vs. distributed architectures?– PDES over unreliable transport and nodes

• Distributed simulations that work!– High quality, robust systems

• Data Distribution Management– Implementation architectures– Managing large numbers of multicast groups– Join/leave times– Integration and exploitation of network QoS

• Battle management• Enterprise control• Transportation Systems

Simulation-Based Decision Support

Real-Time AdvisorReal WorldSystem

livedatafeeds

analysts

forecasting tool(fast simulation)

Informationsystem

Automated and/or interactive

analysis tools

decision makers

Research Challenges• Ultra fast model execution

– Much, much faster-than-real-time execution– Instantaneous model execution: Spreadsheet-like performance

• Ultra fast execution of multiple runs• Automated simulation analyses: smart, self-managed

devices and systems?• Ubiquitous simulation?

Closing Remarks

After years of being largely an academic endeavor, parallel and distributed simulation technology is beginning to thrive.

Many major challenges remain to be addressed before the technology can achieve its fullest potential.

TheEnd

tutorial: parallel & distributed simulation systems: from chandy/misra to the high level...

Documents

computer simulation

single simulation program

real time execution

systems fast model execution

databases slide

time management

high level architecture

time critical decision