distributed computing with daffie glenn bresnahan boston university scientific computing and...

Distributed Computing with Distributed Computing with DAFFIEDAFFIE

Glenn Bresnahan

Boston University

Scientific Computing and Visualization

(http://scv.bu.edu/DAFFIE/Earlab-Oct02)

Scientific Computing and Visualization 2

What is DAFFIE?What is DAFFIE?

Distributed Applications Framework For Immersive Environments

System for easily creating distributed and shared virtual environments

Authored by Glenn Bresnahan, Erik Brisson, Robert Putnam

Scientific Computing and Visualization 3

DAFFIE Messaging SystemDAFFIE Messaging System

Application protocol

Client servicesClient application library (API)

Message router (“event server”)

Scientific Computing and Visualization 4

DAFFIE Message routingDAFFIE Message routing

client

event event serverserver

client

clientproxyproxy

clientproxyproxy

client

client

client

client

client

Scientific Computing and Visualization 5

Application ArchitectureApplication Architecture

App threadApp thread App threadApp thread …

Services(timing, streaming, file sharing)

Message Transport

Event Messaging

Scientific Computing and Visualization 6

API - ControlAPI - Control

id = event_join(server, &nc); rc = event_leave();

id = event_id(); nc = event_nclient(); millisec = event_clock(); event_sleep(millisec);

event_register(class, species, instance);

Scientific Computing and Visualization 7

Event headers informationEvent headers information

EventType(event);EventSize(event);EventFrom(event);EventTo(event);EventSeq(event);

Scientific Computing and Visualization 8

System Events – Join & LeaveSystem Events – Join & Leave

typedef struct { EVENT_HEAD head; int id; int nmember;} EVENT_JOIN;

typedef struct { EVENT_HEAD head; int id; short detail; short exitcode;} EVENT_LEAVE

Scientific Computing and Visualization 9

System Events – IdentificationSystem Events – Identification

typedef struct { EVENT_HEAD head; int tag;} EVENT_IDENTITY;

typedef struct { EVENT_HEAD head;

char host[MAXIDSTR];char class[MAXIDSTR];char species[MAXIDSTR];char instance[MAXIDSTR];int tag;

} EVENT_IDENTIFICATION;

Scientific Computing and Visualization 10

API – SendAPI – Send

rc = event_send(to, type, &event, size);– “to” is a client id or one of the following:

• EVENT_BCAST_ALL• EVENT_BCAST_NOTME• EVENT_BCAST_SERVER• EVENT_BCAST_LOCAL

– Macros• SEND(to, type, event);• BCAST(type, event);

event_drain();

Scientific Computing and Visualization 11

API - ReceiveAPI - Receive

rc = event_receive(&event);

event_receive_type(n, type1, …); event_select(on_off, et_low, et_high);

event_wait(); event_wait_timeout(millisec);

Scientific Computing and Visualization 12

API – Receive queueAPI – Receive queue

event_scan_type(n, type1, …); rc = event_scan(&event); n = event_queued();

event_flush(flag); event_flush_sync(); event_receive_enable(on_off);

Scientific Computing and Visualization 13

API – Active messagesAPI – Active messages

event_callback(type, func);

event_add_callback(type, func);event_delete_calllback(type, func);event_protect_type(on_off, et_low,

et_high);

Scientific Computing and Visualization 14

Client ServicesClient Services

Streaming– Data chunking – Bandwidth management– Dynamic tuning

File sharing– File servers– Conditional upload/download

Scientific Computing and Visualization 15

Job Initiation & MonitoringJob Initiation & Monitoring

Rely on external authentication and job initiation (e.g. ssh, Globus)– Launch scripts (ad hoc)

• cgl-video; visible.bu.edu:vgrab -compress -process; -:vswirl -steps 9999999 -np 4; -:vbroadcast; .:vdirect -object poolanim_grp ;

– Agent initiation daemon– Logging daemon

Scientific Computing and Visualization 16

Message TransportsMessage Transports

TCP/IP– Single channel or striped

Shared memoryMyrinetIP Multicast

Scientific Computing and Visualization 17

Defining Upper Layer ProtocolDefining Upper Layer Protocol

Components– Event type ID (#define in a .h)– Data layout (compile-time) (C struct in a .h) – Data layout (run-time) (event_define)• Needed for canonical byte ordering

Types statically (compile time) defined

Protocol compiler

Scientific Computing and Visualization 18

Message RoutingMessage Routing

Message router (“event server”)– Multiple delivery service classes

• Globally sequenced, reliable delivery

– Client management– Message subscription– Multiprocessor, high bandwidth

Hierarchical servers Proxy routers

– Fan-in/fan-out Tunneling

– Direct routing for P2P

Scientific Computing and Visualization 19

Messaging APIMessaging API

Multithreaded, non-blocking Dynamic join and leave Point-to-point and broadcast messages Polled or callback (active messages) Subscription based receive Low level services– Registration, timing, synchronization, query

Scientific Computing and Visualization 20

Service DiscoveryService Discovery

Client registration/identification– Class (Viewer, Audio, Video, Agent)– Species (video: source, broadcaster, sink,

processor)– Instance (video: stream-name)– ID– Hostname

Client location services Service request/acknowledgement

Scientific Computing and Visualization 21

Client MonitoringClient Monitoring

Dynamic clients– JOIN, EXIT, IDENTIFICATION events

Event server monitoring– Heart beat– Latency monitor– Congestion control– Statistic generation

Client monitoring– Process level monitoring– Congestion management– Remote process termination

Scientific Computing and Visualization 22

Leaky Example: HeadersLeaky Example: Headers

typedef struct {EVENT_HEAD head;char filename[200];

} EVENT_GETPARAM;

typedef struct {EVENT_HEAD head;int logger, frame_size, sample_rate);

} EVENT_START;

typedef struct {EVENT_HEAD head;char msg[1000];

} EVENT_LOGINFO;

Scientific Computing and Visualization 23

Leaky Example: HeadersLeaky Example: Headers

typedef struct {

EVENT_HEAD head;

int client;

int size;

float pressure[1];

} EVENT_ADVANCE;

Scientific Computing and Visualization 24

Leaky Example: HeadersLeaky Example: Headers

#define ET_START 1001#define ET_GETPARAM 1002#define ET_LOGINFO 1003#define ET_ADVANCE 1004#define ET_STOP 1005

event_define(ET_START, “start”, 1, 3, EVENT_DATA_INT);event_define(ET_GETPARAM, “getparam”, 1, 200, EVENT_DATA_CHAR);event_define(ET_ADVANCE, “advance”, 3,

1, EVENT_DATA_INT,1, EVENT_DATA_INT,0, EVENT_DATA_FLOAT);

Scientific Computing and Visualization 25

Leaky Example: InitializationLeaky Example: Initialization

id = event_join(“earlab.bu.edu”, &nc);printf(“Joined as client id %d of %d\n”, id, nc);

event_register(“integrator”, “leaky”, “1.0”);

event_receive_type(ET_GETPARAM, ET_START, ET_ADVANCE, ET_STOP);

event_callback(ET_START, do_start);event_callback(ET_GETPARAM, do_getparam);…

Scientific Computing and Visualization 26

Leaky Example: File ServiceLeaky Example: File Service

GotParamFile = 0;

efs_init();

efs_chdir_receive(“/tmp/earlab”);

Scientific Computing and Visualization 27

Leaky Example: GetParametersLeaky Example: GetParameters

do_getparams(EVENT_GETPARM *egp) {

efs_download_cond(egp->filename,

egp->filename, xfer_done, 0);

return(1);

}

xfer_done(int reqid, int fsid, inr rc, char *file, int tag) {

char path[1000];

strcpy(path, “/tmp/earlab/”); strcat(path, file);

GotParamFile = 1;

Leaky_GetParameters(path);

);

Scientific Computing and Visualization 28

Leaky Example: StartLeaky Example: Start

do_start(EVENT_START *est) {

Logger = est->logger;

while (!GotParamFile) event_sleep(100);

Leaky_Start(egp->frame_size, egp->sample_rate);

return(1);

}

Scientific Computing and Visualization 29

Leaky Example: LogInfoLeaky Example: LogInfo

void LogInfo(char *msg) {

EVENT_LOGINFO einfo;

strcpy(einfo->msg, msg);

event_send(Logger, ET_LOGINFO, &einfo, sizeof(einfo));

}

Scientific Computing and Visualization 30

Leaky Example: AdvanceLeaky Example: Advance

do_advance(EVENT_ADVANCE *ein) {int bytes;EVENT_ADVANCE *eout;

bytes = sizeof(EVENT_ADVANCE) + (ein->size-1)*sizeof(float);

eout = (EVENT_ADVANCE *) malloc(bytes);

Leaky_Advance(&ein->pressure, &eout->pressure);eout->client = event_clientid();eout->size = ein->size;event_send(ein->client, ET_ADVANCE, eout, bytes);

free(ein);return(1);

}

Scientific Computing and Visualization 31

DAFFIE Message routingDAFFIE Message routing

client

event event serverserver

client

clientproxyproxy

clientproxyproxy

client

client

client

client

client