Implementation:Charm++

Orion Sky Lawlorolawlor@acm.org

Background & Roadmap

Your code sits on Charm++: Interface Translator

.ci file, CProxy_*, parameter marshalling

Arrays (Orion++) Reductions, broadcasts, migrations, pup

Groups and Chares (``Milind++) Basic communication calls atop Converse

Basics (++) Registration, readonlies, messages

Where is it? Charm++ core source code in

charm/src/Common/ck-core/ Gets soft-linked over and made in

charm/net-linux/tmp/ Headers in charm/net-linux/include/ Code gets linked into the library

charm/net-linux/lib/libck.a This all gets used when you pass

"-language charm++" to charmc

Basics: We need to be able to call some

C++ method bar on user object foo

But the core`s never heard of foo Solution: register a function

pointer to a "call-method" function:

void _call_foo_bar(void *msg,foo *obj) {obj->bar(msg);

}

...at startup...int barEpIdx=

CkRegisterEp(&_call_foo_bar,..);

Registration & Startup ck-core/register.h .C (300 lines) At startup, each module registers

its methods, which end up in a big entry method table

An entry point's index in this table is its int "epIdx", used everywhere

Translator writes this registration code (the call-method function, "__register" function) in .def file

Groups & Chares (Milind++)

charm.h, ck.C (1000 lines) Lowest layer of Charm++ CkCreateChare, CkCreateGroup CkSendMsg, CkSendMsgBranch Quite simple, thin wrapper on

Converse Very stable-rarely changes Callable from C (!)

CkSendMsg(int epIdx, void *msg,CkChareID

*to) envelope *env = UsrToEnv(msg);

Finds Charm++ msg fields (see envelope.h)

env->setMsgtype(ForChareMsg);

env->setObjPtr(to->objPtr);

env->setEpIdx(epIdx);

env->setSrcPe(CkMyPe());

CmiSetHandler(env, _charmHandlerIdx);

CldEnqueue(to->onPE, env, _infoIdx);

CkSendMsg: Clutter Various unrelated calls clutter up the

Charm++ core (FIXME?) Keep track of message sends; give

"Message being re-sent..." diagnostic _CHECK_USED(env);

_SET_USED(env, 1);

Trace message sends for projections _TRACE_CREATION_1(env);

Notify quiescence detection of send CpvAccess(_qd)->create();

Arrays (Orion++) Many interesting features:

Arbitrary indexed collection of Chares Connects to load balancer Supports migration, broadcasts, reductions Not even clear some operations even possible

But much more complicated Highly fragmented, er, factored code

Array Manager, Location Manager, Reduction Manager, Broadcaster, Reducer

Arrays Diagram

CkReductionMgr--reductions

CkLocMgr--migration

CkArray-- coordinator

ArrayElementArrayElementArrayElementArrayElement-

-user's work

CkReductionMgr ckreduction.h .C .ci (1300 lines) Accept contributions from local

members of a migrating set of reduction "contributors"

Implemented as Group Actually "IrrGroup", since normal groups

inherit from CkReductionMgr.

Exactly one per array; but zero dependence on arrays

Arrays are just a client of CkReductionMgr

Location-- CkLocMgr ck-core/cklocation.h .C .ci (2000) Keep track of location of migrating

elements Handles messaging and migration Each array has a location manager

But a location manager may have several "bound" arrays, which will migrate together

Implemented as a group

Array Manager-- CkArray

ck-core/ckarray.h .C .ci (1300 lines) Keeps track of a set of local array

elements Implemented as a group Glue that connects pieces of array

system together: Inherits from CkReductionMgr Implements CkArrMgr (for location mgr) Interfaces with proxies Controls ArrayReducer, ArrayBroadcaster

Pup util/pup.h .C (1000 lines) Used to migrate, parameter marshall Heavy C++: overload the | operator "Easy to use, tough to understand" Bottom line: (user view)

Say "p|x;" for all your x's.

Bottom line: (system view) We get a "void *" for each of the user`s x.

Translator xlat-i/xi-symbol.h .C (3000 lines) Turn .ci files into .decl/.def files

Standalone program "charmxi" Tokenize .ci with lexx: xi-scan.l Parse .ci with yacc: xi-grammar.y Turn parsed structures into .decl/.def

Almost completely routines named "gen*"-- code-generating routines

E.g., "Entry::genDecls"-- generate code in .decl file for an entry method

Not Covered Startup & Shutdown (init.C) Quiescence detection (qd.C) Dynamic load balancer (ck-ldb/) Callbacks, futures (for sync

methods), debug, TeMPO Libraries (charm/src/Common/lib/)

TCharm, FEM, AMPI, MBlock, AMR, comm, ...

Details! Read manuals, headers, our minds...

Conclusion Interface Translator

.ci file, CProxy_*, parameter marshalling

Arrays (Orion++) Reductions, broadcasts, migrations, pup

Groups and Chares (``Milind++) Basic communication calls atop Converse

Basics (++) Registration, readonlies, messages