Charm++ Arrays, Parameter Marshalling,

Load Balancingand what they have in

common-- PUP9/1/2001

Orion Sky Lawlor

PPL Developer

Bachelor's in Math and CS from

Univ. of Alaska at Fairbanks

MS in CS from UIUC

Arrays: the Basic Idea" So there's some parallel object " How do you talk to it?" MPI approach: send a message to the

processor number " Make migration (almost) impossible" Makes load balancing (almost) impossible

" Array approach: send message to array index (1D, 2D, etc.)

Array Syntax (.ci)

module demo {array [1D] A {

entry A(void);entry void foo(void);

}}

Array Syntax (.h)class A:public ArrayElement1D {public: A(void); void foo(void);

//For migration: A(CkMigrateMessage *) { } void pup(PUP::er &p);};

Array Creation... in main chare ...int n=17;CProxy_A ap=CProxy_A:: ckNew(n);//Create array

//Elements numbered {0..n-1}//Always dense//Always 1D

Sparse Array Creation... in main chare ...int n=17;CProxy_A ap=CProxy_A:: ckNew();//Create array

for (int i=0;i<n;i++) ap[7+2*i].insert();ap.doneInserting();

//Don't forget the doneInserting!

Array Messaging//Just index the array proxyap[7].foo();

//Or can make element proxy:CProxyElement_A aep=ap[9];aep.foo();

//For higher dimensions:ap2(x,y).foo();ap3(x,y,z).foo();

Array Broadcast//Leave off the index:ap.foo();

" Invokes foo on every existing array element

" Works even with migrations" Bad syntax: easy to broadcast by

mistake, causing disaster!

Array Reductionint myData[2]={...};contribute(2*sizeof(int), (void *)&myData, CkReduction::sum_int);

" Called from every array element" Collects result to PE 0 & calls fn:

void myCli(void *userData, int redLen,void *redData);ap.setReductionClient(myCli,userData);

Arrays

PUP

PUP-- Wazzup?" Before PUP, had to write 3 functions:

" size-- compute message size, in bytes" pack-- write object into message" unpack-- read object out of message

" PUP combines all three into one" Basic contract: here are my fields

(types, sizes, and a pointer)" C++ version uses operator overloading

for very spartan syntax-- just parens() and bar|

PUP Single Fieldsclass simple { int x; float y;public: simple() { } ... other methods ... void pup(PUP::er &p) { p|x; p|y; }};PUPmarshall(simple);

PUP Arraysclass withArray { public: int n; double *d;//Holds n doubles simple() {n=0;d=NULL;} ~simple() { delete[] d;} ... other methods ... void pup(PUP::er &p) { p|n; if (p.isUnpacking()) d=new double[n]; p(d,n); }};PUPmarshall(withArray);

PUP Subobjectclass sub { public: simple *s;//Holds a subobject sub() {s=NULL;} ~sub() { delete s;} ... other methods ... void pup(PUP::er &p) { if (p.isUnpacking()) s=new simple; s->pup(p); }};PUPmarshall(sub);

PUP::able & auto-allocateclass A:public PUP::able {public: ... other methods ... A(CkMigrateMessage *) { } virtual void pup(PUP::er &p); PUPable_decl(A);};PUPable_def(A);

(...in an initcall routine...) PUPable_reg(A);(...in some object's pup...) p|a;

PUP::able & treesclass A:public PUP::able { A *left,*right; ... as before ...};void A::pup(PUP::er &p) { p|left; p|right;}

" Packs, allocates, and unpacks left and right children automatically

" Works properly with NULL pointers" Doesn't work for object graphs with

cycles! (Never terminates)

PUP::able & inheritanceclass B:public A { ... the usual ... PUPable_decl(B);};PUPable_def(B);(...in an initcall routine...) PUPable_reg(B);

" Now an A pointer that actually points to a B object will be restored properly-- B::pup will be called.

" Combines well with virtual functions.

PUP

Param. Marshall

Parameter Marshallingentry void foo(void);entry void bar(int a,int b);entry void baz(float f[7]);entry void far(int n,double d[n]);entry void boo(int n,int m, unsigned char area[n*m]);entry void oof(char s[strlen(s)+1]);entry void fub(something_t s);

" Arrays passed as pointers in C++" Array length evaluated on call side" Can declare everything “const", “&"

Param. Marshalling & PUPentry void foo(T a, V b);

" Gets translated into://Send side:CProxy_A::foo(T &a, V &b) { ...make PUP::er &p... p|a; p|b; ...use resulting p...}//Recv. side:...in call-foo function... T a; V b; p|a; p|b; obj->foo(a,b);

" Because there’s a default bar operator, T and V can be C structs; which are copied as bytes.

" Because you can overload the bar operator using PUP_Marshall, if T and V have pup routines they’ll be called properly!

" Makes marshalling flexible.

Param. Marshall

Load Balancing

Load Balancing" All you need is a working pup" Load balancer will migrate constantly" Create the load balancer with:

//In the .ci file: extern module RefineLB;//In the .C file:#include "RefineLB.h"//In the main chare: CreateRefineLB();

//Also: CommLB, GreedyRefLB, HeapCentLB, MetisLB, NeighborLB

AtSync Load Balancing" When you need more control over when

balancing happens." Create balancer as usual, then:

//In the element constructor: usesAtSync=CmiTrue;

//When ready to balance call: AtSync();

//When computation resumes, void ResumeFromSync(void);//gets called on each element.

Conclusion" Arrays support messaging,

broadcasts, reductions, and migration

" Migration is for load balance" Migration uses PUP" Parameter marshalling uses PUP" Can get lots of functionality (almost)

for free with PUP