Always room for improvement? Inter-agency guidelines for M&Eby Tom de Veer

When water supplies cannot be guaranteed,refugees are dying, and more are fleeing into thecamps every day, how can the development ofguidelines for monitoring and evaluatingagencies' sanitation programmes be a crucialexercise in saving time and lives?

knowledge and experience built upshould have positive repercussions forM&E in future programmes.

Lists of indicators for each part ofthe programme, and experiments wereset up for measurement. The goal wasnot only to look at which indicatorsshould be measured, but also to assesswhat was practically possible, giventhe limited time and resources avail·able. It became obvious that severalindicators needed adapting - whileothers were not significant. For thesanitation programme, the final moni-toring system incorporated the follow-ing indicators, which were reported onevery week:• distances travelled by cars and trucks

and their fuel consumption;• financial, materials. and equipment

administration (including cross-checks to uncover theft);

resources were available as contin-gency measures, and how long wouldit take to put these into action?These concerns only related to dailyprogramme-management - additionalinformation was necessary for account-ability to donors, for adapting organi-zational procedures and structuresto field conditions, and forimproving futureprogrammes.

DevelopingindicatorsIn persuadingOxfam to autho-rize the develop-ment of animproved moni-toring and evalua-tion (M&E) sys-tem - to beexperimentedwith in the sanita-tion programmes,it was argued thatan improved mon-itoring and evalu-ation system Bath-time for children in Comawould benefit the .programmes and, once installed, was • number of female and male staff;likely to save time, while the • number of camp inhabitants;

• mortality rates;• morbidity rates for the worst

sanitation-related diseases;• number of latrines produced

and in use;• hygiene-behaviour observa-

tions by hygienists (irregular,unstructured, and highly sub-jective, but still useful. Forexample. most hygienistsbelieved that, despite theirefforts, hardly anyonewashed her or his hands aftergoing to the toilet);

• general camp cleanliness;and

.the amount of time thatprogramme staff spent on

Lack of information on hygienebehaviour - an example

From the information a ailable it \ a,impo ibl to d termine "beth r h. 'giene educa-tion within the camp as cit etl. ere themes a the right one '? Were the)' bing tran"·mi t d effectiveI ? Th onl. wa. to fmd out wt learn mor about peopl' b ha'\'iour, buthygiene indicator w re not mea ured in thecamp and tb re w no ba cline informationabout Ih ir original beha iour. Perh p th

tafT's efforts had little effect, which would meana waste of time, effort, and money. a well as amissed opportunity to enable people to impro etheir h giene beha iour in tbe long-term.

Some questions remained unan-swered, however:• Mortality rates were low, but

how could one determine theextent to which sanitationprogrammes contributed?

• Could the results be achievedmore efficiently?

• Was the work being doneback in programmes' countryoffices and headquarters effi-cient?There were other information

needs in the camps: were pro-gramme staff properly protectedwhen spraying against vectors?Was the distribution of materi-als efficient and theft-proof?What material and human

Questions

IMMEDIATELY FOLLOWING THEgenocide of 1994, fleeing Hutus -mainly women, children, and theelderly - too weak to cross Rwanda'sborder, settled in refugee camps in theFrench military (later UN)-protectedzone. One of the camps, Kibeho, had60 000 inhabitants, rising to 100 000after three months; Ndago shelteredapproximately 20 000 people, whichrose to a figure of 45 000.

By early 1995, when the authorbegan working in the camps forOxfam, the sanitation programmes hadmonitoring systems in place whichincluded the measurement of indicatorssuch as the number of latrines pro-duced and used; camp population size;and the number of deaths. In addition,programme managers and hygienistschecked - but in an unstructured way- on levels of cleanliness and opendefecation. The health and sanitationsituation appeared to be acceptable:mortality rates were low and decreas-ing, while latrine coverage remainedstable at around 1:50 people.

WATERLINES VOL. 15 NO.1 JULY 1996 27

Figure 2. Although the staff/refugee ratiopopulation decreased, the quality of serviceswould appear to have remained high.

_Femalestaff

---6-Total no. qfstaff

-+- Male staff

~No.ofpeople/staff

Crisis managementAn emergency situation arose in April1995, when Rwandan soldiers entered

almost automatically, ensuring a rapidreaction.• Several important indicators tooktime and energy to measure, for exam-ple, counting the camp population,while other indicators, such as fuel use,were easy to measure.• It was concluded that programmemanagers needed a management tool toguide them in setting up an adequatebut easy M&E system for counteringproblems, saving time, and increasingefficiency.

3 6 7 9 11 13 16

700

600

600400

300200

100

o1

the camps and herded people into verysmall spaces with few water and sani-tation facilities. Local staff were pre-vented from working, so three Oxfam

engineers had to organize waterprovision for 250 000 people.For the six-day occupation, therewas only time to provide water;sanitation was not attended to.Any longer, and there wouldhave been mass epidemics.

This experience did ensure,however, that the indicatorswhich could and should bemeasured in crisis situationswere decided on: the amount ofwater delivered, its turbidity, andresidual chlorine levels. If theoccupation had continued, how-ever, a major effort would havebeen needed, both to step upactivities to relieve the people'sneeds, and to measure moreindicators about water use -although the water deliveredwas relatively clean, it becamecontaminated by dirty contain-ers, by people walking in

tapstand. When a the remains of storage tanks,no choice but to and through unhygienic

handling. Water quality and

Findings• A lot of the informationneeded to manage the pro-gramme on a daily basisdoes not merit reporting,for example the amount offuel used for pumpingwater.• The most time-consum-ing part of the programmemanager's M&E responsi-bilities is processing thedata for reporting; theactual measurement waslargely done by pro-gramme staff. Initial M&Etraining also requires asubstantial input from themanager, and monitorsmust be monitored.• Once the system was reasonablywell developed and staff had someexperience, much of the informationreached the programme manager

o1 2 3 4 5 6 7 8 9 101112 13 14

Week numbers in 1995

50

250

100

.-

A soldier looks on as Rwandan refugees queue for thenew emergency arises, sanitation programme staff haveconcentrate on maintaining water supplies.

§J200150

c:i

lQI:2~

Figure 1shows the steady drop in deaths and sickness in the camps.

providing hygiene education, andbeing trained to improve their skills.

Figure ] shows clearly that the inci-dence of death and disease were low,and decreasing. But there were no stan-dards to judge against. Standards formortality were obtained from anOxfam publication.' Due to lack oftime and knowledge, indicators on vec-tor control (how does one measure theefficiency and effectiveness of spray-ing, what are the quality standards?);solid-waste; hut, road, and waterpointdrainage; hygiene behaviour; safety forspraying; contingency issues; and thehygiene of cattle-slaughtering couldnot be included.

Figure 2 shows that, as more andmore people moved into the camp, thestaff:refugee ratio declined. Becausethe programme activities appeared tokeep step, the indications are thatthe programme was becoming moreefficient.

28 WATERLINES VOL. ]5 NO.1 JULY 1996

Evaluation will improve iffieldworkers are suitably skilled, and trained appropriately.

Monitoring and evaluation: the agencyrecommendationso Develop simple, clear, and comprehensive monitoring, evaluation, andreporting guidelines specifically for water and sanitation in camps, basedlargely on existing guidelines, and staff experience and know-how. If possi-ble, the implementing, donor, and support agencies will collaborate on collat-ing the information, which should be supplemented by additional investiga-tions where needed, and the results field-tested. One organization could act asco-ordinator.o Develop training courses on monitoring, evaluation, and reporting for staffof implementing and donor agencies.o Employ staff with water and sanitation expertise, who have the necessary'social' skills in participation, hygiene education, and monitoring, evaluationand reporting. For implementing agencies these criteria should apply to field-workers, while donor agencies may require appropriately skilled staff both inthe field and in their offices.

quantity parameters must be monitoredin an emergency.

Self-assessmentBetween November 1995 and May1996, the author worked with 12 agen-cies - Oxfam, MSF-Holland, MSF-France, ORA, ECHO, aDA,RedR, ICRC, IFRC, Unicef,UNHCR, and AICF - todetermine what further devel-opments were needed: whatM&E and sanitation systemsdid they use? What were theirideas and beliefs? Whatdevelopments did they want?The assessment comprised areview of relevant literature, aquestionnaire completed byeach participating organiza-tion, an analysis of guidelinesand evaluation reports fromseveral of the agencies, plusindividual feedback. The mainfindings were:• In camps, many factorshamper the proper executionof M&E, in particular the lackof time, and insecurity. Butthere is considerable room forimprovement, including rais-ing staff awareness, motiva-tion, and skills, for example,through training staff in oremploying those with skills in the'social' side of W&S; improving man-agement tools for M&E; introducingstructural methodologies forprogramme planning, and improvingorganizational structures to optimizethe use of available information.

• Proper planning is imperative -some organizations have useful experi-ence of structural methodologies suchas the Logical Framework Analysis.2• Almost all organizations have and/oruse some M&E guidelines, which also

often include formats for reporting.They all contain valuable information,although many are fairly general.Many also have a high abstractionlevel. Several guidelines included listsof indicators covering parts of theinformation needed for daily manage-ment and programme evaluation, and

were used as the basis of an initialoutline of specific generic guidelineson the M&E of W&S programmes incamps, included in the final report.• Donor agencies' guidelines forfunding applications contain M&E andreporting conditions. These are often

quite general, however, and could befurther specified.• M&E can also be looked at in alarger context: quality control. Theintroduction of formal quality-controlstructures for all processes executed

by an organization is quickly gmnlllgimportance in business and, in manycases, certification such as ISO 9000is demanded by clients: a trend soon toarrive in the 'aid business'?• People still have little experience ofsubjects related to the M&E of W&Sprogramme in camps, for example, the

M&E of 'efficiency' and 'sustainabil-ity' and hygiene behaviour.• Many of the reports containedimportant programme information. Itwas clear that different kinds of reportreviewed different programme aspects,and in varying detail.• The use of information needsattention. This could be done byimproving or implementing certainorganizational structures/procedures,such as measures to ensure optimumreport use, and introducing mecha-nisms for continuous assessment ofthe appropriateness of existingorganizational structures on the basisof available information. These areimportant at all levels.

ReferencesI. Mears. c.. and Chnwdhury. S .. HI'altll Carl'

for Rljl/gees alld Dis{llacI'd Peo{llt'. OXI~II11Practical Health Guide No.9. Oxfam. Oxford,1994.2. A tool for programme dcvelopmcnt that, iFproperly used, can form a basis for thc cvaluationprocess.

TOIll de Veer is a fn'elallce COII.l'lIltwlt j(Jrwater alld sallitatioll ill ellll'rgl'lIcil's.rehabilitatioll alld dl'l'e/Oflllll'lIt progrlllllllll'sill the South. The 12-agellcy aSSl'SSlIll'lItwas carried Olll with fUlldillg fromthe Nether/allds Millistry of Foreigll Alfairs.He call be colllacted at: Dude Vest 33A. 23/2XR Leidell. The Netherlallds. E-mail:tdel'eer@lIld.lOolllet.org

WATERLINES VOL. IS NO.1 JULY 1996 29