open doc - planning and organising

8/12/2019 open doc - planning and organising

1/21

PLANNING AND ORGANISING USEFUL EVALUATIONS

UNHCR INSPECTION AND EVALUATION SERVICEJANUARY 1998

TABLE OF CONTENTS

DEFINING THE FUNCTION

UNDERSTANDING THE PURPOSE

IDENTIFYING THE EVALUATION NEEDED

PLANNING THE EVALUATION

PREPARING TERMS OF REFERENCE

SELECTING THE EVALUATORS

IDENTIFYING THE METHODOLOGY

SUPERVISING THE EVALUATION

PREPARING A FINAL REPORT

FOLLOWING UP ON EVALUATION FINDINGS

ANNEX : Using evaluative frameworks and questions to

analyze activities.

Introduction

These guidelines are intended to assist UNHCR field and headquarters staff inevaluating and analysing the broad range of operational activities undertaken byUNHCR. The guidelines attempt to provide an elementary understanding of thefunction, a description of the different approaches that can be considered, as wellas suggestions as to how evaluations should be planned, implemented andfollowed up. The methods suggested are based on commonly appliedprofessional approaches and techniques as well as the experience gained insupervising and carrying out evaluations in a wide variety of operational

environments.

These guidelines were prepared by Lowell Martin, UNHCR, EvaluationCo-ordinator.

DEFINING THE FUNCTION

1. Evaluation is often defined and conceptualised in such a broad fashion that thefunction takes many forms and is carried out in a variety of ways. Consequently,evaluations can range from relatively thorough project monitoring that questionsassumptions and assesses whether project objectives have been achieved, tomore rigorous in-depth evaluations designed to comprehensively analyse all

aspects of an activity or operation.

2. In UNHCR, evaluations are often thought of as an examination of theappropriateness of operational aims and the extent to which they are being


2/21

achieved. In most instances, evaluations also examine whether activities arehaving the desired impact and are being carried out in an efficient andcost-effective manner.

3. A commonly employed and somewhat broader definition of evaluation used bymany UN agencies describes the function as a "process which attempts todetermine as systematically and objectively as possible the relevance,

effectiveness and impact of activities in the light of their objectives". Definitions ofevaluation tend to vary somewhat depending on how organisations use thefunction. Consequently, it is often equally important to understand what thefunction is intended to do.

UNDERSTANDING THE PURPOSE

4. UNHCR evaluations can serve many purposes. The principal aim is to provideanalyses and proposals regarding issues and concerns the organisation isattempting to address. In most cases, evaluations are intended to serve as acatalyst for change by furthering understanding and assisting decision-making.Hence, evaluations often take on both a learning and trouble-shooting character.

Evaluations can, for example, permit programme managers to answer questionsas to whether projects, programmes and activities are achieving their objectivesas well as whether there are better ways of attaining goals.

5. Evaluations are commonly thought of as an action-oriented management tooland organisational process for improving activities still in progress as well asfuture planning. They may also serve many useful functions such as providing anoccasion to bring operational staff or programme partners together to reviewactivities for which they are jointly responsible. Although evaluations aresometimes intended as a means of providing analytical information on results thatcan be used for control or accountability, they are generally much less successfulin this role.

IDENTIFYING THE EVALUATION NEEDED

6. The type of evaluation required will depend on the purpose and scope of theevaluation, as well as the time and expertise of those carrying out the work. Inmost cases, a relatively simple common sense approach applied objectively andsystematically will meet field managers' needs. Such an approach will usuallyprovide an approximation of results and causes that is sufficient for a generalunderstanding of what has transpired and generally adequate for taking decisions.

7. The most common UNHCR evaluation called for is the ad hocevaluationcarried out during implementation, often before a decision is made to enter a newphase. Such evaluations are generally requested when there are misgivingsabout implementation or the direction the activity or operation is taking.

8. Lessons learned evaluations intended to improve future operations are mostcommonly carried out after or during the late stages of an emergency orrepatriation. Lessons learned evaluations may also be organised to analyse newor innovative approaches. In most instances, they are carried out in response to aHeadquarters' request.

9. A quick and simple self-evaluation should generally be given first considerationwhen the need for decision-making information arises. Although more elaborate

evaluations can be considered when circumstances dictate, complex approachescan quickly become costly and burdensome.

10. Staff who have a general understanding of the activity to be evaluated, as well


3/21

as good analytical and listening skills, should be able to carry out a quick andelementary self-evaluation of most field activities or operations. Such anevaluation can rely on programme documents and data, as well as theperspective of those associated with an operation who are in a position to judgeits results. An evaluation of this nature would, for example, seek the views ofthose who support or are responsible for activities, including governmentpersonnel, NGO and UNHCR staff, as well as donor officials and project

beneficiaries.

11. Self evaluations have a number of advantages which include :

providing immediate and direct feed-back;

allowing staff participation in the process which increases their commitmentand sense of responsibility for the evaluation;

providing a quick and low cost means of organising an evaluation; and,

facilitating broad evaluation coverage.

allowing staff participation in the process which increases their commitmentand sense of responsibility for the evaluation;

providing a quick and low cost means of organising an evaluation; and,

facilitating broad evaluation coverage.

12. Where questions of objectivity, independence or the need for a freshperspective arise, an outside evaluation using a consultant or staff from anotheroperation is a commonly used approach. Furthermore, if external staff orconsultants are experienced in undertaking evaluations and can be devotedexclusively to the exercise, it is evident that a more rigorous exercise can becarried out. In attempting to undertake a large multi-faceted evaluation it is almostimperative that staff are solely dedicated to the exercise.

13. Using internal staff to the extent they are available not only allows staff anopportunity to improve their analytical and problem-solving skills, it also permitsthem to become familiar with the activities and problems of other parts of theorganisation as well as increases the potential for "cross fertilisation" ofsuccessful ideas, techniques and strategies within UNHCR. Thus, staffparticipation in evaluations can provide an important means of personal careerand organisational development.

14. A joint evaluation with another partner, the government or a donor may alsobe considered. Joint evaluations can prove useful in developing recommendationsthat are acceptable to more than one party or agency. They can also offer the

advantage of providing a broader perspective, more expertise, promoting a betterdialogue and facilitating future collaboration. On the other hand, they can becumbersome to organise and schedule and often lead to delays and more costlyevaluations. In addition, the absence of standardised evaluation approaches andprocedures can also lead to both friction and results which are a dilutedcompromise that fails to satisfy any of the evaluation partners.

PLANNING THE EVALUATION

15. To be successful, evaluations require thorough planning. Among the tasksinvolved are deciding on the goals of the evaluation as well as focusing theexercise on the issues and activities to be covered. In addition to clarifying the

information needs of decision-makers, a number of fundamental questions shouldalso be asked at the outset, such as:

Why is the evaluation being undertaken?


4/21

What is it expected to achieve?

How will the results be used and by whom?

16. It is crucial that users' needs are identified and effectively integrated into theplanning process. An initial step in planning an evaluation involves identifying theappropriate decision-makers and determining their information needs. A series ofdiscussions and consultations that involve decision-makers as well the staff who

implement activities on a day-to-day basis and any other interested parties shouldenable those planning the evaluation to gain a sufficient sense of the informationand analysis required as well as finalise the goals of the evaluation.

17. Before embarking on a full-scale evaluation, mull over alternative ways inwhich the objectives can be achieved. There may be a simpler approach that canbe used, such as a mission, a meeting, or a small study. The problems, theircauses, and the various solutions may already be clear. Before commencing,some thought should also be given to the various interests involved in the issue oractivity to be evaluated, the potential cost of the exercise, and the availability ofqualified staff to carry out the evaluation.

Clarifying objectives

18. At a very early stage, planners should also begin discussing the aims ofparticular programmes, projects or activities. In many instances there is a lack ofclarity about the precise goals and as a consequence, planners will often have tocollaborate with policy makers and operational staff to better define objectives,taking into account the original intent of the programme or activity as well as newthinking. In addition to goals that are ambiguous, unrealistic, or need prioritising,there may also be competing or conflicting aims that need to be considered alongwith the unintended effects of the programme or activity.

Identifying vested interests19. The various interests in the programme or activity to be evaluated should begiven some consideration. Planners will often find that many parties will beinvolved in the use of evaluation results. Their potential reaction to the findingsshould be borne in mind and their co-operation sought.

20. In addition to the decision-makers most directly involved, there is often a longlist of persons with an interest in the evaluation such as policy makers, donors,operational partners, organisations with competing interests, beneficiaries andvarious parts of the host government. The extent to which these groups may seekinvolvement in the evaluation will not always be clear. Nor will it be evident how

the interest of each are engaged or how they will respond to a particular outcome.Such aspects should, however, be given some consideration in consultations andplanning.

21. Planners should be sensitive to the interests that are likely to lead toevaluation results being intentionally misused. Misuse can take many forms, butthe most common types tend to be :

using an evaluation to justify decisions already taken;

undertaking a study in order to delay action; and,

organising a partisan evaluation with the aim of advocating or defendingparticular positions.

22. There are situations in which either evaluations should not be attempted orspecial approaches should be considered. For example, this may be the casewhen resistance is so great that a valid and reliable study cannot be made.


5/21

Finally, if an evaluation is only being undertaken because of bureaucratic orformalistic reasons, it is likely that the results will be ignored. In such cases, theevaluation is likely to be a poor use of resources.

23. In conceptualising the assignment, some thought should be given to theavailability of persons to carry out the evaluation. Planners should, however,avoid the all too frequent tendency to quickly decide on who should carry out the

assignment and the locations to be visited before the work is evenconceptualised.

Considering the feasibility and timing

24. Consideration must be given to the appropriateness and feasibility of anevaluation. After some reflection planners may decide that for several reasons it isnot the appropriate time to undertake a particular evaluation. The reasons mostfrequently cited are that activities are undergoing significant change, or are sonew that an evaluation would be premature.

25. Situational reasons may also be invoked when a postponement is sought. An

emergency or conflict in the area that would lead to staff security risks often canlead to delays. In such cases, it may be decided that the evaluation should berescheduled or that the issues or decisions can be addressed in another manner.

26. It is sometimes suggested that an evaluation should not be undertakenbecause data on results is not yet available or because the activities areundergoing change. These reasons, however, should not necessarily preclude anevaluation. In general, any project, programme, or activity can be sufficientlyevaluated at some level to adequately serve the purposes of decision-makers.

27. The timing of an evaluation is extremely important. Consequently, the annualprogramming cycle is often a crucial consideration in planning. It is essential that

analysis is provided at a time when it can influence decision-making.

28. While it is useful to anticipate and budget for evaluations, in most instances adegree of flexibility in scheduling is desirable. Problems or changes in the activityto be evaluated often make precise, pre-established plans and schedulessomewhat unrealistic.

Gaining co-operation

29. Regardless of the focus of the evaluation, it is critical that all concernedunderstand and accept from the outset the principal audience and purpose of theevaluation. To those responsible for the operation being reviewed, an evaluation

can be a sensitive matter, particularly when the evaluation seeks to document andanalyse action taken at all levels with a view to identifying the effectiveness andimpact of operational activities.

30. Staff generally feel somewhat apprehensive in view of their commitment to theactivities they manage and their fear that a superficial or unobjective evolutioncould produce unwarranted judgements. The extent to which controversy isanticipated can be a good indicator of the need to make every effort to involvethose concerned in the evaluation's planning.

31. Successfully carrying out an evaluation is normally dependent upon obtainingthe trust and co-operation of those likely to be affected by it. This includes thosewith overall responsibility as well as staff responsible for implementation in thefield. In addition, there are likely to be many other partners who have the capacityto enhance or frustrate the evaluation. Apprehension can often be reduced and


6/21

acceptance facilitated when evaluation and operational personnel sit downtogether before the evaluation to discuss goals, build a consensus, and plan theevaluation approach that will be taken.

Foreseeing follow-up

32. The importance of planning how the evaluation will be followed up can not be

over emphasised. In general, planning is even more important in producingchange than strict follow up procedures which can easily become a mereformality. In most cases, the key to successful follow-up is simply foreseeing thatthe evaluation addressees the immediate problems and issues confrontingdecision-makers and then ensuring that the recommendations are provided in atimely fashion so they can be incorporated into a decision or solution.

33. Finally, it is always wise to try to keep the exercise simple. An in-depthevaluation may not be required. In general, there is a tendency to take decisionswithout the necessary information and analysis when very little additionalknowledge is required. On the other hand, once a decision is made to evaluate,the exercise frequently becomes overly complex and comprehensive relative to

the information actually required to take the necessary decision. As aconsequence, it is often helpful to continually reassess the need for an evaluationas the exercise is being conceptualised and planned, as well as useful to look foralternative approaches that are simpler and more timely.

PREPARING TERMS OF REFERENCE

34. Precise terms of reference are essential for each evaluation in order to reducethe likelihood of misunderstanding during the study. In most cases, they constitutethe planner's most important tool for linking the evaluation's design with thedecisions to be taken. The terms of reference should provide a formal record ofagreement as to what will be done and should outline the obligations of all

participants in the evaluation. Submitting the terms of reference to managementfor their approval may also help in gaining commitment to the evaluation.

35. Detailed terms of reference for a programme evaluation study shouldgenerally include :

the reasons for the evaluation and its objectives;

a statement of the scope and specific issues to be addressed in the study;

a detailed workplan of the evaluation indicating what questions are to beanswered and tasks are to be carried out including, if possible, whatinformation is to be collected and how;

the locations to be visited;

which participants are responsible for various tasks;

a statement of the expected output;

a timetable indicating when various tasks will be completed as well as thedue dates and recipients of any periodic reports or outlines; and,

a budget indicating the costs associated with the evaluation.

36. Terms of reference, or an accompanying agreement prepared for an externalconsultant who will be undertaking the evaluation, should specify that an initialreport will be submitted in draft and provide for time to make corrections or

changes to the draft once it has been reviewed. It is also useful to stipulate that:the report will be written in accordance with an accepted outline; any substantivechanges are to be agreed upon; and that the final report and all informationgathered during the evaluation are the property of UNHCR and will not be used


7/21

for other purposes. In addition, UNHCR should retain the right to make editorialrevisions. Finally, the agreement should specify that final payment for theevaluation should be based on submission of an acceptable final report ratherthan the end of the time period foreseen to carry out the assignment.

37. Staff organising the evaluation should try to avoid the temptation to limit termsof reference to a few sketchy generalities such as to examine "the issues" and

"make recommendations for improvement". Poorly planned evaluations often thendelegate the more detailed terms of reference to the consultant.

Consultants are seldom in a position to interpret the aims and issues to beaddressed by the evaluation and the work can too easily follow the interests of theconsultant rather than the needs of the decision-makers.

38. Allowing an external consultant employed to undertake an evaluation to draftthe terms of reference can also indicate a neglect of three important maxims thatshould always be followed in the use of consultants. First, never let the consultanttake control. Second, to get something out of a consultancy the staff managingthe assignment must have a clear sense of what they expect from it. Third, the

benefit from the consultancy, is proportionate to the time invested in it by thoseresponsible for the work.

SELECTING THE EVALUATORS

39. There are advantages to using persons external to the operation to undertakean evaluation as well as benefits in having those who have been responsible forimplementation carry out an assessment of the operation. In many situations,however, an outside evaluation is the most practical approach in view of thepressure operational staff are under to perform their regular duties. If internal staffare used, it is particularly important to work out a role that ensures a degree ofautonomy from programme management.

40. An outsider with no vested interest in the operation can generally be moreobjective than staff directly involved in implementation, but a great deal of time willalso be required in familiarising an outsider with the activities under review. Usingstaff skilled in evaluation work also has advantages, but there are approachesthat can be easily understood and carried out by persons with modest expertise, ifthey are familiar with UNHCR's general policies and practices and the means theorganisation follows in achieving its aims.

41. In general, personal qualities and substantive experience are more importantfactors in the selection of evaluation staff than specific technical or methodologicalknowledge. Ideally, personnel engaged for an evaluation should be inquiring,analytical, thorough, detached, as well as have the ability to think systematicallyand rigorously. An evaluator should also have an openness to new ideas and asensitivity to the complexity and constraints associated with UNHCR's activities. Itis also important, but sometimes overlooked, that the evaluator has credibility withthe managers who will implement the evaluation recommendations.

42. The need to use outside consultants in order to ensure independence andobjectivity is frequently debated. Generally, a consultant or evaluator from outsideUNHCR is more credible if the intended users of the evaluation are also external,such as the host government or a donor to the operation. On the other

hand, if the evaluation recommendations and lessons are to be used by UNHCRmanagers, they will generally find the conclusions and recommendations morerelevant if they are an internal product. This does not, however, preclude the useof a consultant who is aware of the organisation's basic purposes and style of


8/21

working, particularly one who has been able to establish a relationship of trust andconfidence at some previous time.

43. Regardless of a consultant's evaluation skills, if an individual or team does nothave some experience with refugee operations and a familiarity with the politicalcontexts common to refugee situations, they will be greatly hampered in carryingout a comprehensive evaluation. A sectoral consultant can function more

effectively without such experience, but even then there is an increased risk thatrecommendations will be made in isolation from the operation's overall aims andwithout sufficient regard to the context. Consultants with only limited experience ofrefugee operations will also require a time-consuming briefing that can beburdensome for those called upon to supply the briefing.

44. UNHCR personnel who have not been directly associated with the operationcan often bring the same objectivity as an outsider. Seconding staff from otherparts of the organisation can frequently provide a combination of both experienceand objectivity, particularly if seconded staff are combined with a consultantperceived as completely independent.

45. Self-evaluation is another approach that can be employed. Self-evaluationsare sometimes viewed as an efficient and inexpensive way of carrying outevaluations and documenting lessons learned since they enable staff who havebeen directly involved with the programme to contribute what they know withouthaving to filter their observations through an outsider. In most cases,self-evaluations and lessons learned or problem-solving workshops areapproaches that should be given first consideration.

46. On the other hand, self evaluations often suffer from the tendency of staffresponsible for implementation to be overly critical of those they were dependentupon for support or implementation, while at the same time placing their ownefforts in the best possible light. Furthermore, those preparing a self-evaluationmay not be sufficiently knowledgeable about all aspects of the operation, to makean accurate assessment. In the worst of cases, a staff member performing aself-evaluation may have difficulties with superiors or colleagues if theself-evaluation is perceived as being critical and thus may be forced to downplayor suppress any negative results.

IDENTIFYING THE METHODOLOGY

47. Carrying out evaluation studies does not involve routine and repetitive tasks.The issues addressed vary greatly from evaluation to evaluation, as do the rangeof analyses that can be employed in carrying out the actual study. As a

consequence, there can be no detailed step-by-step procedures that areapplicable in evaluating all operations and activities. General principles guide thework, but an evaluation must be carefully designed for each situation.

48. When selecting evaluation methodology it is generally better to avoidelaborate and complex designs. Most methodologies nevertheless involve avariety of methods and techniques in collecting and analysing data that, forexample, include :

a review of plans and policy guidance;

interviews, group discussions and possibly questionnaires;

analysis of reports and data;

on site observations and possibly statistical sampling and inferences; and,

comparisons and analysis of standards, costs and performance.


9/21

49. Of paramount importance is the need to work systematically and objectivelythrough an analysis of issues and activities. In most situations, a variety ofcommon sense approaches normally employed in informal research will produceacceptable results. More rigorous techniques can be employed when the situationand the skills of the evaluator permit.

50. Rigorous and relatively complex evaluations require considerable resources

and time and are best carried out by persons with some background in evaluationwork or at least a degree of expertise. Such persons have the advantage ofunderstanding how to apply the variety of analytical approaches and evaluationtechniques that can potentially be used.

51. There are many analyses and techniques which are based on certainassumptions, conditions and principles which need to be understood by theevaluator in order to avoid misusing the analysis and drawing erroneousconclusions. Although often attempted, sampling, for example, is susceptible tomisuse without some knowledge of sampling theory. Similarly, preparing andinterpreting questionnaires generally requires a degree of expertise for the resultsto have meaningful validity.

52. For most UNHCR evaluations, a general exposure to a variety of analyticalapproaches is sufficient. Staff familiar with the planning and monitoring of refugeeoperations, but without any particular expertise in evaluation, can use anapproach primarily based on interviewing. Such an approach provides anapproximation of results and causes that is sufficient for a general understandingof what has transpired and for providing advice regarding decision-making.Despite limitations to the approach, it is perhaps the most commonly employed. Itis also one of the easiest to carry out and should not be difficult for anyone with ageneral understanding of the activities to be evaluated.

53. In evaluating complex programmes or operations, complicated researchmethodologies are generally inappropriate, although more sophisticated designscan be considered in evaluating various sectoral activities. Furthermore, the lackof hard data in most situations often means that evaluations of field activities mustrely heavily on more subjective information obtained through interviews andon-the-spot assessments.

54. Most evaluations of refugee activities are not viewed as a scientific researchactivity but rather as an aid to decision-making and management. UNHCRevaluations, therefore, generally aim at producing objective but not necessarilyconclusive results. Consequently, evaluators must often aggregate inferencesobtained in a variety of ways rather than seeking a definitive assessment through

a more complex and rigorous method.

55. Once staff planning the evaluation have held thorough discussions concerningthe operation, its activities, issues and any problems being encountered, manysteps involved in the study should be reasonably clear. The evaluator mustdevelop a conceptual understanding of the operation as a process for achievingcertain ends. At the same time, the evaluator should essentially disaggregate theoperation, viewing each element of management, (planning, staffing, etc) andeach activity and sector, from a number of different perspectives. (See Annex forexamples of several frameworks and questions that can be considered.)

56. The overall operation and its parts should be analysed and performance

assessed, using whatever measures are available, including implicit or statedobjectives, criteria and standards. Much of the evaluation work may be devoted togaining an understanding of what can reasonably be expected in a given context.Organisational, national or international standards are often useful measures, but


10/21

in most instances can only be tentatively applied.

57. The most essential characteristic of evaluation studies is their attempt tomeasure results against objectives. Unfortunately it is not always easy to achieveclarity and precision in objectives. Objectives are often vague, general and difficultto measure. In many instances, there are several objectives with differentpriorities and complex inter relationships which may, in fact, not even be

complementary.

58. Multiple and ambiguous objectives need not be a major obstacle in carryingout useful evaluations. In most cases a wide range of evaluative perspectives canbe applied rather than simply assessing the extent to which the operation and itsactivities have achieved their stated objectives. For example, the evaluation couldalso analyse the impact of programme or project activities have had on therefugee target group or in other words, how the operation or activities havecontributed to the refugees' well-being or significantly changed their condition. Inaddition, the evaluation could attempt to analyse the cost effectiveness ofactivities and how efficiently resources have been used. Finally, the evaluationcould consider the economic, cultural and institutional impact of programme

activities, as well as any unanticipated effects.

59. Evaluations should not only examine the results, or likely results, but also tryto understand how various interventions actually produce the final outcome. Ananalysis of the relationships which can be seen to exist between activities is oftenmore illuminating than a simple analysis of accomplishments. Casualrelationships are particularly important since the evaluation's conclusions andrecommendations regarding a situation will normally emerge from the causes.

60. In trying to document lessons, evaluators may want to ask such questions as:if the operation were being mounted again what approaches, systems orprocedures could be changed to provide a better framework within which tooperate, or conversely, what policies were pursued in the operation that wererestrictive or inappropriate to the situation.

SUPERVISING THE EVALUATION

61. It is essential that staff responsible for planning and organising the evaluationthoroughly guide and supervise those who carry out the exercise particularlyduring the initial and final phases of the work. Of particular importance is athorough discussion of the assignment's objectives, assumptions, potentialoutcomes and the interests of various parties involved during the early stages ofthe evaluation.

62. It is often useful to accompany an outside evaluator to early meetings. At alater stage the topics covered during the evaluation can be discussed with asample of those being interviewed to ensure the relevant questions are beingasked. Discard the notion that external consultants should be as independent andself-sufficient as possible and instead endeavour to support the evaluators asmuch as possible. If the capacity is available, do basic research and providedocuments. This will save time and help to ensure the work is properly done.Finally, try to make office space available which can ensure regular consultationsand permit private meetings.

63. Experience has shown that one of the most crucial points of supervisory

involvement occurs when conclusions and recommendations are formulated andwhen the report's structure is developed. It is at this stage that external staff oftenneed assistance in shaping evaluation findings and formulating recommendationsthat are feasible and realistic. Guidance in developing the report outline can also


11/21

help ensure that the most important messages are brought to the fore.

PREPARING A FINAL REPORT

64. A report presenting the evaluation conclusions and recommendations is animportant management tool and an essential element of any evaluation. Thereport should serve as a focal point for debate and discussion, facilitate

decision-making, and serve as an important record for the future. A writtendocument also provides those who did not have an opportunity to contribute to theevaluation to add their comments later. For those who may not have been directlyinvolved in the operation, such as staff responsible for training or similaroperations elsewhere, the report may provide a wealth of ideas and informationthat can help guide other operations or serve as a case study for planning andtraining.

65. Reports should be tailored to the audience and the material being presented.For most readers, however, the more concise, analytical and balanced the report,the more useful it will be. Unnecessarily lengthy reports are a common error thatshould be avoided. This includes the temptation to present superfluous

information in the hope it will be of "use to someone", demonstrate how muchwork was done, or show the thoroughness of the analysis.

66. Reports should normally include a summary or an overview that is thoughtfullyprepared in anticipation that most of the reports recipients will not have time tothoroughly read the whole report. Some background should be included at theoutset that explains the evaluation approach used, what was done, and why.Essential background on the operations or activity should also be presentedbriefly explaining its origin and purpose, organisation and resources involved. Inmost cases, a historical description should, however, be kept to a minimum.

67. In general, evaluation reports should communicate information in a way that

can be easily understood and inspire action. Conclusions should be logical,justified and follow from the evidence presented. Recommendations should also

be well supported by the body of the report which should include sufficient data,examples, argumentation, or expert opinion to sustain the findings.

68. Recommendations should be feasible and clearly identify who is expected totake action. Issues or questions which need further study or consideration shouldbe identified and discussed. Evaluators should, however, avoid or very judiciouslymake commonplace recommendations for more study, more resources and betterco-ordination which have become evaluation clichs. The same general advicewould apply to simplistics exhortations for improved performance.

FOLLOWING UP ON EVALUATION FINDINGS

69. There is always significant potential for an evaluation report to become an endin itself. It is therefore imperative that plans are developed that will ensureappropriate follow-up action on report recommendations and any decisions taken.Decisions should be incorporated into an action plan for follow-up indicating howchanges will be made, by whom and when.

70. Responsibility for follow-up should normally be assigned to staff with at leastsome direct responsibility for the functional areas in which changes are to be

made. There are, however, many different ways to approach follow-up. A few ofthe most common possibilities include :

convening a meeting to review the evaluation's conclusions andrecommendations and to direct implementation of the follow-up decisions;


12/21

using the evaluation as input to a working group;

using the evaluation to formulate the agenda of a policy setting or strategicplanning meeting;

incorporating evaluation findings into policy papers, guidelines or trainingmaterial;

using the evaluation report to support UNHCR's negotiating position in

discussions with government, donor or other agencies;

using the report as a basis for a meeting with other agencies or NGOs;and,

widely disseminating the evaluation report to all countries where similaractivities or operations have taken place.

71. An independent organisational component is often assigned responsibility formonitoring follow-up. Under such an arrangement, those responsible formonitoring follow-up are kept informed of any action taken or changes producedas a consequence of the evaluation. Ideally, the entire process should culminatewith an implementation report describing any decisions taken on the basis of theevaluation and what steps have been taken as a result.

Annex

Using evaluative frameworks and questions to analyse activities

1. In simplest terms, carrying out an evaluation often means making overalljudgements about an activity as well as identifying problems and their causes sothat they can be corrected and avoided in the future. To judge an activity, theevaluator should analyse it as a whole from a number of perspectives using avariety of measures and criteria. In most cases, an analysis is facilitated byseparating the activity into logical parts in order to examine and assess each ofthe parts that make up the activity.

2. In breaking an activity or operation into logical parts, it has always beencommon in UNHCR to separate and analyse activities by each programme sector,such as water, food, health etc., or by functional components such as protection,assistance, external relations, etc. The phases of an activity can also provide auseful way of understanding and analysing it. A useful but less commonlyemployed framework in UNHCR is the "logical framework" model that breaks anactivity into inputs, activities, outputs and objectives.

3. Once the most identifiable or essential elements of an activity have been

identified, each element can then be analysed using basic evaluative conceptssuch as:

Effectiveness which considers the extent to which an activity hasachieved or is likely to achieve its objectives;

Efficiency which considers how well resources are used toundertake activities and achieve objectives;

Economy (or cost-effectiveness) which considers whetherobjectives could be accomplished at a lower cost, examines waste,and assesses whether results or benefits justify the cost; and

Impact which normally considers an activity's contribution to thewell-being of the beneficiaries, and assesses change brought about.


13/21

Other aspects often considered include :

Relevance which considers whether an activity or project is well suited orappropriate to the needs or situation.

Unanticipated consequences which considers any significant unforeseeneffect of an activity, either beneficial or detrimental.

4. A framework based on management functions can also be usefully employed todisaggregate or analyse. Using such a framework, an operation, activity, orsector, can be further reviewed in terms of its :

planning;

organisation;

staffing;

directing (or implementation); and

controling (including monitoring).

5. Various frameworks can provide useful starting points for reflection andanalysis as well as provide a way of ensuring aspects have not been overlooked.Many staff find question based frameworks the easiest to understand and apply.

6. The simplest, and perhaps most frequently applied framework in monitoringand evaluation is a form of :

what is right?

what is wrong?

why? and,

what needs to be done?

7. As the assessment becomes more thorough, other elements are normallyadded to the framework, such as a statement of objectives and assessment oftheir achievement. Inevitably, any discussion of objectives leads to furtherquestions regarding the assumptions that the objectives are based upon, and thecontinuing validity of the original objectives.

8. A simple but slightly more complex framework that is often employed as a basisfor self-evaluation includes questions such as :

what was the objective?

was the objective met? If not why not?

Were the plans and resources adequate?

What changes and alternatives should be considered?

what lessons were learned?

9. Contributions of the various conceptual framework and questions describedabove provide a good starting point, but it will quickly become apparent to moststaff that the number of possible questions is almost unlimited. Consequently, animportant aspect of planning and carrying-out an evaluation is determining whichquestions and answers are the most crucial in understanding and assessing anactivity. In undertaking most evaluations and in organising a final report, it isgenerally useful to keep in mind that the following elements will have to be fully

developed and analysed for each finding or section of the report:

What is the condition, situation or problem that requires correction?

What effect or significance does the problem have?


14/21

What is the cause or reason for the condition?

What conclusions can be drawn and recommendations made to addressthe causes and correct the problem?

10. Such frameworks serve as tools to be used and adopted as needed. Theyshould, however, not be applied so rigidly that they become cumbersome. Rather,they should be modified and adapted to suit the information and analytical

requirements of a particular situation.

11. An example of the kinds of questions that would be reflected upon and askedin reviewing a UNHCR country operation are presented below in order to serve asan illustration of the many aspects that should be considered as well as provide aseries of questions that may serve as a starting point in carrying out such areview.

12. Many of these commonly asked questions can assist staff in preparing termsof reference as well as aid in carrying out the actual evaluation. They can alsostimulate thinking and help in generating many other essential questions.

I. UNDERSTANDING THE SITUATION

What access do we have to refugees

Are they of clear concern to UNHCR

What would happen if refugees were not granted asylum, assisted

Are lives in danger

Are any refoulements, or deportations of asylum seekers

Are communities located and organised in a way that provides security

Are new movements imminent or has the situation stabilized

How rapidly are circumstances changing

Are refugee needs being seen in isolation from other affected persons

What are the principal concerns of the host government

Have refugees had a destabilizing effect or other impact on the area

What is known about the refugees

What types of contacts are maintained with refugees

How are they organized

What is the number and make-up of the group in terms of origins,vulnerability, age, gender, skills and social-economic background

Are there significant gaps in statistics and knowledge

How accurate are populations figures

Are events in the country of origin understood

What is known about refugees' intentions and desires

What options are available to the refugees

What efforts have been made to identify and seek solutions

What is known about the safety and feasibility of repatriation

What is the likelihood of local settlement

How self reliant are the refugees

Will refugees settle spontaneously

What are the prospects for integration


15/21

To what extent will refugees have to find their own way

How will the experience change the refugees and effect the solution

What is the likelihood of refugees returning to their traditional way of life

Are there aspects of the situation that attract refugees

What are their needs, and how are they evolving

What information is available regarding the beneficiaries or situation,including studies, reports, data, etc.

What will happen if refugees are not assisted

Would assistance attract opportunity seekers

Are refugee privileged in any way compared to nationals

How does the refugee presence affect nationals or the host country

How long have refugees been receiving assistance

What are their attitudes toward assistance

How do standards compare to national and country of origin standards

What are the limitations to assisting or protecting refugees

What are the prospects for self reliance, in what ways

How can more self-reliance be achieved, what are the barriers

II. REVIEWING THE PLAN AND APPROACH

Has there been an adequate diagnosis of the situation and analysis of thepotential beneficiaries

Has data been systematically collected and analysed

What data, assessments and research are required

Has feedback and experience been taken into account

What is UNHCR's potential role

Are there any precedents for UNHCR actions

What are the reasons for involvement and criteria for phase out

What are UNHCR's comparative strengths in the situation

What should be the nature, extent and duration of UNHCR involvement

What are our principal problems and constraints

Have all critical factors been considered including the political, economic

and cultural factors, as well as the weather and logistical considerations

Are there lessons from other situations that can be drawn upon

What aspects of the situation are likely to change

Have various scenarios been taken into account

What are the prospects for the future

How can refugees be assisted toward solutions

Have needs been prioritized

What major assumptions were made in designing the activity

Have the assumptions proved accurate

Were there major considerations or problems that should have beenforeseen but were not

Are assumptions and expectations realistic


16/21

Have the objectives and strategies been sufficiently clarified

Were alternative approaches to achieving the objectives considered

Are objective precise or vague statements of intent

Are the objectives sufficiently explicit so that their achievement bedetermined

Are there specific indicators against which the attainment of objective canbe assessed and impact determined

What are the principal strategies

Were alternate approaches and strategies considered

Are activities within UNHCR's mandate and appropriate

Which needs are a priority

What activities are the most critical

To what extent do activities correspond to actual needs

What were the main inputs, activities and outputs

What is the status of activities

What major changes, external, institutional, political, or economic haveoccured, since the activity was planned and will have a substantial impacton results

What changes are being sought or planned

Do activities serve a sufficient number of potential beneficiaries

Can activities be reduced without negatively affecting refugees

What are the potential difficulties associated with activities

What are the principal constraints

Are activities consistent with solutions

Are activities and services linked to longer term integration or repatriation

Will activities contribute to rehabilitation and reconstruction

Do activities discourage initiative and lead to dependence

Does the activity logically fit and meaningfully relate to an overallframework

Does the activity relate to similar parallel, prior, or subsequent activities

What administrative and material support is required

What are the links between protection and assistance activities

Is the plan comprehensive and coherent or a justification for piece-mealinputs

Is there potential to rationalise and harmonize activities

Do activities have a planned completion, can a date be set

Do activities make sufficient provision for phasing out

How have activities begun and developed

Are activities developing a life of their own separate from needs

Do activities imply permanence

Once begun are activities likely to expand

How was the programme to be implemented e,g, how was it organised

Who are the principal beneficiaries and how were they to benefit


17/21


18/21

Are there related country or regional initiatives

Are related initiatives successful, can they be drawn upon

Is there external expertise which can be utilised

In which areas do organizations have a presence

What do others have to offer

Do roles conflict or result in duplicationWhat activities can or should be carried out by others

Are there particular areas and sectors in which partners are interested oruninterested

Do partners have the mandate, expertise, material resources including staffand money to assume activities

What commitments has UNHCR made

Is UNHCR being drawn into development activities

Is there a mismatch of mandates, objectives or approaches

How is UNHCR perceived

Have partners capabilities been identified

Do partners have the capacity and skills required, how can we supportthem

Do partners share common goals

Was there adequate consultation and participation with those related to oraffected by the plans

Is there agreement among partners on UNHCR's plan and strategy

What are the links and arrangement with others

Are activities well coordinated

Are sectoral activities or responsibilities successfully divided amongpartners

To what extent is there competition among agencies

How can UNHCR collaborate more effectively with other organisations

What type of support and assistance is available from donors, partners,other agencies and host government

Is the support provided by partners adequate, good

Can interest and support be increased

Have public information activities been adequately covered

What are the principal external constraints

IV. ANALYZING IMPLEMENTATION

Have assumptions proved accurate

Have plans been followed

Have the objectives changed during implementation

What major changes have occured since activities were planned

Were there problems or considerations which should have been foreseen

Is preparedness adequate

Are activities fragmented and disconnected


19/21

Are assistance activities contributing to protection

Have roles and responsibilities been logically delineated andcommunicated

Are lines of authority and reporting clear

Do all levels have the decision-making authority they need

Are the level and forms of activities consistent among locations

Are there staff or skill shortages

Have staff received sufficient training and orientation

Does the absence of refugee participation and training weaken activities orhinder hand-over

Were national resources, local expertise and materials used to the extentpossible

Do material, staff and financial resources match the needs,

Were resources mobilised as required

What problems were created by the unavailability of resources

Would additional or fewer resources make a significant difference

Have there been delays in the preparation, receipt and signing ofinstructions and agreement

Have schedules been adhered to

Which problems have lead to delays and what were the consequences

Has information been communicated in a timely manner

Is information shared among partners and staff

Has adequate guidance been provided

Was decision-making delegated to the appropriate level

Are decisions made in a timely way and at the right level

What slows or hampers decision-making

Have financial and administrative procedures been followed

Are activities in compliance with UNHCR policies and standards

Have procedures been developed and formalised as required

Are activities consistent with policies, practices and guidelines

What guidelines, policies or decisions are needed

Are co-ordinating arrangements adequate in preventing duplication andensuring all sectors, areas and groups are adequately covered

Are contractual arrangements adequate

How well were activities actually coordinated

Has UNHCR been able to influence and guide agency programmes

Are the actions of all partners directed toward solutions

Have partners performed adequately

What are the principal source of conflicts

Can UNHCR be less operational; what can others do

What sectors or activities is UNHCR most involved in, least involved in

What stages have activities passed through


20/21

Have activities begun well or poorly

What were the strengths and weaknesses of implementation

Have structures, systems and procedures facilitated or obstructed activities

Have approaches been cost effective

What measures could potentially reduce costs

Have skills been developed and operational capacities createdHow well are refugees or returnees being prepared

Are activities routinely monitored and corrective action taken as required

What are the principal constraints or impediments to implementation

What changes would improve implementation or services

V. EVALUATING RESULTS

Is the plan logical and coherent

Were measures to provide security and protection adequate

Are the most needy beneficiaries targeted

Have activities contributed to solutions

Have activities helped to prevent further population displacements

Was social conflict prevented or limited

Are measures adequate to ensure activities will be sustained after UNHCRphases out

Were needs overlooked or neglected

Are the trends satisfactory or worrisome

What initiatives offer potential

Which activities or sectors have not achieved the desired standard

Was self reliance encouraged and achieved

Have we helped to draw attention and mobilize support

Has the operation had a substantial impact on government policies orpublic opinion

Have we honoured commitments

What long term effects is assistance having

What will happen after our withdrawal

Is it likely that the activity will have the expected impactHave activities made a significant difference

Has the operation made a substantial contribution to the refugees' wellbeing

Have activities significantly changed or improved conditions of thebeneficiaries

What important successes or failures occured

Which sectors and activities have been the most successful, the leastsuccessful

What are the most successful and unsuccessful aspects of activities,reasons why, and their potential

What approaches or strategies were the most successful or problematic


21/21

What are the principal problems

What weaknesses or strengths have surfaced

What has been accomplished to date

How do achievements compare to plans and objectives

Did the assistance and services cover needs

To what extent is the activity likely to achieve its objectives, if not, whyWere the resources effectively used

Were activities efficiently carried out

Could the objectives have been accomplished at a lower cost

What areas or activities offer potential cost savings

Do results or benefits justify the costs

Were activities excessively wasteful

Which activities were most costly relative to their priority

Does the activity complement, duplicate, overlap, or work atcross-purposes with other programmes

Was assistance significant or marginal value to beneficiaries

Were objectives relevant and worth pursuing

What are the most useful activities and sectors

Is the activity having any significant unexpected effects, either beneficial ordetrimental

Why did the operation succeed or fail

What factors affected project performance, how

Were there better ways to address the needs

What should we have done differently

What should we now be doing more or less of

To what extent can activities be reoriented

What would happen if parts of the operation were eliminated

What alternatives to the current plan merit consideration

Where is improvement needed

What new programmes, policies, activities, systems or guidelines arerequired

What lessons can be learned

open doc - planning and organising

Documents