UNITED NATIONS HIGH COMMISSIONER FOR REFUGEES EVALUATION AND POLICY ANALYSIS UNIT

Enhancing UNHCR’s capacity to monitor the protection, rights and well-being of refugees

Synthesis of findings and recommendations The full report can be found on: www.unhcr.org/epau

By Ninette Kelley, kelley@unhcr.org Peta Sandison, petasandison@ntlworld.com Simon Lawry-White, simon@vinemanagement.co.uk Consultants.

EPAU/2004/06 June 2004

Evaluation and Policy Analysis Unit UNHCR’s Evaluation and Policy Analysis Unit (EPAU) is committed to the systematic examination and assessment of UNHCR policies, programmes, projects and practices. EPAU also promotes rigorous research on issues related to the work of UNHCR and encourages an active exchange of ideas and information between humanitarian practitioners, policymakers and the research community. All of these activities are undertaken with the purpose of strengthening UNHCR’s operational effectiveness, thereby enhancing the organization’s capacity to fulfil its mandate on behalf of refugees and other displaced people. The work of the unit is guided by the principles of transparency, independence, consultation, relevance and integrity.

Evaluation and Policy Analysis Unit United Nations High Commissioner for Refugees

Case Postale 2500 1211 Geneva 2

Switzerland

Tel: (41 22) 739 8249 Fax: (41 22) 739 7344

e-mail: hqep00@unhcr.org

internet: www.unhcr.org/epau

All EPAU evaluation reports are placed in the public domain. Electronic versions are posted on the UNHCR website and hard copies can be obtained by contacting EPAU. They may be quoted, cited and copied, provided that the source is acknowledged. The views expressed in EPAU publications are not necessarily those of UNHCR. The designations and maps used do not imply the expression of any opinion or recognition on the part of UNHCR concerning the legal status of a territory or of its authorities.

Contents Introduction............................................................................................................................. 1

UNHCR’s mandate and monitoring.................................................................................... 3

Monitoring and programming ............................................................................................. 5

Collecting monitoring information .................................................................................... 13

UNHCR staff capacity.......................................................................................................... 17

Staff development................................................................................................................. 18

Partners and monitoring...................................................................................................... 21

Managing monitoring initiatives........................................................................................ 24

Appendix 1. Checklist of recommendations .............................................................. 27

Appendix 2 Terms of Reference ................................................................................... 35

Appendix 3. Project methodology ................................................................................ 39

Appendix 4. Review of the evaluation process ........................................................... 41

1

Introduction

1. In recent years UNHCR has endeavoured to enhance its monitoring capacity, recognizing the critical importance of this function for improved organizational performance and accountability. For UNHCR, monitoring plays a central role in the fulfilment of its mandate to protect refugees and to coordinate international action on their behalf.

2. In view of its importance, and in the light of recent steps that the organization has taken to improve its project management and monitoring systems, UNHCR’s Evaluation and Policy Analysis Unit (EPAU) commissioned a team of three consultants to provide an empirically supported assessment of UNHCR’s ability to monitor the protection, rights and well-being of refugees.

3. We were asked to concentrate on UNHCR’s monitoring activities in developing countries with relatively large and longstanding refugee populations, and to make specific and realistic recommendations for the enhancement of UNHCR’s monitoring capacity, drawing on examples of good monitoring practice in UNHCR and other organizations.1

4. The main body of our report is comprised of the following seven chapters:

Chapter 1: A typology of monitoring

Chapter 2: The protection framework

Chapter 3: UNHCR's monitoring cycle and the assistance sector

Chapter 4: Monitoring in the protection sector

Chapter 5: Monitoring and the UNHCR management environment

Chapter 6: Monitoring with and through partners

Chapter 7: Monitoring in other agencies

5. This synthesis of our findings does not reproduce the evidence and analysis found in the chapters. Rather, it provides a succinct overview of the findings and recommendations to be found there. Relevant chapters are cross referenced throughout the synthesis.

6. It is important to mention at the outset that we found overwhelming agreement throughout the organization with regard to the importance of monitoring. We also found a high degree of interest in improving it. We witnessed valuable monitoring innovations, both at headquarters and in the field.

7. As described in more detail throughout our report, simple and effective systems have been implemented which have real potential for contributing to

1 Our methodology is described in Appendix 3.

MONITORING

2

improved monitoring. To reap the full benefits of such initiatives, however, certain weaknesses in UNHCR’s monitoring systems and capacity will have to be addressed.

8. One of these weaknesses is the lack of a fully integrated and harmonized approach to monitoring. For example, UNHCR has in recent years issued a plethora of uncoordinated guidelines and instructions which, combined with increasing demands for ad hoc monitoring information, tend to overwhelm staff in the field.

9. Additionally, monitoring is often impeded by the absence of a participatory and team-based approach to planning, in which the protection dimensions of all interventions are recognised and in which monitoring responsibilities are clearly delineated among staff.

10. Other significant impediments to effective monitoring include the infrequency with which senior staff, including programme, community services and protection officers, actually engage with refugees. This is due in part to demanding workloads, staff shortages, excessive reporting requirements, as well as the fact that UNHCR offices are increasingly located at a distance from refugee camps for security reasons.

11. Despite the growing interest in the organization for systematic monitoring, a rigorous and consistent approach to the function has not yet been developed. In the course of this review we observed deficiencies in data gathering tools (such as poorly defined indicators, an absence of targets and means of verification), and the inefficient use of monitoring information provided by partners.

12. Opportunities are also missed for UNHCR to play a strategic monitoring role, synthesising the monitoring information from partners, beneficiaries and from the broader political environment to provide value-added monitoring feedback to the humanitarian community.

13. We also encountered cumbersome reporting formats that do not present monitoring information in a readily accessible manner, so as to enable timely corrective decisions to be made. Demoralization as a result of funding constraints also contributes to weak monitoring; UNHCR staff sometimes consider it futile to report a failure to meet objectives when both the cause and the remedy is beyond the control of the programme. 2

14. In our report we have examined a range of options that could help the organization to overcome some of these difficulties. In this context, it is important to stress that while effective monitoring depends on good management, tools and systems, these alone are insufficient. Equally if not more important is to have staff present among refugees, with the awareness and sensitivity to know what monitoring data it is necessary to collect and communicate.

15. The recommendations of this report are primarily addressed to UNHCR management. Because monitoring engages so many different parts of the organization, it has been difficult at times to target each of our recommendations at a specific department, division and/or unit. We have, however, attempted to do so.

2 As a recent study by ALNAP noted, many other humanitarian agencies face similar challenges. For more on this and the relative strengths and weaknesses in other organizations’ monitoring systems see Chapter 7, ‘Monitoring in other agencies’.

SYNTHESIS REPORT

3

16. Finally, our recommendations are not all cost free, which presents difficulties in the current funding environment. Some, such as the development of new monitoring tools and increased staffing to carry out monitoring in the field, will require additional resources. However, the majority of our recommendations are designed to increase efficiencies, to free staff time and to rationalize current in-house initiatives at no additional cost.

UNHCR’s mandate and monitoring

17. UNHCR is mandated to safeguard the rights and well-being of refugees, to lead and coordinate international action for their worldwide protection and to seek permanent solutions to their plight. 3

18. The protection of refugees has a legal as well as physical dimension. The rights to fair treatment upon reception, not to be returned to prospective persecution and other recognised rights to adequate health, housing, food, shelter, education, are addressed through UNHCR’s wide-ranging legal and material assistance activities.4

19. Effective monitoring contributes to all UNHCR’s mandate to protect refugees, serving as key performance and accountability tool. It provides UNHCR with an on-going review of operations, allowing the organization to ascertain whether its interventions are proceeding according to plan, within agreed timeframes and costs, and whether they are having the desired impact.5 This enables UNHCR to review and redesign its programmes, to improve performance.

20. Monitoring also enables UNHCR to inform other stakeholders of the state of the world’s refugees necessary to advocate for an impartial and equitable allocation of funding to different programmes around the world. The latter task has become of pressing importance to UNHCR as a result of significant funding shortfalls and the perceived weakness of UNHCR’s ability to demonstrate the impact of those cuts.

Types of monitoring in UNHCR

21. UNHCR undertakes various forms of monitoring. Input monitoring looks at whether human, financial and material resources are mobilized and deployed as planned (e.g. monitoring of disbursements to implementing partners). Output monitoring determines whether products or services are being delivered as planned by UNHCR and its implementing partners (e.g. the building of schools). Monitoring outputs is referred to in UNHCR as "performance" monitoring, which it differentiates from "impact" monitoring.

22. Impact monitoring relates to a programme’s objectives and establishes whether the intended outcome of a programme is actually being achieved (e.g. improved 3 For a more detailed discussion of UNHCR’s mandate and mission statement, and a how its work has expanded to include ‘others of concern’ (such as returnees and internally displaced persons) see Chapter 2 ‘The Protection Framework’. 4 Although UNHCR has viewed its legal and material assistance activities part of its protection function, operationally an unhelpful distinction is often made between the two types of interventions. For the consequence of this see Chapter 4 “Monitoring in the ‘Protection Sector’” 5 Section 1.4, UNHCR Manual Chapter 4, June 2003; this is also the definition used in the Operations Management System, described in Chapter 5, ‘Monitoring and the UNHCR management environment’.

MONITORING

4

literacy rates). Measuring impact is considerably more complex and unpredictable than performance or output monitoring.

23. Impact monitoring requires UNHCR to take account of factors outside its immediate control and to concern itself with the perceptions and value placed upon interventions by the primary stakeholders themselves.6 Process monitoring reviews how a programme is managed and implemented (e.g. whether stakeholders are consulted and encouraged to participate).7

24. As well as monitoring its own and its partners’ activities, UNHCR's protection mandate and coordination function requires the organization to develop a much broader knowledge and understanding of refugee situations. This is often referred to as situation monitoring. Its purpose is to provide a knowledge of the overall coverage of needs by actors other than UNHCR, the situation of refugees not directly assisted by the agency, as well as contextual issues that directly or indirectly affect the organization’s ability to meet its objectives.

Who monitors?

25. UNHCR monitoring occurs at refugee camp, field office, sub-office, branch office and Headquarters levels. All programme, protection, community services technical and field staff have monitoring responsibilities, under the overall accountability of the Representative.

26. Large amounts of monitoring data are also collected by UNHCR’s implementing partners (those contracted by UNHCR to provide services) and its operational partners (agencies operating in the field but without a contractual relationship with UNHCR). Nearly half of UNHCR's operational funds are disbursed through implementing partners (IPs), and the data which they collect is transmitted to UNHCR staff by means of reports, co-ordination meetings and other types of informal and formal contact.

27. At Headquarters, monitoring is focused on tracking core programme and protection issues at the country level in order to report the Executive Committee, seek state compliance with the international protection regime, inform donors and other actors of programme and protection concerns and to ensure that UNHCR operations are consistent with the organization’s policies and guidelines.

28. Monitoring responsibilities at Headquarters are spread out across a wide range of different entities: the regional bureaux, which review and approve annual plans for country operations; the Division of Operational Support, which tracks activities in different sectors and priority policy areas; the Programme Coordination and Operations Support Section (PCOS), which is responsible for the development of the Operations Management System (OMS); the Department of International Protection (DIP), which sets protection standards, and monitors whether they are consistently attained in UNHCR operations; the Inspector General’s Office (IGO); which examines the quality of UNHCR management and representation; and the Audit 6 Using tools like Situation Analysis and Action for the Rights of Children (ARC) that focus on the wider operational context and identify refugee capacities and resources as well as problems and needs. See Chapter 1, ‘A typology of monitoring’. 7 Ibid.

SYNTHESIS REPORT

5

Service, which examines the way in which financial resources are used and the extent to which UNHCR’s financial procedures are being followed.8

Monitoring and programming

29. Monitoring forms a crucial part of the project cycle. Quality monitoring is highly dependent on quality needs assessment and programme design, an issue of current concern to UNHCR and the wider humanitarian community. While it was not within our terms of reference to study UNHCR’s assessment capacity, our informants frequently made reference to the fact that some of the weaknesses in UNHCR’s monitoring capacity are directly related to poor or absent initial assessments.

30. To date, budgeting in UNHCR has been resource-led. The High Commissioner has now called for operations to be budgeted on the basis of refugee need, rather than the resources available. Once UNHCR has agreed core standards and indicators, it will be in a better position to assess the extent to which those standards are not met and to express that gap in monetary terms.

31. A lot of work in recent years has gone into the development of planning guidelines that will support staff to design what they should be monitoring. Similarly an increasing number of reports have been developed to communicate the information. Some of these are described below. Much less has so far been achieved in terms of the how and why of monitoring. Indeed the coherence between what is collected, how it is collected, how analysed, communicated and used is unclear.

Clarifying information needs

32. Monitoring is a management tool, and monitoring information is collected for a reason: to respond to refugee needs and to safeguard refugee welfare by means of improved programme quality. Although theoretically motivated by these objectives, monitoring in UNHCR is often dominated by a narrower interpretation of accountability, i.e. the pressure to report.

33. Clarity regarding the use of data should underpin the design of any monitoring system. Monitoring systems should also incorporate appropriate communication methods, be they narrative or numeric, and a clear definition of the differing information needs at each point in the monitoring chain. Information that is needed in a refugee camp (where details of latrine construction are important, for example) is not necessarily required in Headquarters (which may need to know a camp’s morbidity rates, but will not need to know the number of its latrines).

34. Within UNHCR a more rationalised approach to monitoring is required, systematically identifying the monitoring information required by managers and staff members at each level of the organization, as well as the format in which it should be presented. Such an approach should reduce the large number of requests for ad hoc reports received by the organization’s field offices.

8 For the various sections within DIP that have monitoring responsibilities see Chapter 4, ‘Monitoring in the protection sector’.

MONITORING

6

35. Streamlining and rationalizing the standards and indicators that are in use across the organization (discussed below) will help to prioritize information needs. Such action should be accompanied by the introduction of firmer accountabilities (discussed below under ‘Managing monitoring initiatives’) that hold managers responsible for their use of monitoring information.

The logical framework

36. As part of the development of monitoring planning tools, the Operations Management System (OMS) initiated the use of the ‘hierarchy of objectives’, a variant of the logical framework, which provides a framework for country programmes to develop objectives and indicators for each sector at each level of the programme. Logframes are typically based on a 4x4 matrix, whereas UNHCR has adopted a simplified version removing the two columns which describe how monitoring data will be collected (the means of verification or MOV), and the assumptions and risks associated with the each component of the plan. 9 Although the resulting simplicity of the logframe facilitates its absorption in the field, this arrangement has had some negative consequences for monitoring.

37. The MOV describes how the indicator information is to be collected. Proper use of the MOV helps ensure that resources (participatory or survey skills, time, computer software etc.) are made available for it. It also helps to ensure that the selected indicators are realistic. Its absence may be contributing to weakly defined plans for how programmes will be monitored and who will be responsible for monitoring in UNHCR.

38. Similarly, the removal from the logframe of the assumptions and risks associated with each component of the plan may reinforce UNHCR’s tendency to focus on the performance of programme interventions rather than on monitoring the external conditions and context in which they take place. The latter is directly relevant to planning, implementation and impact.

Recommendation:

DOS should retain the simplicity of its two-row logframe design but should re-insert the columns showing the means of verification and indicators for assumptions and risks so that UNHCR is clear as to how planned objectives and outputs are going to be monitored. An example of such a logframe, based on a WFP model, is included in Chapter 3.

Standards and indicators

39. PCOS has drafted a core set of standards and indicators for use by UNHCR operations. The Core Set is intended to provide quantifiable and universal (rather than regionally specific) standards and indicators, whose consistent use will facilitate the enable global comparison of programme outcomes.

9 Chapter 3, ‘UNHCR’s monitoring cycle and the assistance sector’, examines UNHCR’s logical framework.

SYNTHESIS REPORT

7

40. In addition, PCOS is developing standards and indicators beyond the core set to support field monitoring. This is a very positive initiative which should be carried forward, with attention to the following principles.

Streamlining across sectors

41. In UNHCR, each assistance sector has guidelines that set out applicable standards and indicators for both planning and monitoring. An extensive analysis of the various UNHCR toolkits established for the different assistance sectors (health, education, water, sanitation, shelter, environment, community services) is provided in Chapter 3, and for the protection sector in Chapter 4.

42. As explained in those chapters, the standards and indicators used by UNHCR in different sectors are generally comprehensive and clear. However, their usefulness is compromised by variations in the way that indicators are defined, the unreasonably large number of indicators that staff are supposed to monitor in each sector, the lack of prioritization among indicators and the failure to mainstream protection indicators throughout each assistance sector.

43. At present, UNHCR’s main monitoring tools propose a total of 406 assistance indicators10 and over 175 protection indicators.11 These are in addition to those indicators found in the Core Set, those in the Global Appeal 2003, 12 and those in cross-cutting monitoring tools such as the draft set of indicators for monitoring the

10 From the sectoral guidelines for community services, education, health, food and nutrition, environment, and, for water, sanitation and shelter, from the ‘Practical Guide’ 11 ‘Designing protection strategies and measuring progress: checklist for UNHCR staff’, July 2002. 12 ‘UNHCR's global objectives and indicators of progress’, in ‘UNHCR Global Appeal 2003’, p. 8.

DRAFT SGBV (103)

Protection (175)

Food/Nutrition (61)

Core Set (50

Country Set

Environment (168)

Shelter (9)

Sanitation (17)Water (18)

Education (45)

Health Toolkit (59)

Comm. Serv. (29)

Other ?

MONITORING

8

implementation of the High Commissioner’s Five Commitments to Refugee Women (which alone total 103 indicators).13

44. Even with full staff capacity and programme resources, no UNHCR office could hope to collect, analyze and respond to information relating to such a large number of indicators. Finding a balance between the ideal amount of information needed and what is realistic to collect is thus a critical challenge.

45. Concerns are rightly expressed by UNHCR staff that limited sets of indicators such as the Core Set would not provide country offices with enough information to assess and revise programmes effectively. While the development of the Core Set is a laudable start, field based monitoring requires more than the Core Set.

Recommendations:

Sectoral specialists in DOS should streamline the standards and indicators for their own sector, prioritizing them in order to provide both a minimum and an optimum set of indicators.

DOS and DIP should ensure that protection concerns are integrated into assistance indicators and that such indicators are gender and age sensitive.

Consistency

46. Monitoring depends on reliable indicators to measure progress towards an agreed standard. While there is a demonstrable commitment within UNHCR to the use of such indicators, there is a lack of consistency in how indicators and standards are defined; the same variable is an indicator in one document and a standard or an objective in another.14

47. Inconsistency in the language used by UNHCR’s guidelines could weaken the integrity of comparison between programmes and projects, and increase staff workload by demanding different sets of monitoring information. It also adds to the genuine difficulties that most people have in understanding this approach to programme planning.

48. UNHCR is by no means alone in using the terminology in different ways or even interchangeably. However, its mandate and coordination role demand clarity, consistency and agreement on the language and definitions that are most likely to yield high-quality monitoring information.

49. UNHCR's reduced ‘market share’ in the humanitarian sector means that the organization has an increased need to obtain monitoring information from other actors. In this respect, it makes sense for UNHCR to align itself with global technical standards such as those of the Sphere Project, which are commonly used by other stakeholders.

13 The most recent draft of this set was provided by the Refugee Women and Gender Equality Unit 29 June 2003. 14 Examples and a detailed discussion are included in Chapter 3, ‘UNHCR's monitoring cycle and the assistance sector’.

SYNTHESIS REPORT

9

Recommendation:

PCOS should swiftly resolve UNHCR’s position on whether to support the Sphere Project standards, with or without formal endorsement, and streamline the language in its own guidelines so it can relate more effectively to partners who use Sphere terminology.

Integration of protection principles

50. At a policy level, UNHCR has embraced a broad, multifaceted and integrated view of protection which embraces the whole range of the organization’s activities. But in practice this approach has not been fully operationalized.

51. As various evaluations have pointed out, and as we have observed, UNHCR staff too frequently concentrate on their particular area of expertise, viewing issues of protection as being within the domain of protection officers and, to some extent, community services staff.15 Many protection officers also share this view. This situation contributes to an overly narrow view of protection responsibilities with the result that the protection dimensions of assistance activities are not readily recognized and monitored.

52. In addition, community services are under resourced and over-stretched. This fragmented approach has a direct bearing on monitoring the protection, rights and well-being of refugees, which, to be of value, requires a fully integrated protection-oriented perspective.16

53. DIP has produced a Protection Checklist which is essentially a management planning and implementation tool that highlights the protection concerns that must be addressed in UNHCR’s multi-sectoral day-to-day activities. Its focus is wide, extending not just to the work of the protection sector but also to the activities of other programme and technical sector staff.17 If it is to be used effectively, senior staff members require a knowledge of its content and must ensure that it is used as an integrated planning and measurement tool.

Recommendation:

UNHCR must ensure that protection objectives and activities be integrated into the operations planning and implementation process. To that end, DIP should prepare a revised and more user-friendly version of the Protection Checklist. The Checklist should be issued to senior managers by the High Commissioner, with an instruction that it be used in programme planning, implementation and monitoring.

15 See, for example the following evaluations available on www.unhcr.org: ‘Meeting the rights and protection needs of refugee children: An independent evaluation of the impact of UNHCR’s activities’, (EPAU/2002/02), May 2002; ‘A beneficiary-based evaluation of UNHCR’s programme in Guinea, West Africa’ (EPAU/2001/02) January 2001; ‘UNHCR policy on refugee women and guidelines on their protection: an assessment of ten years of implementation, May 2002. 16 See Chapter 4, ‘Monitoring in the protection sector’. 17 For a detailed discussion of the Protection Checklist see Chapter 4, ‘Monitoring in the protection sector’.

MONITORING

10

Involving beneficiaries

54. UNHCR’s programmes have not always built upon the capacities of refugees themselves or engaged refugees consistently in programme design, delivery and monitoring. As a result, gaps in protection have developed and a climate of dependency has been engendered, with significant psychological, social and economic costs.

55. UNHCR’s recent emphasis on community development approaches, in which the organization recognizes the role of refugees as resourceful partners, should lay the foundation for a new focus on participatory forms of monitoring. In this respect, it should be noted that participatory monitoring (and evaluation) is undermined, if not impossible, if refugees have not fully participated in the assessment and design of the programme being monitored.

Recommendation:

DOS should ensure that existing planning tools are revised to provide guidance on how to involve beneficiaries in programme planning, design and monitoring. The degree to which full participation is feasible will depend on the context and the capacity of implementing partners. Nonetheless, greater consideration could be given to participatory monitoring as a means of generating ownership, increasing the self-esteem of the refugees, sharing responsibility and reducing costs.

Performance, beneficiary-based and context monitoring

56. As mentioned earlier, UNHCR’s mandate renders it responsible for various different types of monitoring including input, output, process and impact monitoring as well as tracking changes in the external environment. Current monitoring practices, however, tend to emphasize quantitative data concerning inputs and outputs. Rarely do they incorporate qualitative monitoring, direct feedback from beneficiaries or an analysis of changes in the external environment.

57. With respect to process, UNHCR has planning tools such as Situation Analysis and People Oriented Planning (POP), which are intended to derive credible information from and about refugees, as well as their needs, problems and capacities. However, current monitoring guidelines do not include process indicators that measure refugee participation and perceptions.

Recommendation:

DOS should ensure that each set of sectoral monitoring indicators contains at least one process indicator in its core set.

58. Ideally, UNHCR and its partners should focus on monitoring the impact of their programmes, both intended and unintended. Impact monitoring most closely addresses the question ‘how are UNHCR’s activities affecting the protection and well-being status of the refugees?’

Recommendation:

SYNTHESIS REPORT

11

PCOS should ensure that the Core Set uses impact, rather than performance indicators, as far as possible. These should be reported on in the Annual Protection Report and Country Report and be part of the data from which the Country Operations Plans (COPs) are developed.18

Participatory and team-based planning

59. Within a country operation, each programme is designed according to a set of objectives, which are formulated at the sectoral level. Given that all sectoral interventions have a protection dimension, effective monitoring requires coordinated action by programme, technical, protection, community services, and other field staff.19

60. Built into a jointly planned programme cycle, monitoring and the sharing of monitoring data and which enables staff to understand how activities in one sector affect other sectors. For this to happen, managers must be held responsible for supporting the team-based approach and for ensuring that those individuals with monitoring responsibilities are held accountable for them. The following paragraphs examine the means whereby UNHCR could ensure greater accountability for the establishment of fully integrated monitoring systems at country level.

The role of senior managers

61. UNHCR Representatives have two key responsibilities in relation to monitoring. One is to communicate UNHCR policies and priorities and to coordinate with all relevant external parties. This requires the regular collection of monitoring information with regard to the operational activities of other actors, including their roles and resources, so as to identifying any gaps in provision and planning.

62. The other responsibility is to ensure the effective monitoring of programme results and impact, including compliance with UNHCR policies and procedures. We are of the view that the monitoring responsibilities of Representatives should be reinforced and supported by appropriate delegation to other senior managers.

Recommendations:

Representatives should be required to ensure that a team-based approach to monitoring takes place, so as to generate an integrated view of the protection, rights and well-being of refugees.

In order to obtain such an integrated view, Representatives and Heads of Office should also encourage UNHCR team members to undertake joint monitoring visits, as well as joint meetings with implementing and operational partners.

18 See Chapter 3, ‘UNHCR's monitoring cycle and the assistance sector’. 19 See Chapter 2, ‘The protection framework’.

MONITORING

12

To guarantee adequate UNHCR presence in the field, Representatives should monitor and record the amount of time that senior staff spend in field and have direct contact with refugees.

Deputy (or, where appropriate) Assistant Representatives should be accountable for the effective implementation of field and country-level monitoring systems, and ensure that all staff know their respective monitoring responsibilities. This should involve monthly team meetings (or more frequently, depending on the operation) that draw together programme, protection, community services staff and sectoral specialists. Such meetings should be used to review the information collected against agreed indicators; to assess the implications of this data for the protection, rights and well-being of refugees; and, in the light of these findings, to make any necessary adjustments to plans, budgets and activities. In countries where there is no Deputy or Assistant Representative, this role should fall to either the Senior Programme Officer or Senior Protection Officer.

In each major UNHCR operation, one or more staff members should be dedicated to information management, so as to ensure that monitoring information is properly collated and summarized. In many operations there is already an equivalent role which can be adapted if necessary to take on the function of data quality control. This role can be filled by a competent national staff member.

Joint strategic assessment and planning exercises

63. UNHCR has had some positive experiences in joint strategic planning exercises with other agencies. An example is to be found in last year’s planning exercise in Kenya, where UNHCR, NGOs and other partners jointly designed one country plan on the basis of beneficiary needs and another on the basis of available resources.

64. The exercise revealed the gap between what was needed to be done and what could be done with the resources at hand. The exercise also helped in the coordination of UNHCR and NGO field activities and enabled the participating organizations to arrive at a common monitoring strategy.20

Recommendation:

UNHCR Representatives should examine recent examples of UNHCR-led planning exercises that include UN, government and NGO partners, with a view to gathering best practice on joint planning

Monitoring beyond the programme

65. UNHCR's monitoring tends to focus on the performance of programme interventions rather than on monitoring the context in which they take place or monitoring the situation of refugees who are not assisted by the organization. These

20 The Kenya exercise is specifically acclaimed in the NGO Statement on Programme and Funding made to UNHCR’s Standing Committee, 24-26 June 2003.

SYNTHESIS REPORT

13

include urban refugees, refugees living with host families, as well as those who are difficult to access or who are living in geographically dispersed locations.

66. Given the challenges already faced by the agency in monitoring the situation of accessible and assisted caseloads, the apparent absence of monitoring for unassisted refugees is understandable. Intensive monitoring for unassisted or low-risk caseloads is unrealistic and would withhold resources from priority caseloads. However, such groups may nevertheless experience protection problems or their situation may deteriorate over time, thereby necessitating UNHCR’s intervention:

Recommendation:

UNHCR should engage in targeted occasional monitoring of unassisted refugees, in coordination with humanitarian and development agencies.21

Collecting monitoring information

67. As noted earlier, we recommend that the means of verification (MOV) in UNHCR’s logframe be reinserted to ensure that planning accounts for how monitoring information will be collected. Several, though not all, of UNHCR’s sectoral guidelines include template forms for data collection.

68. Feedback from the field suggests that even where such formats exist, staff are either not aware of them, or are motivated regardless to create their own. Contextual differences and individual preferences lend themselves to adapted or innovative models. Standard template forms, however, are useful to enable analysis and comparison across the organization’s programmes. This suggests a need to find a balance between providing and promoting templates while supporting appropriate adaptation to context.

Recommendation:

Template data collection forms setting out minimal core indicators should be developed to enable analysis and comparison across the organization’s programmes, while permitting additions according to specific country programme needs.22

Database systems

69. Over the past decade, UNHCR has developed the capacity to collect and publish disaggregated data on refugee populations and other people of concern. By contrast, UNHCR has no such process for consolidating either performance or impact data.

70. Work on corporate PeopleSoft-based systems and on a field-based database protection and programme management system (PPMS) was suspended in 2001. The PeopleSoft development restarted in 2002 under the name of the Management

21 Ibid, p.19. 22 Ibid, p. 22.

MONITORING

14

Systems Renewal Project (MRSP), which is focused on supply chain and financial data.

71. There is not sufficient overlap between the planned implementation of MSRP and the need for a field-based information system to support monitoring. For monitoring capacity to be enhanced, common systems will need to be installed in all field offices, supported by technical backup and training.

Recommendations:

DOS, working with Information and Technology Development Service, should lead the development of a database system for field offices which provides for the capture of the information from UNHCR logical frameworks in planning documents, and allows for qualitative and quantitative information to be entered against them on a monthly basis.

The design of such a system should take account of current country-based information systems initiatives and previous design work on the PPMS.23 The system should also be compatible with future operations aspects of the MSRP.

Initially the system should not attempt to link budget to objectives but should concentrate on the capture of performance and impact data.

The system should be tested in three offices over the next 18 months.

Monitoring guide

72. For a detailed analysis of the many monitoring-related guides in technical sectors and for protection-related activities, see Chapters 3 and 4 respectively. As indicated in those papers, while UNHCR has a variety of specialized guides, it does not have a generic monitoring guide that includes cross-sectoral tools (such as participatory, qualitative and quantitative survey techniques) or a consolidated set of minimum monitoring requirements and indicators for each sector. We see value in developing such a guide, which would be targeted at generalist field staff, but would also be of use to specialists. More detailed guides would be retained and used by technical specialists for more in-depth monitoring.

Recommendations:

DIP and DOS should jointly create a generic monitoring guide, under the supervision of a senior manager. The guide should incorporate a set of minimum standards and indicators for each sector, drawn from the monitoring-related sections of all UNHCR’s existing manuals, checklists, guidelines and training materials.

The guide should cover the objectives of monitoring, the monitoring of protection across all sectors (including gender and children’s concerns), as

23 These are covered in more detail in Chapter 5, ‘Monitoring and the UNHCR management environment’.

SYNTHESIS REPORT

15

well as a selection of monitoring techniques for the collection of qualitative and quantitative data.

In establishing such a guide, UNHCR should make use of the comprehensive monitoring and evaluation guides available on CD-Rom from UNICEF and WFP.

The guide should incorporate the planned PCOS ‘Field Guide to Standards’ the Situation Analysis tools currently being developed by DOS and the monitoring section of the planned Operational Protection Toolkit.

The guide should include selected priority indicators drawn from existing monitoring checklists.

The guide should be field tested in at least three locations before wider distribution and training.

The guide should include guidance on how to analyse and communicate qualitative information and become an integral part of UNHCR’s programme management tools. 24

Analysing monitoring information

73. Monitoring is not simply a question of collecting data. If the data collected is not properly analyzed, the story told by the data will not be understood or acted upon. For example, monitoring data may reveal high malnutrition rates, even in situations where a full food ration appears to be provided. Malnutrition in such situations could be caused by the provision of inappropriate food, corruption in the food distribution system, or the unavailability of cooking fuel and may point to protection concerns such as sexual exploitation.

Recommendation:

UNHCR Representatives should ensure that responsibilities for the analysis of data are clarified. Relevant staff should be given the training, support and time needed to undertake such analysis and to identify significant trends and anomalies.

Reporting

74. The narrative report remains one of the key methods of formal communication in UNHCR. There are currently some 13 different types of standard report, as well as less formal methods of communication.25

75. Many staff members interviewed complained that fulfilling these regular reporting demands paled to insignificance when compared to the need to file ad hoc reports on subjects, sectors or issues of special interest to the organization and its donors, such as the environment, disabled refugees, women, gender equality,

24 Chapter 3, ‘UNHCR's monitoring cycle and the assistance sector’. 25 Ibid.

MONITORING

16

reproductive health and HIV/AIDS. The inclusion of these issues into regular reporting formats would reduce the burden placed on staff and also enable more meaningful analysis

76. Our review of UNHCR’s reporting documentation also revealed a number of other weaknesses: absence of documented assessment (either situation or needs assessment), or baseline data; confusion between objectives and indicators; imprecise indicators and indicators that are not consistent with sectoral guidelines; vague or unstated means of verification; progress or results reported, but not in comparison to plans; and positive impact claimed without supporting evidence.26

77. We also observed that Annual Protection Reports (APRs) do not consistently highlight protection issues arising in the assistance sectors and that large sections are repeated from previous reports. We propose that routine reporting be modified as follows below.

Recommendations:

The monthly Situation Report (SitRep) should include tables drawn up from each sector’s logical frameworks, with indicators including but going beyond the core standards. SitReps should show month on month progress, rather than provide a monthly snapshot. Narrative reporting should be reduced to a two-side exception report, highlighting only key changes in context and constraints. This approach should be applied at the camp, sub-office and country level. Weekly SitReps will be required for emergency or other fast-moving situations.

UNHCR’s regular reporting cycle should be as follows: (i) the weekly collection of base performance data at the point of delivery; (ii) a monthly Sit Rep collated from data collected in the past four weeks and plotted against previous months; quarterly or half yearly (for regional bureaux to determine for each operation), with a commentary on results achieved.

End of the year reporting should be streamlined as follows:

Stage 1. An internal protection report is written encompassing all areas covered by the APR and Country Report (CR) but with one integrated assessment of protection issues. Creating such an integrated analysis of changes in context, performance, and impact will require a team effort. Use should be made of an adapted logframe with standards and indicators, so as to enhance the ability of UNHCR offices to identify protection gaps and progress, and to integrate the information contained in the protection report with the next Country Operations Plan (COP).

Stage 2. A summary public report is written, based on the internal report but with confidential material removed. This document becomes the country input to the UNHCR Global Report.

26 See also Chapter 5, ‘Monitoring and the UNHCR management environment’, and Chapter 4, ‘Monitoring in the protection sector’.

SYNTHESIS REPORT

17

All static background information at present found in APRs and COPs should go into a separate document that does not have to be rewritten every year, leaving the main reports to focus on significant developments during the year.

All regular reports (including SitReps) should concentrate on impact and changes in the protection, rights and well-being of refugees, with systematic comparison with the objectives set out in the COP.

UNHCR staff capacity

78. Since 1998, UNHCR has operated with an average 11 per cent shortfall in its regular budget. The shortfall, combined with an upward-moving staff cost structure, has created a pressure to cut posts. Cuts in UNHCR staffing levels evidently have an impact on the organization’s capacity to undertake effective monitoring.

National staff

79. The majority of UNHCR staff who are engaged in monitoring are not international staff but national staff. It is their verbal and written feedback that provides the basic building block of UNHCR monitoring.

Recommendation:

UNHCR Representatives should ensure that national staff receive induction and on-the-job training to ensure that they can perform monitoring activities related to their respective function (programme, protection, community services, etc).

Protection staff

80. A frequently cited reason for insufficient protection monitoring at the field level is that many operations experience a shortage of protection staff. Such shortages have had serious consequences in some operations, where UNHCR has not been aware of persistent human rights violations.27 In response to this situation, DIP has started to establish benchmarks, so as to determine the number of protection staff needed in a particular situation, as well as the grade and function of such staff. EPAU is also planning to undertake a comprehensive review of UNHCR’s protection staffing capacity in 2004.

Recommendation:

We support the development of benchmarks for determining protection staff requirements, as well as the introduction of more flexibility into the current protection qualifications profile.

27 Chapter 4 ‘Monitoring in the protection sector’.

MONITORING

18

Community services staff

81. Community services staff also are in short supply, as a recent evaluation of the function has documented.28 In its response to the community services evaluation, UNHCR has said it is committed to ensuring that “positions of community services staff are maintained and that, as far as possible, community services posts in key operations are reinstated over the coming years.”29

82. To strengthen a joint approach to the protection needs of refugee women and children, UNHCR has also said that it will promote the establishment of multi-sectoral teams in all Branch Offices and Field Offices, as a replacement for the more traditional approach of appointing focal points for refugee women and children. This is a positive development, in line with our recommendation that UNHCR adopts an integrated team approach for all aspects of programme assessment, planning, implementation and monitoring.

Technical staff

83. Sectoral staff in Geneva repeatedly point out that generalists cannot monitor the quality of a technical programme in a meaningful manner. As each sector has only a handful of specialists in the field,30 overworked generalist field officers and their assistants are most often responsible for monitoring. As a result, only basic output-orientated data collection is possible on a regular basis. 31

Recommendation:

DHRM and DOS should develop benchmarks for determining the number and kind of technical staff required for particular programmes or caseloads, recognizing that smaller country programmes will not be able to have a full complement of technical specialists.

Staff development

84. UNHCR training budgets peaked in the late 1990s but have declined markedly since 2000. The improvement of UNHCR’s monitoring capacity will require further training, and consequently increased investment.

Induction and orientation

85. Surprisingly, there is currently no requirement that new staff be provided with any induction and orientation with regard to the organization’s protection mandate before commencing work in the field. By contrast, the International Committee of the Red Cross (ICRC) provides professional staff with a four week introductory course in ICRC’s mandate and its results-based programming system. 28 ‘The community services function in UNHCR: an independent evaluation’, March 2003 29 ‘UNHCR Response to the three evaluations/assessments of refugee women, children and the community services function’, June 2003, p. 8. 30 See Chapter 3, ‘UNHCR’s monitoring cycle and the assistance Sector.’ 31 For example, the number of latrines constructed, rather than whether they were appropriately designed or whether they have positively impacted on refugee health.

SYNTHESIS REPORT

19

Recommendations:

Induction should be mandatory for all new UNHCR staff, and induction materials should be revised to provide at least a basic understanding of protection issues.

Senior protection officers or regional training officers should conduct protection training for all newly recruited protection staff on a yearly or biannual basis.32

Learning programmes

86. Since 1999, UNHCR has implemented four core Learning Programmes covering Protection, Operations, Senior and Middle Management. Other learning programmes and training initiatives relevant to monitoring include Programme Management and Disaster Management workshops, Protection Management Workshops and Protection Strategies in Areas Affected by Armed Conflict.33 The learning programme approach is popular.

Protection Learning Programme (PLP)

87. The PLP materials are of high quality and if used more widely would contribute to enhancing the monitoring function.

Recommendations:

We support DIP’s plans to incorporate monitoring issues into almost all units of the revised PLP, and suggest that the monitoring guidance provided in those units be consistent with other initiatives under consideration, such as the proposed Operations Protection Toolkit, the planned PCOS Field Guide to Standards and the generic monitoring guide proposed by this review. 34

The Protection Learning Programme should be extended as soon as possible to cover all protection staff. New protection staff should commence the PLP within one year of posting.

We encourage the much wider use of the PLP to raise the understanding of protection and protection monitoring amongst all field-based staff, not just protection or community services staff.

As part of this strategy, we suggest that the feasibility of raising funds for a special project dedicated to the delivery of the PLP be examined with the aim of training all front-line staff over a five-year period.

32 Chapter 4, ’Monitoring in the Protection Sector’. 33 Protection-related training initiatives are discussed further in Chapter 4, ‘Monitoring in the protection sector’ and those relating to management in Chapter 5, ‘Monitoring and the UNHCR management environment’. 34 Chapter 4, ’Monitoring in the Protection Sector’.

MONITORING

20

Protection Management Workshop (PMW)

88. From a monitoring perspective, the development of the PMW should help to enhance management accountability for effective monitoring, a current weakness in any operations.

We recommend that the proposed monitoring tools, once developed, be referred to in the PMW, especially the generic monitoring guide proposed in this review. In the absence of such a guide, the Situation Analysis tool being developed by DOS and the Field Guide to Standards proposed by PCOS should be referred to in the PMW.

Operations Management Learning Programme OMLP)

89. The application of logframes to humanitarian projects is not easy and on-going training is required. OMLP has contributed to a concerted UNHCR effort to provide training on the logframe planning format. This, in turn, provides an important basis for the introduction of results-based management and an effective monitoring system.

Recommendations:

PCOS should develop a network of officers outside Headquarters who are able to provide on-the-job coaching to field staff in the use of logical frameworks and specific, measurable indicators.35 This training-of-trainers approach could be used to support those in the OMLP, and others who are not involved in the programme.

PCOS should update the OMLP and other forms of programme management training to include a greater, practical coverage of monitoring, building on the contents of the Generic Monitoring Guide proposed in this review. The OMPLP should also provide clearer guidance on the link between protection and assistance activities, including specific references to the Protection Checklist and, examples of best practices with regard to integrated planning, teamwork and beneficiary participation.36

Senior and Middle Management Learning Programmes (SMLP and MMLP)

90. Monitoring is largely absent from the current Senior and Middle Management Learning Programmes. Those programmes are currently being redesigned, providing an important opportunity for this situation to be rectified.

Recommendation:

The issue of monitoring should be incorporated into the SMLP and MMLP. Staff involved in these programmes should learn how to maintain UNHCR’s monitoring system and how to make effective use of the information it provides.

35 Chapter 5, ‘Monitoring and the UNHCR management environment’. 36 Ibid.

SYNTHESIS REPORT

21

Partners and monitoring

91. Roughly a third of UNHCR’s Annual Budget goes to the more than 700 NGOs, governmental and intergovernmental agencies that act as its implementing partners. As a result, they are the main source of monitoring information for UNHCR. They are also contractual partners for whose quality of work UNHCR is directly accountable. UNHCR must therefore ensure access to their monitoring information whilst monitoring their performance at the same time.

92. While IPs continue to be pivotal to UNHCR's monitoring capacity, the extent to which they rely on UNHCR funding, and hence the organization’s formal influence, continues to decline. Operational partners implement programmes without UNHCR funds and are not contractually obliged to report to UNHCR. Although UNHCR has a number of Memoranda of Understanding that stipulate reporting responsibilities with, for example UNFPA, UNICEF, WFP and WHO, there are no formal mechanisms for the exchange of monitoring information within the UN system.

93. UNHCR’s monitoring capacity is also weakened by its failure to work in a fully collaborative manner with both implementing and operational partners. Among the common criticisms of UNHCR are that it fails to consult with its partners, that it is willing to take information from partners but provides little in return, and that it does not engage its partners very actively in programme design or evaluation.37

94. UNHCR should offer something in return for the monitoring information provided by partners such as sharing a composite picture of the beneficiary population drawing from partner monitoring information, UNHCR context monitoring and beneficiary feedback as well as providing leadership on globally accepted standards.

Recommendations:

95. UNHCR Representatives should involve partner organizations in all strategic planning exercises and engage with the Common Country Assessment, the UN Development Assistance Framework and the Consolidated Appeal Process. Such exercises should include the development of appropriate joint monitoring mechanisms.

UNHCR’s Donor Relations section should encourage donors to ensure that NGOs which receive bilateral funding are required to coordinate with UNHCR and to provide the organization with monitoring information.38

Given the amount of monitoring undertaken by partners, UNHCR should, where possible and appropriate, harmonize its monitoring tools and data collection with them.

37 Noted in ‘A beneficiary-based evaluation of UNHCR’s programme in Guinea, West Africa’ and ‘UNHCR Policy on Refugee Women and Guidelines on Their Protection: An Assessment of Ten Years of Implementation’ above note 17. This was also raised by staff in interviews at Headquarters and in the field. 38 OFDA already does this, stating that “OFDA will not fund organizations that do not share programmatic data and information with appropriate humanitarian information coordination bodies in the field’, USAID/OFDA 2002 guidelines.

MONITORING

22

UNHCR should concentrate on enhancing its particular value-added by creating monitoring ‘additionality’: a strategic monitoring role that will enable UNHCR field staff to complement partner's monitoring with periodic beneficiary-based monitoring and situation monitoring Emphasise UNHCR's role as the "standard-bearer" of humanitarian standards and norms for refugees and people of concern

Enhancing partners’ monitoring capacity

96. The need for UNHCR to coordinate more effectively with partners, to provide clear guidance to them on the monitoring information needed by the organization was frequently raised with us in the course of this evaluation. It was also noted in the Inspector General’s 2002 report.39

97. Some implementing partners will need more support and direction in meeting UNHCR’s monitoring needs and requirements. It is consequently important for UNHCR to assess their monitoring capacity and to identify those that need additional guidance and support. Such assessments should cover the IP's staff skills, the monitoring systems which it has in place, its own institutional reporting requirements and accountability mechanisms.

Recommendations:

We strongly endorse DIP’s current attempt to document and consolidate operationally-oriented training initiatives in the field for UNHCR staff and partners, and recommend that UNHCR develops a policy for the targeted training of IPs.

UNHCR staff should be provided with training and checklists in organizational assessment, so that they identify those IPs whose monitoring capacity needs to be strengthened.

Monitoring of partners

98. The Sub-Project Monitoring Report (SPMR) submitted by UNHCR’s implementing partners is intended to “reflect the proper use of inputs and to compare actual achievements against the planned outputs.” The narrative element of the SPMR is typically produced by IPs twice a year, and a financial report four times a year. It is the primary source of written monitoring information about the sub project-level activities of implementing partners.

99. UNHCR staff frequently commented during this evaluation that the SPMRs were viewed above all as a financial and contractual tool, rather than a programme quality monitoring tool. It was also pointed out that UNHCR officers and many IP field staff were often not familiar with the objectives and outputs included in sub-agreements.

39 ‘Observations from IGO missions: implementation and proposals for policy considerations, March 1999 to March 2002’.

SYNTHESIS REPORT

23

100. We are of the view that the SPMR process is too infrequent and cumbersome to be effective as a monitoring tool. For many partners, their own internal reporting and performance management mechanisms already demand higher-frequency reporting than UNHCR.

Recommendations:

Partners should be reporting to UNHCR on a monthly basis, using a version of the sub-project logframe to indicate progress against planned outputs (and impact where possible), accompanied by a brief analytical narrative of no more than one page. This would not obviate the need for the SPMR but would generate more timely feedback and hence increase responsiveness.

Officers in charge of monitoring sub agreements should make fuller use of the work plans accompanying them (currently optional), so as to provide better planning and tracking of progress over the year.

101. IPs interviewed raised a number of other problems with regard to UNHCR’s monitoring practices: (a) staff with whom project plans were negotiated were not authorized to amend them, making it difficult to respond in a timely manner to monitoring data; (b) sub-agreements are circulated only in printed form, rather than electronically, so they may not reach partner staff in the field (and sometimes UNHCR staff) and are more difficult to amend; (c) UNHCR demands an excessive number of ad hoc reports, and (d) the work of IPs is often monitored by UNHCR staff who lack appropriate technical experience.

Recommendations:

The electronic rather than paper exchange of contracts should be encouraged so as to ensure that UNHCR and partner staff are aware of agreed objectives.

UNHCR staff who are responsible for the day-to-day management of sub-agreements with IPs should be given increased authority to amend such agreements (within the total budget and against agreed criteria) in accordance with changes in context.

102. Poor financial follow-up of SPMRs continues to be an issue for UNHCR. This problem applies to some of UNHCR’s largest international implementing partners as well as smaller, national partners. UNHCR’s capacity to carry out financial monitoring is likely to decline further, as the Division of Financial and Supply Management has lost 50 per cent of its posts, and there will be no central capacity to provide training for field finance staff from 2004.

Recommendation:

UNHCR should target support to weaker partners and reduce the surveillance of more capable, trusted partners correspondingly. The organization should provide financial training to its weakest implementing partners, as identified by Branch Office assessments and by the UNHCR auditors. As the organization’s financial training capacity has been dismantled, this may have to be contracted in.

MONITORING

24

Managing monitoring initiatives

103. Throughout our report we have made recommendations that will require coordinated implementation and management oversight. In this regard we believe that specific attention should be paid to four basic principles.

104. First, initiatives should only be introduced if there is the field capacity to absorb them.

Recommendations:

New guidelines should be introduced only after they have been field tested. Compliance can only be required from field offices when training and other support has been provided.

New computer systems should be introduced only when they are accompanied by training and a system of on-going advice and technical support.

The end use to which monitoring information is put should be properly explained to UNHCR staff before any new monitoring systems or requirements are established.

105. Second, accountability for collecting monitoring information and acting upon it must be clarified. Currently, accountability in UNHCR has tended to focus on financial matters and there is little practical guidance available to managers on accountability for results or impact. Without stronger and results-oriented accountability, especially for Representatives, it will be difficult to enhance UNHCR’s monitoring capacity.

106. Third, UNHCR’s quality improvement initiatives should be prioritized; it is not possible to make improvements on all fronts at the same time. The development and implementation of an effective monitoring system is not a trivial task.

107. UNHCR should set a realistic timeframe to devise and implement a new monitoring system, with appropriate resources and personnel. A first priority must be to build the capacity to collect, interpret and act upon data in the field, and only later to develop the capacity to bring this information together at Headquarters for the purpose of making global comparisons and mapping trends.40

Recommendations:

A three year timeframe should be established for the development of UNHCR’s monitoring system, with field-level monitoring as the priority.

Donors should be asked to decide jointly what their expectations are with respect to the monitoring information they require from UNHCR. Donors

40 The lack of centrally available management information is well understood. “UNHCR, as an organisation, is suffering from an inability to extract data and build meaningful reports in a timely and efficient manner, both for internal and external use”, from the MSRP update 2003, see http://intranet.hcrnet.ch/support/msrp.

SYNTHESIS REPORT

25

should support the development of new field-based monitoring systems with sponsorship (funds, technical personnel, etc) for the three-year period.41

The Executive Office should assign responsibility for the development and maintenance of a universally applicable and straightforward monitoring system in field operations to a senior manager, who would coordinate and direct the implementation of all quality improvement initiatives. This senior manager should also be required to ensure that all new training initiatives, instructions, guidelines, manuals and computer systems are within the capacity of field offices to absorb. The manager who is assigned the responsibility will evidently need to work closely with DIP and UNHCR’s regional bureaux.

41 Chapter 5, ‘Monitoring and the UNHCR management environment’.

27

Appendix 1. Checklist of recommendations

A. Recommendations on monitoring and programming

1) DOS should retain the simplicity of its two-row logframe design but should re-

insert the columns showing the means of verification and indicators for assumptions and risks so that UNHCR is clear as to how planned objectives and outputs are going to be monitored. An example of such a logframe, based on a WFP model, is included in Chapter 3.

2) UNHCR must ensure that protection objectives and activities are integrated into the operations planning and implementation process. To that end, DIP should prepare a revised and more user-friendly version of the Protection Checklist. The Checklist should be issued to senior managers by the High Commissioner, with an instruction that it is used in programme planning, implementation and monitoring.

3) Representatives should be required to ensure that a team-based approach to monitoring takes place, so as to generate an integrated view of the protection, rights and well-being of refugees.

a) In order to obtain such an integrated view, Representatives and Heads of Office should also encourage UNHCR team members to undertake joint monitoring visits, as well as joint meetings with implementing and operational partners.

b) Monthly team meetings (or more frequently, depending on the operation) should draw together programme, protection, community services staff and sectoral specialists.

c) Such meetings should be used to review the information collected against agreed indicators; to assess the implications of this data for the protection, rights and well-being of refugees; and, in the light of these findings, to make any necessary adjustments to plans, budgets and activities.

4) To guarantee adequate UNHCR presence in the field, Representatives should

monitor and record the amount of time that senior staff spend in field and have direct contact with refugees.

5) Deputy (or, where appropriate) Assistant Representatives should be accountable for the effective implementation of field and country-level monitoring systems, and ensure that all staff know their respective monitoring responsibilities. In countries where there is no Deputy or Assistant Representative, this role should fall to either the Senior Programme Officer or Senior Protection Officer.

6) In each major UNHCR operation, one or more staff members should be dedicated to information management, so as to ensure that monitoring information is properly collated and summarized. In many operations there is already an equivalent role which can be adapted if necessary to take on the

MONITORING

28

function of data quality control. This role can be filled by a competent national staff member.

7) UNHCR Representatives should examine recent examples of UNHCR-led planning exercises that include UN, government and NGO partners, with a view to gathering best practice on joint planning

8) DOS should ensure that existing planning tools are revised to provide guidance on how to involve beneficiaries in programme planning, design and monitoring. The degree to which full participation is feasible will be dependent on the context and the capacity of implementing partners. Nonetheless, greater consideration could be given to participatory monitoring as a means of generating ownership, increasing the self-esteem of the refugees, sharing responsibility and reducing costs.

9) PCOS should swiftly resolve UNHCR’s position on whether to support the Sphere Project standards, with or without formal endorsement, and streamline the language in its own guidelines so it can relate more effectively to partners who use Sphere terminology.

10) Sectoral specialists in DOS should streamline the standards and indicators for their own sector, prioritizing them in order to provide both a minimum and an optimum set of indicators.

a) DOS and DIP should ensure that protection concerns are integrated into assistance indicators and that such indicators are gender and age sensitive.

b) DOS should ensure that each set of sectoral monitoring indicators contains at least one process indicator in its core set.

c) PCOS should ensure that the Core Set uses impact, rather than performance indicators, as far as possible. These should be reported on in the Annual Protection Report and Country Report and be part of the data from which the Country Operations Plans (COPs) are developed.

11) UNHCR should engage in targeted and occasional monitoring of unassisted

refugees, in coordination with humanitarian and development agencies.

B. Recommendations on the collection of monitoring information

12) Template data collection forms setting out minimal core indicators should be developed to enable analysis and comparison across the organization’s programmes, while permitting additions according to specific country programme needs.

13) DOS, working with the Information and Technology Development Service, should lead the development of a database system for field offices which provides for the capture of the information from UNHCR logical frameworks in planning documents, and allows for qualitative and quantitative information to be entered against them on a monthly basis.

SYNTHESIS REPORT

29

a) The design of such a system should take account of current country-based information systems initiatives and previous design work on the PPMS.42 The system should also be compatible with future operations aspects of the MSRP.

b) Initially the system should not attempt to link budget to objectives but should concentrate on the capture of performance and impact data.

c) The system should be tested in three offices over the next 18 months. 14) DIP and DOS should jointly create a generic monitoring guide, under the

supervision of a senior manager. The guide should incorporate a set of minimum standards and indicators for each sector, drawn from the monitoring-related sections of all UNHCR’s existing manuals, checklists, guidelines and training materials.

• The guide should cover the objectives of monitoring, the monitoring of protection across all sectors (including gender and children’s concerns), as well as a selection of monitoring techniques for the collection of qualitative and quantitative data.

• In establishing such a guide, UNHCR should make use of the comprehensive monitoring and evaluation guides available on CD-Rom from UNICEF and WFP.

• The guide should incorporate the planned PCOS ‘Field Guide to Standards’ the Situation Analysis tools currently being developed by DOS and the monitoring section of the planned Operational Protection Toolkit.

• The guide should include selected priority indicators drawn from existing monitoring checklists.

• The guide should be field tested in at least three locations before wider distribution and training.

• The guide should include guidance on how to analyse and communicate qualitative information and become an integral part of UNHCR’s programme management tools.

15) UNHCR Representatives should ensure that responsibilities for the analysis of

data are clarified. Relevant staff should be given the training, support and time needed to undertake such analysis and to identify significant trends and anomalies.

16) SitReps should show month-on-month progress, rather than provide a monthly snapshot.

a) Narrative reporting should be reduced to a two-side exception report, highlighting only key changes in context and constraints. This approach

42 These are covered in more detail in Chapter 5, ‘Monitoring and the UNHCR management environment’.

MONITORING

30

should be applied at the camp, sub-office and country level. Weekly SitReps will be required for emergency or other fast-moving situations.

b) The monthly SitRep should include tables drawn up from each sector’s logical frameworks, with indicators including but going beyond the core standards.

17) UNHCR’s regular reporting cycle should be as follows: (i) the weekly collection

of base performance data at the point of delivery; (ii) a monthly Sit Rep collated from data collected in the past four weeks and plotted against previous months; quarterly or half yearly (for regional bureaux to determine for each operation), with a commentary on results achieved.

18) End of the year reporting should be streamlined as follows:

a) Stage 1. An internal protection report is written encompassing all areas covered by the APR and Country Report (CR) but with one integrated assessment of protection issues. Creating such an integrated analysis of changes in context, performance, and impact will require a team effort. Use should be made of an adapted logframe with standards and indicators, so as to enhance the ability of UNHCR offices to identify protection gaps and progress, and to integrate the information contained in the protection report with the next Country Operations Plan (COP).

b) Stage 2. A summary public report is written, based on the internal report but with confidential material removed. This document becomes the country input to the UNHCR Global Report.

c) All static background information at present found in APRs and COPs should go into a separate document that does not have to be rewritten every year, leaving the main reports to focus on significant developments during the year.

19) All regular reports (including SitReps) should concentrate on impact and

changes in the protection, rights and well-being of refugees, with systematic comparison with the objectives set out in the COP.

C. Recommendations on staff capacity

20) UNHCR Representatives should make sure that national staff receive induction and on-the-job training to ensure that they can perform monitoring activities related to their respective function (programme, protection, community services, etc).

21) The review supports the development of benchmarks for determining protection staff requirements, as well as the introduction of more flexibility into the current protection qualifications profile. DHRM and DOS should develop benchmarks for determining the number and kind of technical staff required for particular programmes or caseloads, recognizing that smaller country programmes will not be able to have a full complement of technical specialists.

SYNTHESIS REPORT

31

D. Recommendations on staff development

22) Induction should be mandatory for all new UNHCR staff, and induction materials should be revised to provide at least a basic understanding of protection issues.

23) Senior protection officers or regional training officers should conduct protection training for all newly recruited protection staff on a yearly or biannual basis.43

24) The review supports DIP’s plans to incorporate monitoring issues into almost all units of the revised PLP, and suggest that the monitoring guidance provided in those units be consistent with other initiatives under consideration, such as the proposed Operations Protection Toolkit, the planned PCOS Field Guide to Standards and the generic monitoring guide proposed by this review.

25) The Protection Learning Programme should be used more extensively:

a) New protection staff should commence the PLP within one year of posting.

b) The review encourages the much wider use of the PLP to raise the understanding of protection and protection monitoring amongst all field-based staff, not just protection or community services staff.

c) As part of this strategy, the review suggests that the feasibility of raising funds for a special project dedicated to the delivery of the PLP be examined with the aim of training all front-line staff over a five-year period.

26) The review recommends that the proposed monitoring tools, once developed,

be referred to in the PMW, especially the generic monitoring guide proposed in this review. In the absence of such a guide, the Situation Analysis tool being developed by DOS and the Field Guide to Standards proposed by PCOS should be referred to in the PMW.

27) PCOS should develop a network of officers outside Headquarters who are able to provide on-the-job coaching to field staff in the use of logical frameworks and specific, measurable indicators. This training-of-trainers approach could be used to support those in the OMLP, and others who are not involved in the programme.

28) PCOS should update the OMLP and other forms of programme management training to include a greater, practical coverage of monitoring, building on the contents of the Generic Monitoring Guide proposed in this review. The OMLP should also provide clearer guidance on the link between protection and assistance activities, including specific references to the Protection Checklist and, examples of best practices with regard to integrated planning, teamwork, and beneficiary participation.

43 Chapter 4, ’Monitoring in the Protection Sector’.

MONITORING

32

E. Recommendations on monitoring and partners

29) UNHCR Representatives should involve partner organizations in all strategic planning exercises and engage with the Common Country Assessment, the UN Development Assistance Framework and the Consolidated Appeal Process. Such exercises should include the development of appropriate joint monitoring mechanisms.

30) UNHCR’s Donor Relations section should encourage donors to ensure that NGOs which receive bilateral funding are required to coordinate with UNHCR and to provide the organization with monitoring information.

31) Given the amount of monitoring undertaken by partners, UNHCR should, where possible and appropriate, harmonize its monitoring tools and data collection with them.

32) The review strongly endorses DIP’s current attempt to document and consolidate operationally-oriented training initiatives in the field for UNHCR staff and partners, and recommends that UNHCR develops a policy for the targeted training of IPs.

33) UNHCR staff should be provided with training and checklists in organizational assessment, so that they identify those IPs whose monitoring capacity needs to be strengthened.

34) Partners should be reporting to UNHCR on a monthly basis, using a version of the sub-project logframe to indicate progress against planned outputs (and impact where possible), accompanied by a brief analytical narrative of no more than one page. This would not obviate the need for the SPMR but would generate more timely feedback and hence increase responsiveness.

35) Officers in charge of monitoring sub agreements should make fuller use of the work plans accompanying them (currently optional), so as to provide better planning and tracking of progress over the year.

36) The electronic rather than paper exchange of contracts should be encouraged so as to ensure that UNHCR and partner staff are aware of agreed objectives.

37) UNHCR staff who are responsible for the day-to-day management of sub-agreements with IPs should be given increased authority to amend such agreements (within the total budget and against agreed criteria) in accordance with changes in context.

38) UNHCR should target support to weaker partners and reduce the surveillance of more capable, trusted partners correspondingly. The organization should provide financial training to its weakest implementing partners, as identified by Branch Office assessments and by the UNHCR auditors. As the organization’s financial training capacity has been dismantled, this may have to be contracted in.

SYNTHESIS REPORT

33

F. Recommendations on the management of monitoring initiatives

39) Principles for the management of monitoring initiatives are proposed:

• New guidelines should be introduced only after they have been field tested. Compliance can only be required from field offices when training and other support has been provided.

• New computer systems should be introduced only when they are accompanied by training and a system of on-going advice and technical support is put in place.

• The end use to which monitoring information is put should be properly explained to UNHCR staff before any new monitoring systems or requirements are established.

40) A three-year timeframe should be established for the development of UNHCR’s monitoring system, with field-level monitoring as the priority.

41) Donors should be asked to decide jointly what their expectations are with respect to the monitoring information they require from UNHCR. Donors should support the development of new field-based monitoring systems with sponsorship (funds, technical personnel, etc) for a three-year period.

42) The Executive Office should assign responsibility for the development and maintenance of a universally applicable and straightforward monitoring system in field operations to a senior manager, who would coordinate and direct the implementation of all quality improvement initiatives. This senior manager should also be required to ensure that all new training initiatives, instructions, guidelines, manuals and computer systems are within the capacity of field offices to absorb. The manager who is assigned the responsibility will evidently need to work closely with DIP and UNHCR’s regional bureaux.

35

Appendix 2 Terms of Reference

Enhancing UNHCR's capacity to monitor the protection, rights and well-being of refugees

Background

In a recent exchange of letters with the government of Canada in May 2002, UNHCR agreed:

to strengthen the processes within UNHCR which provide information on programme results. This includes the capacity of the Evaluation and Policy Analysis Unit to undertake evaluations of a high quality, the synthesis of lessons learned from evaluation into useful tools, as well as other measurement mechanisms for effective management and organizational knowledge development.

The exchange of letters also states that UNHCR:

will strive to utilize the Evaluation and Policy Analysis Unit's expertise in building a monitoring system that will allow the organization to learn and develop better programming as a result of evaluations undertaken.

Since those words were written, events have confirmed the need for UNHCR to review its monitoring and measurement mechanisms. On one had, the organization now has an obligation to report against the Objectives and Indicators set out in the 2003 Global Appeal. On the other hand, recent evaluations relating to refugee women, refugee children and community services, for example, as well as the 'sexual exploitation' scandals that have been reported in West Africa and other locations, all appear to suggest that UNHCR's monitoring capacity may have diminished in the past few years, despite the efforts that the organization has made to improve its programme management systems.

Donor states have pointed out that UNHCR has found it difficult to generate systematic data that measures the impact of recent funding shortfalls on the well-being of refugees. In addition, the former Inspector General’s paper on 'Observations from inspections' draws attention to the fact that some UNHCR offices lack the basic data required to provide effective services for refugees and other people of concern to the organization.44

To address these issues and to meet UNHCR's commitment to the Canadian government, EPAU is initiating a project which is intended to enhance UNHCR's

44 The issue of UNHCR's declining level of day-to-day contact with refugee populations has also been the subject of a UNHCR staff seminar, titled 'Why do we know so little about refugees… and what can we do to learn more?" An EPAU Powerpoint presentation prepared for the seminar suggested that this trend is the result of several factors: the growing tendency for UNHCR staff and offices to be located at a distance from refugee camps; the penetration of e-mail and internet facilities to the field; increased reporting requirements; staff shortages in the most difficult duty stations, as well as staff absences caused by MARS and VARI; and the difficulties involved in meeting refugees at a time of budgetary constraints and declining levels of assistance.

MONITORING

36

monitoring capacity. A consultancy team will be engaged to undertake the project, incorporating expertise in the areas of management systems, programme implementation and refugee protection.

The project will initially focus on UNHCR's monitoring activities in developing countries with relatively large and longstanding refugee camp populations. To the extent possible, the project will also address the issue of monitoring in other contexts: refugee emergencies, urban refugee populations, reintegration programmes and IDP situations. In accordance with UNHCR's evaluation policy, the project will take a disaggregated approach to refugee populations, giving special attention to the issues of age, gender and socio-economic differential.

The project will take due account of related initiatives taking place within UNHCR, especially a DOS project on standards and indicators, as well as the need for UNHCR to report on the implementation of the Agenda For Protection. The project will include a comparative element, taking into account the monitoring systems used by a sample of other UN and international agencies, such as ICRC, IFRC, UNICEF, WFP and selected NGOs.

Objectives and issues

The overall purpose of this project is to provide an empirically supported assessment of UNHCR's monitoring capacity and to make specific and realistic recommendations to UNHCR management with regard to the enhancement of the monitoring function, drawing on examples of good monitoring practice in UNHCR and other organizations.

The project will propose a series of practical and achievable steps for UNHCR to enhance its monitoring capacity. The project will not develop tools or training materials but will make proposals for the further development of monitoring tools and training materials.

The project will examine a number of principal themes and make recommendations for change with respect to each of them:

1. Definition

According to UNHCR's Mission Statement, the organization "is mandated by the United Nations to lead and coordinate international action for the worldwide protection of refugees…" The Mission Statement goes on to say that "UNHCR's primary purpose is to safeguard the rights and well-being of refugees." Building on these words, the project will commence with a working definition of monitoring as "the ongoing and systematic collection and analysis of information relating to the protection, rights and well-being of refugees, undertaken by UNHCR with the purpose of ensuring that the organization's objectives and relevant international standards are being attained."

The project will review the adequacy of this definition and examine the use of the monitoring concept by other relevant agencies. The project will also develop a typology of the purposes and levels of monitoring appropriate to the work of

SYNTHESIS REPORT

37

UNHCR. It should be noted that this project will not review the monitoring of individual staff performance, nor will it review the financial monitoring of UNHCR's implementing partners.

2. Monitoring practices

The project will ask whether current monitoring practices in UNHCR provide operations managers in the field and at headquarters with timely and relevant information, and examine the methodological and practical issues raised by the monitoring of refugee protection, rights and well-being. It will provide guidance on how managers can make effective use of data and analysis in making decisions to improve programme design and implementation.

3. Procedures, systems and tools

The project will review the procedures, systems and tools governing monitoring within UNHCR, assess whether they are appropriate and effective, examine the extent to which they are consistently applied in practice, and make proposals for their rationalisation and improvement. The project will examine the qualitative and quantitative indicators currently in use or development in UNHCR and make proposals in relation to way they can best be applied within UNHCR monitoring systems.

4. Responsibility for monitoring

The project will examine the respective roles of different headquarters entities and field staff for the monitoring function and make proposals for the clarification of responsibilities. The project will also assess the adequacy of the training in monitoring provided by UNHCR to international, local and implementing partner staff, and propose improvements as appropriate. It will also assess the potential for refugees to be more actively involved in monitoring activities.

5. Partnerships

The project will assess how and how well UNHCR monitors its partners' activities and their contribution to the protection, rights and well being of refugees. The project will assess the support that UNHCR provides to its implementing partners in relation to monitoring, especially those agencies with limited expertise and capacity in this area.

6. Practicalities

The project will examine the human, financial and other resources at UNHCR’s disposal and the impact on the effectiveness of monitoring of various organisational constraints including; budgetary shortfalls, staff shortages and turnover, host government policies, insecurity, and the availability of reliable quantitative and qualitative data. On the basis of this analysis, the project will make proposals on an

MONITORING

38

appropriate level of ambition for UNHCR's monitoring activities in sub-optimal operational contexts.

Project implementation

The project will be undertaken in accordance with UNHCR's evaluation policy and Code of Conduct. It will be guided by a Steering Committee consisting of key stakeholders within and outside UNHCR, and chaired by a member of EPAU. The Steering Committee will help to develop the terms of reference for the review, monitor the progress of the project and ensure that its findings, recommendations and outputs are effectively utilized.

The selected consultants will conduct interviews and documentary research at UNHCR’s Geneva headquarters, and will undertake missions to a maximum of three field locations, in at least two regions of the world, to be chosen in association with EPAU. To the extent possible, these locations will include at least one operation that is considered an example of good practice in monitoring; at least one relatively high profile and one relatively low profile operation.

The consultants will develop an appropriate means of sampling the opinions of UNHCR staff, offices and other stakeholders in countries which are not visited.

The project will include a range of activities designed to gather views, refine proposals, and generate consensus on steps to enhance UNHCR monitoring. One output of this project will be a report, providing an empirically supported analysis of the issues identified in these terms of reference, as well as a set of practical recommendations related to the enhancement of the monitoring function within UNHCR.

The project report will be placed in the public domain. UNHCR will not exercise any editorial control over the report but will provide comments on the draft and will proof-read and format the report prior to publication.

The team leader will be invited to provide a briefing on the outcome of the project to the October 2003 meeting of the Executive Committee

After the completion of the project, the consultants will be asked to prepare a brief ‘lessons learned’ report, analyzing the way in which this project was managed and undertaken. This report will be used to enhance UNHCR’s evaluation procedures and methods.

jc/EPAU, 11.4.03

SYNTHESIS REPORT

39

Appendix 3. Project methodology

The project team for this project comprised three external consultants, with expertise in protection, operations and management. Each consultant made a 90 day input to the project over the period April to September 2003. The project team45 was guided and assisted by members of EPAU; thanks go to Jeff Crisp and Sue Mulcock in particular.

Field missions. Field visits were made to Sierra Leone in April 2003 and to Zambia in September 2003. Although three field missions were initially intended, due to prior commitments of field offices, we were unable to secure a third mission within the project period.

Document review. The team undertook an extensive document review of UNHCR evaluations, guidelines, instructions, as well as the UNHCR Manual (especially chapter 4) and training documentation. In addition, reporting documents were reviewed (Country Operations Plans, Annual Protection Reports, Country Reports, Situation Reports, Letters of Instruction, Sub-Project Agreements and Sub-project Monitoring Reports) from country offices in Africa, Asia and Latin America. In-depth focus was given to a complete set of reports from Sierra Leone, Zambia, Columbia, Kenya, Indonesia, and Mexico.46 The monitoring systems of a number of other UN agencies and international NGO’s were reviewed, together with recent research papers on humanitarian issues.

Interviews. A large number of UNHCR staff were interviewed in UNHCR Geneva headquarters, in Sierra Leone and in Zambia. A number of NGOs were also interviewed during missions and by telephone. Meetings were held with ICRC, OCHA and IRC in Geneva.

Country office survey. A survey questionnaire on monitoring was sent to UNHCR Representatives in 36 countries as selected by EPAU. Only eight replies were received, so no statistical analysis of responses has been attempted, but the comments from country office respondents have been taken into account in the project analysis.

Donor meeting. A meeting of representatives of five donors States was held in Geneva with members of the project team and UNHCR Donor Relations, facilitated by the US Mission.

Interim presentations and discussions. There was no Steering Group for the project. Instead, three separate meetings were held during the course of the project bringing together interested parties to discuss the terms of reference, the issues emerging and the initial findings. A presentation of draft recommendations was made to a wider

45 Where the views of the project team are given, the team is also referred to as ‘we’ 46 In addition, the COPs, CRs and the APRs from Branch Offices in Angola, Bangladesh, Ethiopia, Malaysia, Nepal, the Phillipines, Somalia and Sudan were reviewed.

MONITORING

40

group of twenty-five staff and the comments received used to refine the findings and recommendations.

41

Appendix 4. Review of the evaluation process

The evaluation team was put together by EPAU, each member chosen to reflect a different although complementary area of professional expertise. Bringing together three individuals from diverse professional backgrounds and separate geographic locations was a challenge at times. Each team member initially viewed monitoring through a slightly different prism, and this led to an analysis of UNHCR monitoring from several different perspectives. Given the breadth of the project, we believe that these different analytical approaches were important, and helped to generate an inclusive set of observations and recommendations that reach across the various divisions and units within the organization.

Another, and more vexing challenge with this type of team approach, turned out to be administrative. Scheduling availability of team members proved to be difficult. We believe this could be avoided in future by each team member setting aside a continuous block of uninterrupted time to complete the project.

During the course of this project, many people within UNHCR expressed surprise at the breadth of the project’s scope. We also were initially concerned at our ability to proved useful recommendations to UNHCR given the many different types of monitoring the organization engages in and the wide range of issues that arise in each. At the same time, we also agreed with the thinking behind the ToR, that to restrict our scope more narrowly would leave out important components of the monitoring picture and in the end limit the usefulness of the recommendations made.

Given the size of the review, we were asked to write a number of stand alone papers, reflecting a different issue being covered in the project. These form the seven Chapters of our report. A synthesis of the Chapters was also recommended, in place of the more traditional executive summary.

This methodology may have been more time-consuming than writing a single report but the result hopefully has provided UNHCR with far more depth than a single report would have allowed. If this approach is adopted in the future we would advise the author of each paper to provide a brief summary of their paper for inclusion in to the synthesis report.

As initially conceived, there were to be three missions to the field. It turned out to be very difficult to secure even the two that were arranged, due to other demands and scheduling constraints within Country offices. More advance planning would have lessened the difficulties associated with securing these two missions and may have ensured another. That being said, the lack of a third mission turned out to be less of a drawback than initially thought because monitoring problems proved to be largely systemic and fine points of evidence were not required to draw out the main findings.

The field missions, although few in number, were very helpful, especially given that they occurred at different points in the evaluation. The mission to Sierra Leone near the beginning of the evaluation was beneficial both to ensure that the issues framed at the outset were relevant and inclusive as well as to highlight best practices and constraints to monitoring encountered in a field location. The mission to Zambia, at

MONITORING

42

the end of the evaluation, provided a good opportunity to test the soundness of our observations and to discuss and test the feasibility of many of our draft recommendations.

On both occasions EPAU arranged for a UNHCR professional staff person to accompany the team, to help with logistics as well to provide useful knowledge concerning field operations more generally. We are very appreciative of this assistance and found it very helpful.

We have benefited enormously from the willingness of UNHCR staff at headquarters and the field to meet with us, share their expertise, identify key problems with UNHCR’s monitoring function and suggest possible ways to improve it.

The team benefited from three meetings at Headquarters with a group of interested individuals from different units and departments across the organization. As helpful as these meetings were, however, they were not an adequate substitute for a Steering Committee.

A Steering Group would have ensured that the scope of the project was reasonable, the end users were clear, that there was greater ownership by senior management in the project before consultants were brought into the project. A Steering Group could have ensured that that the Chapters were reviewed by key internal players before completion.

We also believe it advisable for EPAU to assign one of its professionals to oversee a project of this kind. Such a person could agree to and review work plans, provide ongoing comment, and hold the team to account for progress and agreed upon deadlines.

The time, critical insight and editing suggestions provided by Jeff Crisp were exceedingly valuable. We are very thankful to him and to the support we received from Sue Mulcock throughout this project.