Theory and Reality: An NGO Perspective of

Challenges to Development Assistance Impact Evaluation

Victor Hsu

ABSTRACT

Non-governmental organisations (NGOs) are key actors in development assistance-related project implementation fulfilling critical roles in the frontline of combating abject poverty and world hunger. Universally recognised as skilled and cost-effective stakeholders in the long chain of aid effectiveness in Official Development Assistance, a survey of donor governments and multilateral institutions reveal that Impact evaluation is not explicitly required of NGOs. While in theory (by way of donor policy) that impact evaluation must be an integral component of all development assistance, in practice there are practical NGO realities that remain to be overcome to ensure that policy and implementation are aligned. Led by DAC/OECD, donors themselves are still debating challenges in impact evaluation ranging from theory, methodology, process and scope. While NGO groups and a handful of NGOs are doing their best to bring their managerial and operational procedures into alignment with the current emphasis on aid effectiveness, the overall reality is that NGOs face a complex set of internal and external challenges that have to be addressed, in the first place, by governments if impact evaluation is to be an urgent and priority concern, and ultimately is routinely integrated into their DIME. Donors have a critical part to play to facilitate this outcome. It is in the interest of all stakeholders to get this right as soon as possible.

EXECUTIVE SUMMARY

The primary purpose of this paper is two-fold: to assess the state of impact evaluation in the NGO community and to determine the best policy recommendations for enhancing and improving NGO impact evaluation of their development assistance projects. The paper begins with a survey of a representative group of governments and multilateral institutions’ policy for partnership with NGOs. This is followed by a survey of the existing donor policy on impact evaluation and their expectation of NGOs. A comparison is made between the INTRAC facilitated 2001 NGO Policy on Impact Assessment and a 2008 review by Vinod Thomas on the state of impact evaluation to highlight any divergence or convergence in the definition of impact, methodology, participation, assessment tools and use of indicators. Using the very same categories a similar comparison is made of the latest manuals from the DAC Quality Standards

2

for Development Evaluation (2010), the UNDP Handbook for Impact Evaluation (2009), the Millennium Challenge Corporation Impact Evaluation Guidelines, and the Norwegian Health Result-based Financing Impact Evaluation Network (2010). The comparison demonstrates the continuing variety of approaches within the aid community. Moreover, a survey of donors using Thomas’ assessment review reveals that information about IE of NGO projects, the practical problems encountered and the lessons learned are scarce because monitoring and evaluation documentation is often unavailable. The 2005 study conducted by F. James Levinson and Isabel Madzorera of Friedman School of Nutrition Science and Policy, Tufts University is cited to highlight some of the preciously lessons and insights they gained about IE in the field. The paper examines the institutional response by the two largest NGO organisations in development assistance: InterAction based in Washington, DC and ACT Alliance based in Geneva, Switzerland. This is followed by a comparison of the Impact Evaluation manual developed by CARE in 1999 and the ACT Alliance’s Guide to Impact Evaluation of January 2010. Again, it shows some evolution in the understanding of how NGOs are to conduct impact evaluation. Assessment is made of the factors that affect the ability of NGOs to engage in impact evaluation: the expertise deficit, impossible managerial scope and demands, financial constraint, the three-year project cycle, the CNN Factor, limited host or local partner participation, cultural sensitivity gap, managing for results: one size fits all and NGO political image considerations. The paper concludes with policy recommendations addressed primarily to donors who need to facilitate NGO participation in this essential aid mechanism. They have a responsibility to set a reasonable framework for impact evaluation by NGOs and to provide generous financial support to NGOs to engage in these activities.

PURPOSE

The primary purpose of this paper is two-fold: to assess the state of impact evaluation in the NGO1 community and to determine the best policy recommendations for enhancing and improving NGO impact evaluation in development assistance projects. This paper seeks to illuminate the multiple constraints that they face so that NGOs themselves, their partners and all stakeholders will be better prepared to formulate a realistic and improved framework for IE. It is in the interest of donors and other stakeholder to get this right because of the pioneering, piloting and frontline roles NGOs play. Getting it right should be an integral part to the current search for aid effectiveness and accountability.

NGOS IN THE LIMELIGHT

The Organization for Economic Cooperation and Development (OECD), the World Bank (WB), and the Dutch Ministry of Foreign Affairs Department of International Development

1 The NGOs discussed in this paper are by necessity limited to established traditional NGOs which have been partners to multilateral institutions in development assistance for many years and therefore provide a documented track record for research and evaluation.

3

Cooperation all speak glowingly of NGOs.2 The 2009 OECD review of aid practices of Development Assistance Committee (DAC) members states that

“Partnerships with local NGOs and community-based organisations enable donors to reach out to otherwise inaccessible regions and excluded communities, and to deliver humanitarian assistance. This is because NGOs tend to work effectively with highly vulnerable groups and because their staff in partner countries is largely local. In situations where donors are obliged to suspend their own development assistance operations, working through NGOs is often the only option.”3

It goes on to say that NGOs “Offer official donors operational alternatives…. they are skilled in mobilising grassroots communities and poor or marginalised people. They…. deliver services and programmes, and build coalitions and networks to co-ordinate civil society and enhance its impact.”4

The WB5 explains the critical NGO role in its NGO Policy designed mainly to guide its staff:

NGO involvement can contribute to the sustainability and effectiveness of Bank-financed projects by introducing innovative approaches and promoting community participation. NGOs can help expand project uptake and reach, and can facilitate greater awareness of diverse stakeholder views6

However, no government or multilateral institution is as explicit as the Dutch Ministry of Foreign Affairs Development Cooperation web site7 in identifying the essential role of NGOs in development assistance:

“Much of the actual development work on the ground is done not by governments but by NGOs. They are an important channel for the Dutch government. NGOs can fight poverty in countries where the Netherlands does not wish to work with the government, either because there is no government to approach (as in Somalia), or because the government pursues extremely bad policies. Another advantage of working with NGOs is that they often collaborate with local partners, who know the needs of the local population and are familiar with their culture. This kind of expertise is essential if projects are to lead to a reduction in poverty in the long term.”

2 These agencies use a variety of terms such as civil society organization, non-governmental sector, or private voluntary organizations. 3 DAC/OECD. Managing Aid: Practices of DAC Member Countries, (2009 Better Aid Series.) 66 4 Ibid. 5 Available from http://www.gdrc.org/ngo/wb-ngos.html 6 Appendix A 7From http://www.minbuza.nl/en/Key_Topics/Development_Cooperation/Partners_in_Development/Support_via _Non_Governmental_Organisations

4

Despite these glowing words about the role of NGOs by governments and by multilateral institutions, NGO involvement in impact evaluation (IE)8of their official development assistance (ODA) projects remains highly challenged by a complex set of factors. While NGOs will readily avow that IE should form a necessary component in project design, monitoring and evaluation (DME), their IE performance is not as stirring as their record in managing and monitoring projects.

DONOR POLICY ON IMACPT EVALUATION

The March 2, 2005 Paris Declaration on aid effectiveness called for harmonization, alignment and managing aid for results with a set of monitorable actions and indicators. Three and a half years later, donors and developing country governments and an unprecedented number of NGOs gathered in Accra for the Third High Level Forum to review progress in achieving the goals for aid reform. They called for joint progress toward enhanced aid effectiveness that, inter alia, emphasized:

1. Results and results based evaluation, and 2. Mutual accountability for development results on both donors and partners.

To accelerate progress in these areas, Accra also outlined two key additional steps: donors will provide 3-5 year forward information on their planned aid, and use partner country systems rather than their own donor to deliver aid as the first option9.

These are significant steps taken by the international community to improve the existing practice in ODA and other bilateral development assistance mechanisms. Whether they will generate a positive difference on NGO IE methodology and process is unclear. What they have done so far is to put NGOs on notice that their partnership with governments and multilateral institutions imply and oblige them to give IE their top priority.

An examination of current donor IE policy and practices reveals that donors only have expectations and not requirements that NGOs would include IE as part of their contract with the grant. Most if not all DAC member countries include in their ODA budget items to support NGOs’ development-related activities. Most of the funds go to national NGOs. Between 2005 and 2006, 5.2% of total bilateral ODA from all DAC member countries went to or through NGOs, ranging from 0.4% to 19.5% for individual donors.10

8OECD/DAC study in 1997 defines impact as ‘improvements in the lives and livelihoods of beneficiaries’. The definition formulated by F. Blankenberg in his 1995 Methods of Impact Assessment Research Programme, Resource Pack and Discussion has gained currency in integrating into a framework key IE concepts: long-term and sustainable changes introduced by a given intervention, unanticipated changes and consequences for the beneficiaries caused by an intervention both positive or negative. 9 According to the 2008 Report by Reality of Aid Organization, Aid Effectiveness and Democratic Human Right only 52% of donors are following these two recommendations. 10 DAC/OECD. Managing Aid. 181

5

A detailed survey of the DAC member websites11 yielded little evidence of the importance that DAC members attach to IE. Though there is universal mention of supporting, promoting and ensuring aid effectiveness in development assistance there is almost no mention of IE. Some members do include evaluation procedures in regard to their grants or ODA. But these tend to be evaluations based on whether a recipient has met the goals set as spelt out in the tender, contract or Letter of Understanding or Memorandum of Understanding.

Some governments and multilateral institutions have manuals for IE but as far as I can determine there are no IE requirements of NGOs. Perhaps donors are aware of various constraints that NGOs face so that IE is not obligatory or explicitly stated. We sample four big donors to obtain a representative picture of donor expectation of NGOs.12 Netherlands

Ministry of Foreign Affairs13

Swedish International Development Agency14

UK Foreign Secretary Department for International Development15

World Bank16

Policy Much of the actual development work on the ground is done not by governments but by NGOs. They are an important channel for the Dutch government. NGOs can fight poverty in countries where the Netherlands does not wish to work with the government,

Sida's financial support for civil society organisations (CSOs) amounts to SEK 1.2 billion for 2010. It is channelled through Swedish framework organisations and their respective development partners.

In 2010 Sida is working with 15 framework organisations. About 500 Swedish NGOs and other groups

The Department for International Development (DFID) leads the British government’s fight against world poverty. We work with many partners in order to achieve this, including civil society organisations (CSOs).

DFID supports civil society organisations through our country offices and centrally-managed funds. The latter include the Partnership Programme Arrangements (PPAs), the Civil Society Challenge Fund

The World Bank collaborates with CBOs, national and international NGOs in a variety of different ways.

Achieving the full potential benefits of NGO involvement implies enhanced roles for NGOs earlier on in the project cycle.

NGO involvement can contribute to the sustainability and effectiveness of

11 Appendix B. 12 Quotes taken directly from websites. 13 Retrieved from http://www.minbuza.nl/en 14 Retrieved from http://www.sida.se/English/Partners/Civil-Society-/Civil-society-organisations/How-Sida-supports-civil-society-organisations/ 15 Retrieved from http://www.dfid.gov.uk/Working-with-DFID/Funding-opportunities/Not-for-profit-organisations/PPAs/ 16 Appendix A

6

either because there is no government to approach (as in Somalia), or because the government pursues extremely bad policies.

The Netherlands works with both international NGOs (e.g. the mine clearance organisation Halo Trust) and Dutch NGOs such as Novib and Cordaid (see co-financing)

were involved in development cooperation programmes in more than 100 countries together with more than 2000 organisations and associations.

(CSCF), the Governance and Transparency Fund (GTF) and the Development Awareness Fund (DAF).

PPAs were established in 2000 to improve funding arrangements with 10 large UK-based CSOs. They now provide unrestricted funding to civil society organisations (CSOs) with which DFID has a significant working relationship, a common ethos and vision and a strong match in priority areas. These include both UK and non-UK organisations. Total PPA funding amounts to some £90m a year.

PPA funding is linked to a set of strategic level indicators – which the organisation is accountable for delivering over the 3-6 year time frame.

Bank-financed projects by introducing innovative approaches and promoting community participation.

NGOs can help expand project uptake and reach, and can facilitate greater awareness of diverse stakeholder views.

Expectation Another advantage of working with NGOs is that they often collaborate with local partners, who know the needs

Overall, Sweden aims to promote vibrant and democratic CSOs, based on their roles as the voice of poor and marginalised groups, and

Over the last 8 years, we have run 3 competitions for applications for new PPA partners. PPA partners must satisfy a range of criteria including:

NGOs are active contributors to the Bank's economic and sector work (ESW) and participate in lending activities from

7

of the local population and are familiar with their culture. This kind of expertise is essential if projects are to lead to a reduction in poverty in the long term.

provider of services such as health and education. SIDA’s support of CSOs is focused on:

Activities that are aligned to the current development cooperation strategy.

The capacity of civil society organisations to be strengthened by focusing on the development of the organisation’s decision-making structures and systems, internal and external communication, and actual performance.

Organisations and networks that strengthen the role of civil society as an arena for citizens’ engagement, and promoting transparency, cooperation and networking among organisations.

· sufficient consistency between CSO and DFID priorities

· high standards of corporate governance

· extensive ‘reach’ in poor countries and/or in the UK for building public support for development

· significant engagement in policy formulation with DFID or similar organisations at international level

identification through to evaluation.

Currently, NGO involvement is most frequent during implementation. Evidence shows, however, that NGOs can provide particularly valuable input during project identification and design.

Where NGOs are expected to participate in implementation, they should also be consulted during design.

Consult with NGOs on appropriate strategies to support their institutional development.

Where appropriate, build a training component for NGOs into project design.

Encourage partnerships between

8

Opportunities that promote CSOs role to influence the design and implementation of poverty reduction strategies, in dialogue with the governments in partner countries.

international and local NGOs.

Promote networking and information-sharing among NGOs.

There are clearly stated objectives and expectations in government partnership with NGOs. These include enhancing aid effectiveness and accountability through capacity building in DME and M&E, increasing the breadth and depth of partnership within civil society at local and national levels and advocacy. The channel of support for NGOs is instructive. The Netherlands, Sweden and the United Kingdom all require that their NGO partnership have similar goals and aspirations as the governments in development assistance. In that sense, to qualify for grants, NGOs must be what SIDA has defined as “framework organisations.17” It is legitimate to ask why IE is not included as part of the expectations in the framework.

GOVERNMENT AND MULTIPATERAL AGENCIES REALITIES

Vinod Thomas, Executive Director of the WB’s Independent Evaluation Group18 noted in his Introduction to the Report of the 7th meeting of the DAC Network on Development Evaluation that while IE was high on the development agenda there remained significant challenges. He said that because most of the activities were relatively new they were therefore in pilot and exploratory stage, e.g. the WB’s own DIME Initiative, the IFC’s Advisory Services Impact Evaluation Initiative, the Spanish World Bank Trust Fund for Impact Evaluation and the Africa Impact Evaluation Initiative19. The situation may be different today, some two and a half years later. Thomas indicated in his review that a number of initiatives had been taken by IEG to explore further how to refine the methodology so that IE could be both cost effective and contribute to “SMART policy. ” From a NGO perspective these activities are seen as a runaway train in ODA administrative requirements and costs, a train that they will be unable to catch. More seriously,

17 Ibid., SIDA 18 Introduction to the Report of the 7th meeting of the DAC Network on Development Evaluation 19 Ibid., 2

9

the relevance for NGOs of the current multilateral institutions studies on IE is at stake too. Quoting from Thomas’s introduction:

Of the over 200 studies already catalogued, more than half are in social protection, many for

cash transfers. Just 2 percent are for agriculture, showing that there are clear gaps in coverage. Little progress seems to have been made on the seventh MDG dealing with the environment, but this remains hard to assess because it is also an area with poor data and little evaluation.

Most of the NGOs are not involved in social protection and cash transfer and few are

working on environment issues. Thomas’ proposal to limit impact evaluation to a relatively small percentage of operations is most welcome and makes a lot of sense for government agencies. His argument is that the evaluation should be to build a collection of policy-relevant knowledge . However, it is unclear how they can apply to NGOs whose operations are usually extremely limited in scope and targeted to the most remote areas or the most vulnerable groups.

Moreover, these activities are being carried out by the operations departments of the

multilateral agencies . In the NGO world, to expect operations department to perform impact evaluation will run into a storm of protests about staff’s lack of capacity, competence and expertise and existing work load. To give evaluation to the main implementers of projects invites questions about impartiality of the evaluation.

Given these complexities in the current practice, it is useful to compare Thomas’ assessment

to the goals and objectives set out in the NGO policy on IE in 2001 by a group of European NGOs facilitated by the International NGO Training and Research Centre (INTRAC.) In the past two decades, NGOs have played a vital role in partnership with governments, UN agencies, and others. They have networked and organized themselves effectively, thanks in large part to the Internet. Information about major international events or concerns is instantly transmitted as is the policy decisions by a government or a multilateral institution. Consequently, they are able to consult with each other and work collaboratively leading to a more cohesive and coordinated response to governments. It is therefore necessary to assess NGO approach and government expectations in IE. See table below.

NGO Policy 200120 Vinod Thomas IEG 200821 Impact Definition

Sustainable change Result oriented outcomes and process based

20 Ibid. 21Introduction to the 7th meeting Report. 2-6

10

Methodology Build IE into program and improve monitoring methodology, A dynamic process not a final act Need to accept creative tension between open and structured framework

Still experimental Need to streamline to a few operations Should be factored in ex ante Should contain incentives Cost-benefit analysis, good M&E, and the judicious use of indicators,

Participants Consultative, Co-operative, Collaborative, Collegiate Local Stakeholders to develop ownership

Good access to project implementers to ensure access to data, though care must be taken to preserve independence of study. Good relationship will also enhance the impact of the study findings—especially if preliminary findings are shared and discussed, so there are no surprises on either side when the final report is issued

Assessment Tools

Triangulation: systematic application of qualitative methods using a mixture of methods, tools, and perspectives, e.g. working in teams and using multiple data sources to ensure reliability and validity of the process.

Costly Must limit scope

Use of Indicators

keep to a minimum but use rating system

Too many indicators

Lessons NGO Policy 2001 Vinod Thomas 2008 Complexities Confusion between

program objective and long-term impact Poor tools and methodology Data paucity affects quality of IE

Must be timely Gaps in coverage of MDGs

Skills and Experience

Close management and monitoring of triangulation

Evaluation being carried out by operations instead of by independent evaluation group.

11

Data Quality Much improved with rating system

Should be for policy-relevant knowledge

Participation Local stakeholders needs time to appreciate the tools and assimilate implications

Influence on government generally is greater when the message comes from an organization trusted by government, which usually means working in collaboration with a government agency or a local policy or research body that has government’s ear

It is quite clear that since the 2001 NGO policy paper on IE was developed the issues and

challenges remain manifold and complex when the DAC Network on Development Evaluation met in February 2008. Both the methodology and the framework are still in testing stages and continue to be debated. Satisfactory evaluations are few and far in between.

Thomas mentioned two challenging factors in IE that have particularly relevance for NGOs.22 The first is the scope of the evaluation. Till now, even though a project may be limited in terms of time frame and geographic area, the detailed requirements of putting together an evaluation can be rather daunting as we will demonstrate in this paper. The other is very well-known: the cost factor. The “cheap” example cited by Thomas cost US$40,000 while an expensive project can skyrocket to US $500,00023. Most NGOs will find it a real funding challenge to come up with such an amount of funds or to justify this expenditure vis-à-vis their donors. Donor reluctance to allocate funds for impact evaluation cost by NGOs places a serious constraint on the NGOs to carry out this worthwhile task. The table below was prepared by Homi Kharas of Brookings Institution. It compared the 2006 expenditure of the top ten NGOs involved in ODA24. An IE expense of $500,000 as mentioned by Thomas would mean 8.3% to 9.5% of the expenditures of the bottom seven NGOs.

22Ibid. 23Ibid., 2-3 24“The New Reality of Aid.” (2007)

12

But we also would indicate that even when an NGO is ready to embark on evaluation it has to take into account a host of considerations regarding training, orientation and staff capacity. For the sake of the integrity of development assistance, certainly donors should be more open to supporting evaluation costs but that in themselves will not address the multiple constraints NGOs face. We now use the same categories to compare the main IE categories in the latest manuals. The DAC Quality

Standards for Development Evaluation25

UNDP26 Millennium Challenge Corporation27

Norwegian Health Results-based Financing (RFB) Impact Evaluation Network28

25 The 2006 draft Standards were revised based on experience of a range of OECD development partners. A 2008 survey of the use of the Standards, a 2009 workshop in New Zealand and comments submitted by the members of the DAC Network on Development Evaluation, helped to improve and finalise the text. The Standards were approved by the DAC Network on Development Evaluation on 8 January 2010 and endorsed by the DAC on 1 February 2010. 26 Programme Handbook on Planning, Monitoring and Evaluating for Development Results. ( 2009.)

27 Retrieved from http://www.mcc.gov/pages/activities/activity/impacy-evaluation 28 Evaluation of 8 pilot countries for generating new knowledge that can inform and improve RBF, to help governments and partners to effectively design and use RFB in other contexts.

13

Impact Definition

Systematic and objective assessment of an ongoing or completed development intervention, its design, implementation and results

Development results that accomplish national priorities.

Measures changes in individual, household or community income and well-being.

Methodology Based on DAC principles of development assistance: impartiality, independence, credibility and usefulness. Includes specification and justification of the design of the evaluation and the techniques for data collection and analysis.

Integrated PME principles in the context of result-based management and managing for development results with planning and programme definition, stakeholder involvement, communication, monitoring and evaluation.

Us e of counterfactual to identify what would have happened to the beneficiaries absent the program. Part of M&E.

The HRBF Impact Evaluation Network: World Bank Task Team Leaders (TTLs), Country-level Principal Investigators, international multi-disciplinary team comprised of experts in health systems, health provider payment schemes, child and maternal health outcome measurement, economics and impact evaluation. Network will collaborate on the systematic and coordinated program of 8 country-level rigorous impact evaluations to identify causal effect of pilot programs on the welfare of the

14

population, with a special emphasis on maternal and child health outcomes.

Participants Partnership involving different stakeholders such as governments, parliamentarians, civil society, intended beneficiaries and international partners

Engagement of stakeholders throughout the process. Encourage ownership.

Independent professional researchers.

Assessment Tools

Quality control throughout the process with internal and/or external mechanism such as peer review, advisory panel or reference group.

PME M&E by independent professionals "Performance management is the systematic process of monitoring the results of activities; collecting and analyzing performance information to track progress toward planned results; using performance information to influence program decision making and resource allocation; and communicating results achieved, or not attained, to advance organizational learning and tell

Causal effects, costs and operational feasibility, access and quality of preventive and curative health care, health expenditures, and health outcomes. Analysis of distributional effects and any impact of improved health outcomes on socioeconomic outcomes including unanticipated consequences.

15

the Agency's story.29"

Use of Indicators

Indicators for measuring achievement of the objectives are to be validated according to accepted criteria such as SMART: Specific, Measurable, Attainable, Realistic and Timely

Is there national ownership? Promote national capacity development for sustainability. Promote gender inclusiveness, gender mainstreaming and women empowerment.

standardize as much as possible the methods, measurement and data quality control across countries, while tailoring to local policy interests, operational ability, and political and cultural climate.

The chart shows that, with the exception of MCC, NGO participation will be substantial in the methodology proposed in these IE manuals. In socializing IE among aid agencies and NGOs with ‘behind-the-scenes’ impact of advisers, and consultants, we must note the warning given by Reality of Aid organization: Managing for results can only be effective when the results being targeted are poverty reduction and the promotion of human rights and gender equality. When the ‘results’ being managed are economic policy reforms, then this ‘principle’ becomes a justification for conditionality. The principle of mutual accountability requires the development of specific mechanisms by which aid recipients can hold donors to account. Once again this must not be limited to recipient governments, but must also include the communities most affected by aid expenditure.30

In surveying the scene of IE among donors it soon became apparent that information about IE by NGOs in the field, the practical problems encountered and the lessons learned are scarce. The most substantive, and in the author’s opinion, the most authoritative, is the 2005 study31 conducted by F. James Levinson and Isabel Madzorera of Friedman School of Nutrition Science and Policy, Tufts University. Their study is based on a number of projects for which monitoring and evaluation documentation is available: the Tamil Nadu Integrated Nutrition Project in South India, the Bangladesh Integrated Nutrition Project, the Iringa Project in Tanzania, the Posyandu projects in Indonesia, the Integrated Child Development Service (ICDS) program in India, the UNICEF-assisted Dular project in Bihar state in India, the HKI Nutrition–Focused Child Survival Project in Mali, the CAREassisted Child Survival Project in Nicaragua, the Enhanced Outreach Strategy of the Ethiopian Child Survival Project, Save the Children (US) positive

29USAID defines monitoring and evaluation in ADS 203 30 Reality of Aid Report (.2007.) 21. 31 Food Policy and Applied Nutrition Program Discussion Paper No. 28. Available from http://nutrition.tufts.edu/1184937206715/Nutrition-Page-nl2w_1184937205502.html.

16

deviance-based projects in Vietnam and its Jibon o Jibika Project in Bangladesh, and the Progresa Project in Mexico.

Levinson and Madzorera drew nine lessons for IE from these collective experiences. I highlight only those that appear to me to be new elements that should be taken into consideration in the quest for the most satisfactory framework for IE and that have so far not surfaced in our surveys and comparisons.

1. Monitoring and evaluation will be greatly facilitated by a conceptual framework and attention to behavioural outcomes32

Traditional project evaluation method seeks to determine whether project objectives have been achieved by comparing pre-post differences in project and control areas. While such an approach may accomplish the narrow purpose of evaluation, it missed the opportunity to understand the means by which project impact might have been achieved or the reasons it failed to accomplish desired impact. The increasing trend in project monitoring and evaluation systems is a conceptual framework in the form of mapping the variables. While this methodology has worked well for nutrition-related projects, particularly those which have a behavioral change component, the behavioural outcome is still the most frequently neglected aspect in nutrition project-related monitoring and evaluation. Other projects which included behavioural change communications as a major or the major intervention failed to collect data on behaviours at all. These projects chose instead anthropometric change. The consequences of such neglect were serious. “Without knowing whether behaviors have changed, it will be exceedingly difficult to gauge the sustainability of impacts achieved,” they said33. The use of a conceptual framework can be tremendous advantageous because it would allow “backward mapping,” the process of combining evaluative data with monitoring data. It can determine whether behavioural change or lack of it is responsible for the success or failure of the project outcome. By contrast, if services have been well provided but outcomes and impacts are not achieved, there may a structural problem in the nature of the intervention34.

2. Monitoring data must be utilized locally35

Neither the NGO Policy nor Thomas was as explicit about how institutions conducting IE used monitoring data. Levinson and Madzorera criticized the transferring of data like a post office in the MIS. The process is cumbersome, time consuming and tend to negate the primary purpose of monitoring, i.e. the ability to quickly correct problems in implementation. Furthermore, these MIS reports, generated usually many months after the data have been collected, are seldom in a sufficiently disaggregated form to be useful.

Levinson and Madzorera plead for a monitoring system which places a premium on local utilization of data. It means that IE would initially involve a critical local examination of data

32 Ibid., 2 33 Ibid., 3 34Ibid. 35 Ibid., 4

17

collected, particularly to identify lapses or gaps in coverage. At every link of the chain of data aggregation, and prior to aggregation taking place, data from the localized areas are assessed for problems or inadequate performance. This “problem link” would be “triggered” so that the appropriate report on follow-up action becomes an automatic part of the process36.

Recognizing, however, that monitoring data frequently overestimate project success there must be a built in periodic quality checks on monitoring data.

3. Ongoing external monitoring and evaluation presence accompany large projects37.

Levinson and Madzorera suggest that an ongoing external monitoring and evaluation

presence accompany large projects. The NGO policy did not make such a recommendation while Vinod Thomas’ suggestion was to streamline the operations.

The authors noted the following serious problems which have plagued M&E systems:

• Separate bidding for baseline surveys, mid-term evaluations and endline surveys often result in two or even three external entities, often private consulting firms, being responsible for a survey.

• Monitoring data are often inflated, requiring periodic quality checks. • Monitoring data also require periodic disaggregation according to context.

• Ongoing review of monitoring data frequently leads to questions which need quick answers and which require operations research38. The value of an ongoing external presence is not only to ensure the integrity of the project’s

baseline and evaluation surveys, but also to carry out the essential periodic checks of monitoring data, periodically disaggregate monitoring data, and be prepared to address operations research questions as they arise. It should result in evaluation reports based on a thorough understanding of the project39.

4. Control groups are inadequately comparable40

Many evaluation results were suspect due largely to the absence of control groups that are

genuinely comparable to project areas. This absence of appropriate controls often relate to one of the following three situations:

• Some projects with serious funding limitations remain unconvinced that spending limited resources on a control group is necessary.

• In larger national or regional programs, governments often prefer to choose project areas (and by default control areas) on political grounds.

36 Ibid. 37 Ibid. 38 Ibid., 5 39 Ibid. 40 Ibid.

18

• In some countries, the presence of proliferating local activity throughout the country, often generated by NGOs, makes it impossible to find true control areas41.

In their study, they encountered difficulty in ascertaining whether control groups had been

randomly chosen because this is a critically essential part of the methodology. 5. Need to evaluate the distribution of benefits in projects42

Differences analysis which compare and average the post-minus-pre changes often

camouflaged the effect of the project on some population groups. In some situations where projects successfully attained stated objectives they might not have been successful in either improving the nutritional status of women or children in low income households, or in bringing about the same percentage improvement compared with their more advantaged counterparts43.

6. Exit strategies and sustainability also need to be evaluated44

For Levinson and Madzorera, “evaluation of an exit strategy, by definition, requires the collection of data on outcomes and impacts, at periods subsequent to the termination of project inputs.” But in their survey this effort was “hardly ever reflected in project budgets and rarely carried out.” Including exit strategy in DME is necessary because it would be “enormously valuable in sharpening the sustainability promoting features of projects, and in turn increasing the likelihood that project impacts and outcomes will, in fact, be sustained well beyond the period of a project.”45

NGO GROUPS RESPONSE TO IMPACT EVALUATION

With regard to how the NGOs are responding to the donor interest in aid effectiveness, accountability and IE we cite two major NGO organisations: InterAction46 in the USA and ACT Alliance47 in Geneva, Switzerland. Both have working groups on IE. The purpose for these groups is to encourage their members to be aware of the existing IE practice in the NGO community and to offer technical help to those interested in becoming engaged. Among InterAction's mandates is to support member organizations’ capability to conduct programming that meets and exceeds internal self-certification standards for quality and effectiveness. This is done through working together to strengthen M&E capacities through experience and resource sharing, as well as opportunities to work with outside experts. The rationale for this endeavour is stated as:

41 Ibid. 42 Ibid., 7 43 Ibid. 44 Ibid. 45 Ibid., 7-8 46 Largest alliance of over 190 USA-based international NGOs 47 It has more than 100 member organizations who work in 130 countries.

19

When integrated deliberately into our work and conducted with appropriate rigor, M&E supports learning and quality improvement at an intervention, program or sector level, for a population or area, as well as for an organization itself. Tracking and assessing our work allows us to make crucial improvements to our performance and offers invaluable assurance to our partners that their collaboration with us produces positive change….. InterAction’s Evaluation and Program Effectiveness Working Group (EPEWG) is dedicated to building NGO monitoring and evaluation capacity, and is currently focused on NGO accountability, impact evaluation and advocacy evaluation. The working group also informs advocacy initiatives to influence the U.S. and global aid effectiveness discourse.48

Created in 1 January 2010, ACT Alliance is composed of more than 100 churches and church-related organisations that work together in humanitarian assistance and development. The alliance works in 130 countries and mobilises US$1.5 billion annually in its work. The agencies together have over 30,000 people working globally. ACT is a signatory to the Code of Conduct for the International Red Cross and Red Crescent Movement and NGOs in Disaster Relief and is committed to the Sphere Humanitarian Charter and Minimum Standards in Disaster Response and the Global Humanitarian Platform’s Principles of Partnership.

Its Impact Assessment Working Group developed a guide for its members49.

It may be instructive to compare the two existing documents: CARE Impact Guidelines and ACT Alliance Guide developed some ten years later.

CARE50 ACT Alliance51 Impact Definition

Sustainable improvements in human conditions or well-being

Lasting and significant changes in people’s lives including unintended changes, whether positive or negative to which the project contributes directly or indirectly

Methodology Case studies on best practices Participatory and empowerment Participants All key stakeholders Senior and programme managers, PME

specialist, evaluators, frontline staff and intended beneficiaries

Assessment Tools

Sound diagnosis, analysis and design

Mix of 24 complementary tools

Use of Indicators

Employ only tested ones. Quantitative or qualitative evidence used to assess the extent to which intended changes are achieved. Indicators grouped around principles.

Exit Strategy Project final goal must be 48 Retrieved from http://www.interaction.org 49 Retrieved from http://www.actalliance.org/resources/policies-and-guidelines/impact-assessment 50 Developed in 1999 51 Guide developed by predecessor body (Act Development) in 2008

20

embedded in long term program and strategic framework, must be significant and achievable

The comparison shows divergence in methodology, assessment tools and use of indicators. Had data been available from InterAction’s own IE Working Group, it would be possible to make an analysis of the current practice of IE in the NGO community. The information on the InterAction web site indicates that its working group met in January 2010 and Act Alliance working group schedule is not published. It is quite conceivable that there is not much energy being devoted to IE. In describing briefly these two large NGO groupings, I wanted to highlight the fact that neither group chose to use any of the existing IE manual as prepared by CARE or ACT Alliance. Mercy Corps DME manual was developed as recently as 2005.

International NGOs engaged in development assistance face multiple challenges already. These are well documented in many studies.52 Even with systems and structures in place, NGOs encounter issues which impinge on their performance and accountability. Therefore, to address the inadequacies in the NGO performance, it is important to understand the various internal and external factors which serve as obstacle to overcome if they are to integrate IE into their activities. Anecdotal review of NGO’s representatives indicates that IE is rarely an integral part of the project reporting process of DME.

NGO OPERATIONAL REALITY

A. Expertise Deficit

If NGOs lack IE know-how, it does not mean that systems are not in place. In fact, as the reference section demonstrates, NGOs have their own project manuals with detailed designing and monitoring tools53. There are also consulting firms such as American Evaluation Association and Food and Nutrition Technical Assistance (FANTA-2) which offer NGOs a variety of IE services. And as we highlighted above, their own associations such as InterAction and ACT Alliance can also facilitate their engagement in IE.

An indication of a general problem in NGO IE is that their staff NGO handbooks have to include simple “how to” detailed instructions. The level of expertise or understanding of the staff will determine whether they are given "How to set up M&E systems for beginners" or "How to tackle different M&E system's issues for advanced." This is true today in such well established “development” NGOs as World Vision, Mercy Corps, CARE or Oxfam. Aid agency manuals from UNDP to USAID also provide detailed instructions to guide their experienced staff in the use of the tools for IE. 52 Perhaps the most notable is John Degngol-Martinussen and Poul Engberg-Pedersen, Aid: Understanding International Development Cooperation. (London and New York: Zed Books, 2003) 53 See especially World Vision International websites.

21

To meet aid effectiveness concerns by donors, there is now an increasing tendency among NGOs to send their own “professionals” from headquarters to train and upgrade their staff capacity in the field. They place emphasis on technical, operational and managerial know how. Development workers of yesteryears are now specialists in capacity building. However, they are not impact evaluation specialists, at least not yet and not until the organisation makes IE an integral part of DIME.

B. Impossible Managerial Scope and Demands

In an attempt to be thorough and comprehensive, the donor manuals for IE tend to be overwhelming for NGOs who have, as noted, few dedicated staff in this role. A big NGO like World Vision do try to encourage their programme staff to integrate IE in their work. Nevertheless, it is not mandated for administrative and financial reasons so the performance across its offices in over one hundred countries is uneven. For Mercy Corps, which developed its DME Manual in 2005 it still does not have the required resources to ensure that all its offices around the world fulfill this expectation as a matter of priority or on a routine basis. In fact, most NGOs do not have any DME manuals and have not viewed producing them to be a necessity. To add IE to the NGO routine would require that NGOs makes it a top priority as part of their programme, systems and structures. This brings to the fore the next major constraint, the lack of financial resources.

C. Financial Constraint

The most serious contributing factor is the lack of funds due in large part to donor reluctance to provide for such administrative costs. This is ironic. In recent years, donors have opted for NGO implementation of programmes because they are “cheap” in comparison to other alternatives. NGOs too pride themselves in low overhead costs and use this as a fundraising edge. In the United States, there is a volunteer watchdog that annually ranks non-profit organisations and NGOs according to their overhead spending.54 All donor agencies also set a percentage ceiling for the amount that an NGO can allocate for project implementation personnel55. This reality hits IE in several ways:

· Limited staff are assigned. · Capacity training funds to equip staff for IE are insufficient or unavailable. · IE that is performed is inadequate and of limited use. · IE becomes deemphasized in the DIME chain.

54 This ranking is publicized in various national magazines such as Money and is available on the Internet. 55 The acceptable percentage is about 25% of the total cost of the programme.

22

When financial constraint limits impact evaluation of government funded activities and projects, it goes without saying that the NGOs privately supported projects tend to suffer an even more severe fate in terms of IE. When NGOs operate on a restricted budget and fail to include an IE component, it can be a serious and costly omission. Having adequate funds will go a long way towards redressing the current performance deficit on the part of NGOs. This can be accomplished either by specifically IE designated funds in the grant or by allowing NGOs to take IE expenses out of the project grant as part of the project cost. It is highly unlikely that NGOs will be able to obtain private donor contribution to IE activities.

D. The Three-Year Project Cycle

The normal duration for a donor funded project is three years. The three-year period coincides with the tenure of assignment of Foreign Service officials in many countries including those with responsibilities for overseeing their countries relationship with NGOs or civil society. NGOs without a national office in the country in which the project is being implemented are also subject to time consuming process of startup which easily cuts into the project period. The three-year project cycle presents IE several challenges.

· Often there is no provision for post project evaluation. · In development assistance sometimes three years is too short a period because it takes

time to set up the project with the donor and the requisite administrative infrastructure with a recipient country.

· It also may take more time for the project to create a real impact. · The integrity of IE is affected by donor or NGO staff turnover. Since the oversight staff

tends to be small, any staffing change easily affects the plans for IE or the IE process already underway.

E. The CNN Factor

It is well known that in a dire situation of need, NGOs tend to become too enthusiastic to become involved without doing the requisite planning and feasibility study about their intervention. While established NGOs tend to avoid such an operational trap, this has not been successfully resisted. The latest most glaring example of such over enthusiasm was the Haiti intervention following the devastating earthquake of January 2010. Almost without exception, NGOs promised the moon to their donors about their disaster relief program in Haiti. Today, some 10 months later, few are able to operate effectively according to the plans and time line they laid out at the outset of the crisis56. While NGOs will defend their modus operandi of

56 CNN and New York Times did a special 6-month review in July. They documented that few NGOs or multilateral agencies were able to carry out their relief activities and that most of the funds promised by governments were still not received by the Haiti Recovery Commission.

23

taking advantage of the so-called CNN factor, i.e. raising funds when the media are covering the crisis, the critical issue of their accountability to the Haitian people remains. This accountability cannot be “excused” by the problems on the ground. These should have been foreseen. Moreover, years of experience should have warned the NGOs that this poorest country in the Western Hemisphere with inadequate transportation infrastructure would be able to accommodate large scale intervention by land, air, or sea.

Rushing into relief operations can result in poorly negotiated contracts with local stakeholders. Given the trend that relief operations often turn into development assistance, a bad contract that lacks safeguards can be costly in terms of resources and time. Conflict with local stakeholders and partners can ensue, leading to accusations of bad faith and ultimately exit from the community if not the country.

No credible IE can result from poorly planned interventions. Special attention must be given to this factor by donors given the number of emergencies which occur every year and the enormous amount of NGO interventions generated.

F. Limited Host or Local Partner Participation

Another set of operational constraints is the limited participation of the indigenous or local community in project implementation. Without the full involvement of the hosts, or if the latter are involved only in project implementation alone, it is hard to imagine an IE process that can proceed effectively. This is especially true in situations where the NGO is not an established presence known to the local population. In determining whether a particular project has had a profound impact on the population, it often requires a wide sampling of the community. In a remote area this task may not be accomplished without cultural sensitivity and knowledge of the local language. A NGO is usually perceived as a foreign organization which had entered the community to do good work. In other situations, after a prolonged presence, when the NGO staff still do not speak the local language and the senior staff are all expats, they perpetuate the notion that the local hosts are not qualified or to be trusted. Such a perception will impede the involvement of local hosts or partners and can create misunderstanding about how the evaluation is to be conducted. To minimize such a possibility, local partners should be involved in DME.

Julie Fisher’s book, The Road for Rio: Sustainable Development and the Non-governmental Movement in the Third World, documented the disastrous failures of development projects (and therefore wasted resources) that ignored local participation. These failures had been cited in WB’s own reports. The reasons for limiting or excluding local participation are usually two-fold. On one hand, the lack of “expertise” and the high cost to the implementation agency on the other.

24

I would note that no development project can be sustainable without local ownership of the initiative. Ownership is hard to achieve without involvement. Furthermore, this ownership cannot be limited only to national government leadership or national ministries. The local communities most affected by development policies must have ownership over them, by being consulted in not only about such policies but also how they should be implemented and evaluated. In linking aid to national development strategies, based on Accra’s recommendations, donors must help ensure that recipient countries develop strategies that will encourage local ownership.

IE done by agencies such as MCC and Norwegian RFB whose methodology emphasizes outside consultants and experts needs to be compared to those that involve local communities.

G. Cultural Sensitivity Gap

Closely related to local participation issue is that of cultural sensitivity. It is instructive in this regard to note the relatively successful international intervention to manage the HIV/AIDS crisis. The Global Fund had demonstrated in several African countries that international efforts in dealing with this pandemic would not have been as successful without the combined dedication of local, national, and international NGOs.

Local NGOs are critical in ensuring that government and civil society factor into their intervention the local population’s cultural practices. The extent to which information and efforts to change behavior are tailored to the cultural beliefs of the local population can make or break a development project. Development programmes with focus on knowledge, attitude and behavior, cannot have fundamental impact if psychological components of attitude development and change within a given cultural context are ignored at the design stage. International NGO’s with national and local counterparts have the advantage of input and feedback from their local affiliates in designing culturally appropriate programs. This input must form an essential component of IE.

Cultural sensitivity and cultural relevance are particularly important when matters of health and welfare are concerned. In fact, an ethnic group’s response to illness, tragedy, natural disaster, and up-rootedness is embedded in its cultural and religious beliefs and practices. Aid workers in Aceh, Indonesia re-discovered this phenomenon in the 2004 Indian Ocean tsunami. The Aceh population was not unwilling to be mass relocated beyond their ancestor homes nor did they assent to large scale reconstruction of their land unless proper respect was accorded to their deities in the relocation process.

The NGO section in the Global Policy Forum website carries an instructive story about the importance of understanding local culture in Ghana during the mass exodus of Ghanaians from Nigeria in the l980s. International NGO personnel arriving on the scene to dispense humanitarian aid were surprised at the lack of relief camps at the border. They brought in

25

supplies and equipment generally found to be essential in such camps. They were unaware that the government had put in place a plan for transporting incoming these refugees immediately to their hometown or location of ethnic origin given the existence of a strong extended family system in the country. The government’s plan thus avoided the hardship and squalor usually associated with emergency squatter arrangements, and placed the initial responsibility for resettlement on the returnees’ families and community.

The NGOs were forced to re-evaluate the role they needed to play in this cultural context.

Furthermore, in multiethnic societies, ethnic competition and rivalry may affect the operational environment. If the NGO is ignorant of the local context, it can be manipulated in terms of choice of location and local partners by a government agency whose staff may have invested self-interest in guiding the NGO in the choice of project location or project implementation personnel. Such a situation will inevitably affect IE of that project.

H. Managing for Results: One Size Fits All

The professionalization of NGOs and the emphasis on accountability and effectiveness are putting pressure on the NGOs management. And in the interest of alignment and harmonization, the NGO administrative and operational manual gives the strong impression that they operate on one set of managerial and implementation tools. This is enforced by the standardization of staff performance appraisal57. NGO CEOs will acknowledge that they have a responsibility to ensure that the services provided are suitable to the population being served, and that no “one size fits all” approach is to be used as a model. Nevertheless, a serious adverse effect of harmonization as put forth by donor policy and practice is to reinforce the donor as the principal arbiters of aid policy. NGO CEOs will inevitably arrange and harmonize their operational model accordingly. To comply with such concerns by donors, there is an increasing tendency among NGOs to send their own “professionals” from headquarters to train their staff in the field to perform functions with a view to aid effectiveness. They place emphasis on technical managerial know how. Whether these managerial procedures are effective and actually lead to poverty reduction becomes a secondary concern.

Furthermore, it is almost irresistible to NGOs to replicate administrative and managerial models. To be sure, given the oft repeated criticism of NGOs laissez-faire attitude, there are indeed advantages for NGOs to become more professional and business-like in their management. In the main, the merits outweigh disadvantages. However, the author contends that such an approach makes sense more in a national context and that it is fraught with pitfalls for international NGOs. Rationalization and organizational alignment do make sense but DME and IE cannot be smooth, efficient or accountable if local realities and context are willfully

57 Pranov Gupta’s presentation in 2004 on “What Makes a Good NGO” has a profound impact on NGO management of their partnership with foreign aid. Gupta’s organization grades NGO performance on its set of common indicators.

26

ignored. These will be inexcusable costly mistakes given the long experience of development assistance since World War II.

The expectation that NGOs should perform IE as in alignment with the bureaucratization and formalization that have taken place or are happening among donors mean that NGOs have to abide by set procedures, or use specific formats starting with grant application. These are so burdensome and require particular expertise that NGOs tend to outsource these tasks or employ former staff of governmental agencies for these onerous responsibilities.

I can attest to the fact that Church World Service rejected seeking development assistance grants from USAID for excellent legitimate reasons. It did not have the staff to manage and oversee grants and it preferred to maintain independence in development implementation. This latter reason also caused the organization not to employ former government officials to handle its grant application and reporting process.

The pressures of accountability and effectiveness have also created a climate of breaking down national problems into small projects. This reality tends to prevent NGOs from working on a broad strategy of accomplishing maximum fundamental impact in poverty alleviation. The shifting donor priorities (e.g. moving to concentrate on climate change from the unfinished battle to conquer TB, malaria, or HIV/AIDS) add not only agenda but also diversion of efforts to root out poverty, the original purpose of ODA. This is true for the entire aid community and not just among NGOs. IE in such a context may be perceived to be superfluous.

In their comprehensive study on the role of NGOs in international development cooperation, John Degngol-Martinussen and Poul Engberg-Pedersen drew four important conclusions:58

1. When everything is taken into consideration, - the size of the problems, the costs etc. – Danish NGOs have done a good job for the immediate benefits of poor target groups and strengthening of partner organizations.

2. NGOs own documentation of achieved results, however, is weak and too narrowly linked to immediate goals. The organizations have done too little to investigate and document things that have functioned well.

3. NGOs had often not been able to plan work strategically and innovatively. They were inclined to do things in the usual way.

4. Efforts were generally too modest to create visible change in society for the lasting benefit of target groups, but it is possible that the Danish organizations had actually contributed to such a development.59

58 Aid: Understanding International Development Cooperation. (London and New York: Zed Books, 2003), 161. 59 The results from the study of assessing the impact of DANIDA’s support to Danish NGOs are available from http://www.um.dk/danida/ngoimpact/synthese_eng

27

Even though the study is based on Danish NGOs, They reconfirm previous studies by DAC in 1997. The conclusions ring true even today. The performance inadequacies in # 2-4 summarise the NGO realities that donors must take into account in setting the parameter and requirements for IE.

I. NGO Political Image Considerations

The final issue is the NGO hesitation to participate in ODA or in entering into any agreement with donors for development assistance. We have noted that donors appreciate the role of NGOs because of their access, skills and the overall cost effectiveness of their operations. Yet the number of NGOs who enter into contractual agreements with governments or multilateral agencies to tackle MDGs is not as many as to be expected. This is extremely regrettable given their ability to access and mobilize communities at the frontline of the MDG battle! Indeed if the MDG goals are to be met, to use a military metaphor, many more frontline troops are necessary to wage a successful war against poverty.

Donors must play their part to facilitate. Nowhere is this facilitation more necessary than addressing the perception that donors are biased in favour of established NGOs and are not sufficiently transparent in the grant award process. Donors tend to allow institutional inertia to set in rather than taking steps to partner with the increasing number of NGOs from Asia, Africa and Latin America which have organised themselves impressively to take on fairly large scale projects nationally. International NGOs with national offices and branches in these regions may be aware of them but they may be reluctant to encourage these Southern NGOs to compete for a shrinking aid pie. These Southern NGOs may raise the issue of whether donors are interested in more progressive aid delivery that they believe they exemplify.

Donors must also be sensitive to the fact that the competition for aid money has led to ill will among new Southern NGOs. They feel not trusted by aid agencies due to an outdated perception about their lack of expertise, poor management and bureaucratization. They strongly suspect too that in the course of routine interaction with donors, traditionally established NGOs tend to reinforce such donor perspectives about them. The climate of distrust and suspicion due largely to limited involvement or exclusion will harm the IE process and raise questions about the credibility of the evaluation outcome.

RECOMMENDATIONS

The recommendations are addressed to policy makers rather than practitioners of IE. The Paris principle of harmonisation of donor terms for aid should be applied to a transparent process of grant making to NGOs among all donors. Such transparency will encourage newly established NGOs to become involved in ODA alongside if not in partnership with established NGOs.

Effective mechanisms are very much needed so that both NGOs and their donors can be held to account in effective aid delivery. They must also be accountable to the recipient countries and

28

the targeted beneficiaries. Therefore IE should be carried out in such a context of accountability among all stakeholders with transparency and openness and with opportunities for full dialogue on setting processes and priorities, and establishing agreement for assessment of progress. In making these recommendations I am moved to recall the observation of Antonio Tujan of the Reality of Aid Organization:

“Valuable resources and opportunities are being frittered away in endless technical debates on aid management when they might improve conditions for poor people to claim their right to education or access to health care, to women’s rights or to decent work.”60

Role of NGOs

NGOs should be involved as a matter of routine to participate in donor IE orientation and training workshops.

Donors should encourage NGOs by convening IE roundtables regularly. These roundtables will review and address the challenges facing the NGOs on IE. The agenda for these meetings should be set in consultation with NGOs.

IE Methodology & Process

The methodology to be employed is not uniform and there are flaws in current practice. Nor is there uniformity in the recommended IE process. These issues should be addressed not unilaterally by an individual agency otherwise the unsatisfactory situation of NGOs having to take on different models will persist. I suggest the following:

Agencies such as the World Bank, DAC and UNDP need to review their IE manuals to develop a section for NGOs. The NGO section can be a set of principles with simplified operational procedures. A roundtable of multilateral agencies, donors and NGOs should be convened for the development of this section.

There are urgent issues concerning the methodology and the IE process:

1. Can donors perform IE for NGOs such as proposed by MCC?

2. To what extent should “impartial” IE be insisted upon or how can it be carried out?

3. How does a full participatory process of all stakeholders ensure the credibility of the IE?

60 Reality of Aid Report. (2007.) 14

29

4. What are the comparative merits of an external independent and impartial evaluation? If this is a non-negotiable component of IE, how does this requirement play out in countries that need approval for proposed international collaborative projects, especially those that include a research component and/or involvement of local populations?

Funding

Donors should be prepared to support NGOs in IE activities. This support should include NGO’s own IE capacity building and their participation in donor orientation seminars or roundtables. The IE cost of any project should be borne by donors as part of the project DIME.

REFERENCES AND NOTES

ACT Alliance, available from http://www.actalliance.org ActionAid (1998) Action Aid Impact Assessment Workshop. London: Action Aid. Adams, J. (2000) The Development of a Methodology to Assess the Impact of Tearfund on its Partners and through them on the Poor. A Paper Outlining the Process and Lessons Learned. Adams, J., Batchelor, S., McKemey, K. and Wallace, I. (1999) Developing a Methodology for Measuring the Impact of an International NGO and its Local Partners. Reading: AERDD. Africa Evaluation Association. Available from http://www.afrea.org Alkire, S. (1997) Impact Assessment: Oxfam Versus Poverty. Oxford: Oxfam. American Evaluation Association, available from www.eval.org Up to date on the latest thinking on monitoring and evaluation with excellent combination of practical issues and theoretical backgrounds. Bebbington, Anthony J., Samuel Hickey and Diana C. Mitlin, Editors. 2008. Can NGOs make a Difference? Bhagwati, Jadish. 2007. In Defense of Globalization. Oxford University Press. Blankenberg, F. (1995) Methods of Impact Assessment Research Programme, Resource Pack and Discussion. The Hague: Oxfam UK/I and Novib. CARE International's Impact Guidelines, available from http://portals.wi.wur.nl/files/docs/ppme/CAREImpactGuidelines.pdf Cohen, L. and L. Manion, (1989) Research Methods in Education. London: Routledge. Degngol-Martinussen, John and Poul Engberg-Pedersen. 2003. Aid: Understanding International Development Cooperation. London and New York: Zed Books. Edwards, M., and Hulme, D. (1995) Non-Governmental Organisations Performance and Accountability Beyond The Magic Bullet . London: Earthscan. Edwards, Michael and Alan Fowler.2002. NGO Management. London: Earthscan Edwards, Michael and David Hulme. 1996. Too close for comfort? The impact of official aid on nongovernmental organizations. World Development 24, FANTA-2 Publications on Assessments, Monitoring and Evaluation. FANTA works with USAID and with food programming in particular, and have developed various DME guides. Monitoring and evaluation skills, such as design, data collection, and data analysis, are core to organizations from both a performance and management standpoint. Available from http://www.fantaproject.org/focus/monitoring.shtml

30

FANTA-2 A Guide to Monitoring and Evaluation of Nutrition Assessment, Education and Counseling of People Living With HIV (2008) FANTA-2 Alternative Sampling Designs for Emergency Settings (2007) FANTA-2 Monitoring and Evaluation Framework for Title II Development-oriented Projects (2006) FANTA-2 Local Capacity Building in Title II Food Security Projects: A Framework (2004) Feurstein, M. T. (1986) Partners in Evaluation: Evaluating Development and Community Programmes with Participants. London: Macmillan. Fisher, Julie. 1993. The Road from Rio: Sustainable Development and the Nongovernmental Movement in the Third World. Praeger. Fowler, A. (1997) Striking a Balance: A Guide to Making NGOs Effective in International Development. London: Earthscan/INTRAC. 298pp. Goyder, H., Davies, R. and Williamson, W. (1997) Participatory Impact Assessment. London: ActionAid Guba, E. and Lincoln, Y. (1989) Fourth Generation Evaluation. London: SAGE. Hallam, H. (1998) Evaluating Humanitarian Assistance Programmes in Complex Emergencies. London: ODI, 126pp. (RRN Good Practice Review No. 7). Hopkins, R. (1995) Impact Assessment Overview And Methods Of Application. Oxford: Oxfam (UK&I) and Novib, 70pp. Howell, Jude. 2001. Civil Society & Development: A Critical Exploration. Boulder, CO: Lynne Reiner. Independent Evaluation Group, available from http://ieg.worldbank.org InterAction International Evaluation Resources, available from http://www.interaction.org/monitoring-evaluation INTRAC (1999) ‘Evaluating Impact: the Search for Appropriate Methods and Instruments.’ In Ontrac, No.12. James, Eric. 2008. Managing Humanitarian Relief: An operational guide for NGOs. Practical Action Publishing. Kharas, Homi. 2007. “The New Reality of Aid.” Brookings Institution. Kruse, S-E., Kylonnen, T. Ojanpera, S. Riddell, R. and Vieljus, J-L. (1997) Searching For Impact and Methods NGO Evaluation Synthesis Study. Finland: Finnish Ministry of Foreign Affairs. (A Report Prepared for The OECD/DAC Expert Group On Evaluation.) Levinson, F. James and Isabel Madzorera. 2005. Recent experience in the monitoring and evaluation of nutrition related projects in developing countries: nine lessons learned. Food Policy and Applied Nutrition Program Discussion Paper No. 28, Gerald J and Dorothy R Friedman School of Nutrition Science and Policy, Tufts University Mercy Corps DME Handbook 2005 Minear, Larry and Hazel Smith, Eds. 2007. Humanitarian Diplomacy: Practitioners and Their Craft. United Nations University Press. NGO Committee on UNICEF. Statement to UNICEF Executive Board, June 2002. NGO Steering Group of NGO Committee on UNICEF. Report of NGO Activities at the United Nations General Assembly Special Session on Children, 2002. Oakley, P. and Clayton, A. (2000) The Monitoring and Evaluation of Empowerment: A Resource Document. Oxford:INTRAC. Oakley, P., Pratt, B. and Clayton, A. (1998) Outcomes and Impact: Evaluating Change in Social Development. Oxford: INTRAC.

31

Reality of Aid Network. The only major North/South international non-governmental initiative focusing exclusively on analysis and lobbying for poverty eradication policies and practices in the international aid regime. It brings together 172 member organizations, including civil society regional and global networks. It has a seventeen-year track record of independent assessment of aid policies and practices, and engages in constructive dialogue with policy makers at national and international levels. Retrieved from http://www.realityofaid.org. Report of the 7th meeting of the DAC Network on Development Evaluation, 20 – 21 February 2008 prepared by the Independent Evaluation Group (IEG) of the World Bank, Impact Evaluation: its Status and its Future. Riddell, R. (1990) Judging Success: Evaluating NGO Approaches To Alleviating Poverty In Developing Countries. London: ODI, 57pp. (ODI Working Paper 37). Roche, C. (1999) Impact Assessment for Development Agencies. Oxford: Oxfam/NOVIB. Ruxton, R. (1995) Participation in Impact Assessment Oxford: OXFAM UK. Save the Children Fund UK (1994) Toolkits: A Practical Guide to Monitoring, Assessment, Review and Evaluation. London: SCF, 354 pp. Development Manual 5. Sphere Project. 2004. Humanitarian Charter and Minimum Standards in Disaster Response. London: Oxfam Tuft's University - discussion paper on monitoring & evaluation. Available from http://nutrition.tufts.edu/docs/pdf/fpan/wp28-mep_nine_lessons.pdf United Nations Development Programme. 2009. Handbook on Planning, Monitoring and Evaluating for Development Results. Available from http://www.undp.org/evaluation/handbook/ This is a reference throughout the programme cycle. The Handbook is about planning, monitoring and evaluating results, not for programme or project management. United Nations Evaluation Group, available from http://www.uneval.org/ First stop for evaluation resources related to United Nations programming. United Way Outcome Measurement Resource Network. Available from http://www.unitedway.org/outcomes Weiss, Thomas G. and Leon Gordenker. 1996. NGOs, the UN and Global Governance. Boulder, Colorado: Lynne Rienner Publishers. World Vision International DME resources. Available from http://www.transformationaldevelopment.org/Ministry/TransDev2.nsf/maindocs/60871459C58610B588256F860053D9DD?opendocument This website includes framework on DME and the tools (processes) for project management phases. Some tools may be useful to context adaptation. World Vision International list of programming resources (which is quite extensive, listed by sector - and provides links to other organizations and resources.) Available from http://www.transformationaldevelopment.org/Ministry/TransDev2.nsf/maindocs/81712E8937293E1E88256F270074C2B6?opendocument World Vision International Integrated Programming Model. Available from http://www.transformationaldevelopment.org/ministry/transdev2.nsf/IPM%20layout%2001d.html This site combines DME with Development theories and promising practices derived from different NGOs. Useful for implementing projects in the field.

32

APPENDIX A

Working with NGOs:

A Practical Guide to Operational Collaboration between the World Bank and Non-Governmental Organizations

I. INTRODUCTION

· NGO is a broad term encompassing a wide array of diverse organizations. · The World Bank defines NGOs as "private organizations that pursue activities to

relieve suffering, promote the interests of the poor, protect the environment, provide basic social services or undertake community development".

· The World Bank collaborates with CBOs, national and international NGOs in a variety of different ways.

· Achieving the full potential benefits of NGO involvement implies enhanced roles for NGOs earlier on in the project cycle.

II.WHY THE WORLD BANK WORKS WITH NGOS

· NGO involvement can contribute to the sustainability and effectiveness of Bank-financed projects by introducing innovative approaches and promoting community participation.

· NGOs can help expand project uptake and reach, and can facilitate greater awareness of diverse stakeholder views.

III.HOW THE WORLD BANK WORKS WITH NGOS

· NGOs are active contributors to the Bank's economic and sector work (ESW) and participate in lending activities from identification through to evaluation.

· Currently, NGO involvement is most frequent during implementation. Evidence shows, however, that NGOs can provide particularly valuable input during project identification and design.

· Where NGOs are expected to participate in implementation, they should also be consulted during design.

IV.KEY ISSUES IN WORKING WITH NGOS

A. Identifying an appropriate NGO partner

· Selecting an appropriate NGO partner involves: i) gathering information about the NGO sector; ii) establishing relevant selection criteria, and; iii) choosing a suitable selection process.

33

· Clear selection criteria should be established based on specific project needs. · Organizational capacity should be assessed according to an NGO's proven track

record, not its stated goals. · It is important to identify "capacity-building" needs and strategies. · Maximum transparency should be ensured in the selection process.

B. Time issues

· Be prepared for possible extra staff/time needs early on in the project cycle. · Be aware of potential time fags/delays and the risk they pose to participatory

processes. · Ensure NGOs have an adequate understanding of project time-frame and

deadlines.

C. Flexibility issues

· Seek NGO/community input early on in the project cycle. · Establish clear mechanisms for responding to local views and needs. · Use mid-term review process to adapt project priorities/processes as necessary

according to local input. · In areas where NGOs have a recognized comparative advantage, take steps to

maximize their institutional autonomy.

D. Funding issues

· NGOs are generally cost-effective. They should not, however, be viewed as a "low-cost alternative" to other types of implementing agencies.

· NGOs should not be expected to provide services free of charge or at lower than market rates (unless according to a co-financing agreement).

· Clarify the expected status of NGO involvement (e.g.: informal unpaid advisor, paid consultant, contractor, etc.) from the outset.

· Establish mutually acceptable fees and overhead costs.

E. Procurement and disbursement issues

· Use simplified bidding documents, where appropriate. · Consider the use of alternative procurement practices. · If necessary, make provisions for advance payments. · Provide training for NGOs in procurement and disbursement procedures.

F. NGO-Government relations

· Consider conducting a state-NGO relations study. · Keep in mind that government-NGO collaboration is not possible/ desirable in all

cases.

34

· Always seek government-NGO complementarity. · Understand how government policies influence the NGO sector and, where

possible, promote an enabling environment for NGOs.

G. Importance of clearly defined roles and responsibilities

· Share all relevant project documents with participating NGOs. · Consider the appointment of an NGO liaison officer. · Organize a government-Bank-NGO information-sharing workshop.

H. Contractual/legal issues

· Adapt standard contract agreements as necessary to meet specific needs of NGOs/community groups.

· Write contracts in simple language. · Consider using a Memorandum of Understanding or other alternative form of

contract.

I. Capacity-building

· Consult with NGOs on appropriate strategies to support their institutional development.

· Where appropriate, build a training component for NGOs into project design. · Encourage partnerships between international and local NGOs. · Promote networking and information-sharing among NGOs.

Source:

World Bank, Working with NGOs A Practical Guide to Operational Collaboration between the World Bank and Non-Governmental Organizations. Operations Policy Department, World Bank, 1995, pp.7-9.

35

APPENDIX B

AUSTRALIA: AusAid

AUSTRIA: Foreign Ministry

AUSTRIA: Austrian Development Agency (ADA)

BELGIUM: Development Cooperation (DGDC)

BELGIUM: Technical Cooperation (BTC)

CANADA: Canadian International Development Agency (CIDA)

DENMARK: Ministry of Foreign Affairs

EUROPEAN COMMISSION: DG Development

FINLAND: Ministry of Foreign Affairs

FRANCE: Ministry of Foreign Affairs

FRANCE: Le Groupe de l'Agence française de Développement (AfD)

GERMANY: BMZ

GERMANY: GTZ

GERMANY: KfW

GREECE: Ministry of Foreign Affairs

IRELAND: Irish Aid

ITALY: Ministry of Foreign Affairs

JAPAN: Ministry of Foreign Affairs (MOFA)

JAPAN: Japan International Cooperation Agency (JICA)

JAPAN: Japan Bank for International Cooperation (JBIC)

KOREA: Ministry of Foreign Affairs and Trade

LUXEMBOURG: Ministry of Foreign Affairs

LUXEMBOURG: Lux-Development

NETHERLANDS: Ministry of Foreign Affairs

NEW ZEALAND: NZAid

NORWAY: Ministry of Foreign Affairs

NORWAY: Norwegian Agency for Development Cooperation (NORAD)

PORTUGAL: Ministry of Foreign Affairs

PORTUGAL: Portuguese Institute for Development Support

SPAIN: Spanish Agency for International Development (AECID)

SWEDEN: Sida

SWITZERLAND: Swiss Agency for Development and Cooperation (SDC)

SWITZERLAND: State Secretariat for Economich Affairs (SECO)

UNITED KINGDOM: Department for International Development (DFID)

UNITED STATES: United States Agency for International Development (USAID)

UNITED STATES: Millennium Challenge Corporation (MCC)

36

APPENDIX C

29 cf4a75e2536704a search/results 1

Impact Evaluation

· Overview · Conducting Evaluations · Evaluations by Sector · Evaluations by Country · Evaluation Data

MCC is committed to conducting independent impact evaluations of its programs as an integral part of its focus on results. These rigorous assessments of project impact often enhance the design of programs, provide critical information regarding the performance of specific activities, and contribute to a broader understanding of development effectiveness.

The Essence of the Impact Evaluation

An impact evaluation measures the changes in individual, household or community income and well-being that result from a particular project or program. The distinctive feature of an impact evaluation is the use of a counterfactual, which identifies what would have happened to the beneficiaries absent the program. This counterfactual is critical to understanding the improvements in people’s lives that are directly caused by the program.

Impact evaluation at MCC is part of a broader monitoring and evaluation (M&E) approach that covers all MCC-funded programs. Performance monitoring activities measure implementation progress and intermediate results, while impact evaluations are designed to measure the impact of projects on the wellbeing of beneficiaries.

Independent Assessments

MCC uses teams of independent professional researchers to carry out impact evaluations. These teams are selected in a competitive process that includes some of the world’s most experienced and respected impact evaluation specialists. MCC’s use of independent professionals is intended

37

to ensure that the impact evaluations represent an unbiased assessment of the activities being studied.

How MCC Impact Evaluations Contribute to Development

MCC expects that the results of its impact evaluations will help guide future investment decisions. These impact evaluations further the work of the MCC and the development field, in general, in a number of ways:

· Impact evaluations help MCC decide whether to expand a particular type of program or avoid it in the future. Projects that have a large positive impact on beneficiaries can be expanded or become models for future MCC projects in similar settings.

· Impact evaluations help MCC economists enhance the precision of their future calculations of economic rates of return (ERRs).

· Impact evaluations contribute to the global academic discourse on what works in the development field. MCC will make the results of its impact evaluations publicly available to be used by other donors, researchers and non-governmental organizations.