the role of evaluation in managing for results · practical tool: results chains focusing on...
TRANSCRIPT
The Role of Evaluationin
Managing for Results
Scott Bayley
Department of Foreign Affairs & Trade
Overview
1. What is managing for results?
2. Developing an evaluation policy
3. How to manage for results
4. The use of M&E by UN agencies
5. Performance feedback is not enough
6. How leadership can support evaluation and RBM
1. What is it?
Managing for results is a management strategy that focuses on using performance information to improve decision-making. It involves using practical tools for strategic planning, program design, risk management, M&E in order to achieve outcomes and demonstrate accountability.
what is it? …
At its core are notions of:
• Goal-orientedness: setting clear goals and results provides targets for change and opportunity to assess progress.
• Causality: inputs and activities lead logically to desired outputs, outcomes and impact; also called the ‘results chain’.
• Continuous improvement: monitoring and evaluating program results provides the basis for adjustments to keep programs on track and to maximize results.
2. Developing Evaluation Policy
An evaluation policy should contain:• A clear explanation of the concept and role of evaluation
• General evaluation standards
• Definitions of the institutional framework, roles, responsibilities & linkages to monitoring, planning, policymaking budgeting, and reporting.
• Complementary evaluation guidelines to operationalize the policy
• A description of how evaluations are to be planned and prioritized
• How evaluations are organized, managed and funded
• The mechanism for the follow-up of evaluation findings and recommendations
• The disclosure and dissemination procedures for evaluation.
From United Nations Office of Internal Oversight Services
evaluation policy…
Features of an effective evaluation system:
• The production of (suitable) quality performance information in a cost-effective manner
• A high level of utilization
• Linking M&E into planning, budgeting, policy making, & reporting / accountability processes institutionalization sustainability over time as governments & officials change.
See World Bank’s website re: evaluation capacity building strategies
3. How do you do manage for results?
To produce real results in the public sector requires:
1) Developing a consensus among relevant stakeholders about what needs to be accomplished;
2) Designing and implementing programs that have the potential to achieve the intended outcomes
3) M&E the implementation and performance of these programs from a variety of value perspectives (using a program logic model/theory of change);
how to do it…
4) Using this performance feedback to stimulate higher performance;
5) Communicating the value of these programs to policy makers, to those who influence resource allocations, and to others who have a stake in the effectiveness of public sector programs.
Practical Tool: results chains
Focusing on Results (outputs, outcomes,
impacts)
Spelling out the expected results
chain
M&E and reporting these results
Using this info in decision making
1.
2.
3.
Strategy/program/project
Managing Using M&E + Logic Models
Deliver
apples
Apples
eaten
Increased
Vit C
levels
Fewer
episodes
of colds
and flu
x x x x
Helpful resources:
Funnell & Rogers, 2011, Purposeful Program Theory,
Jossey-Bass.
CAVAC website: www.cavackh.org x
4. The Use of M&E by UN Agencies
Studies in 2007/2012 found that:
• RBM has given a ‘fresh lease of life’ to the evaluation function in many UN agencies.
• This has provided evaluation offices with a new opportunity to promote the merits of evaluation.
UN agencies …
• RBM has been adopted as the broad management strategy within most UN agencies since the late 1990s/early 2000s.
• As at 2007 only about 1/3 of agencies have established comprehensive RBM frameworks, but by 2012 significant progress has been achieved.
UN agencies …
• RBM and evaluation are perceived by many managers as a corporate reporting and accountability exercise rather than as a tool for operational management, $ under investment!
• The monitoring of results and the use of performance information to improve results were the weakest aspects of RBM in the agencies studied.
5. Performance Feedback is Not Enough
(to drive service improvements)
i. Technical challenges
ii. Financial challenges
iii. Institutional challenges
iv. Political challenges.
feedback is not enough …
6. How Leadership Can Support RBM1. By making statements advocating the benefits of RBM.
2. By consistently modelling/demonstrating a commitment to RBM in their own work. Leaders demand performance information and are able to make use of it in their own work.
3. Through their support for organizational RBM policies, systems, guidelines, tools, and actual practices/culture.
4. By championing an explicit RBM change program that is adequately resourced, implemented and monitored/evaluated.
5. By initiating engagement with key external stakeholders and building a consensus about performance expectations.
6. By publically participating in RBM training and coaching activities.
7. By taking RBM considerations into account when making resource allocation decisions.
8. Through the questions that senior leaders ask of staff and colleagues at meetings. Do conversations focus on undertaking activities/disbursing $ or on achieving/improving results?
9. Through their expectations of their own staff.
10. By emphasizing learning and improvement, enhancing organizational capacity and systems while actively encouraging performance discussions.
11. By their reaction to problems and crises.
12. Through who they employ and their application of rewards, promotions, and sanctions.
13. By constructively managing ‘resistance’ to RBM and supporting units and individuals to make the transition.
Summary
1. What is managing for results?
2. Developing an evaluation policy
3. How to manage for results
4. The use of M&E by UN agencies
5. Performance feedback is not enough
6. How leadership can support evaluation and RBM
Suggested ReadingBehn, (various) ‘Why Measure Performance?’ ‘Measurement is Rarely Enough’,
‘The Magical Performance System’. See his website:
http://www.hks.harvard.edu/thebehnreport/
Brinkerhoff, 1991, Improving Development Program Performance, Lynne Rienner
Carter et al, 1992, How Organizations Measure Success, Routledge
Funnell & Rogers, 2011, Purposeful Program Theory, Jossey-Bass
Mayne, 2007, Best Practices in Results-based Management: A Review of Experience, UN
Perrin, 1998, Effective Use and Misuse of Performance Information, American Journal of Evaluation, vol 19, no 3, pps 367-379
Thomas, 2006, Performance Measurement, Reporting, Obstacles and Accountability; ANU Press
UNDESA, 2012, Results-Based Management in the UN Development System: Progress and Challenges
UN Development Group, 2011, Results-Based Management Handbook
UNEG, 2007, The Role of Evaluation in Results-Based Management.
