leveraging learning in evaluation
TRANSCRIPT
-
8/8/2019 Leveraging Learning in Evaluation
1/32
The views expressed in this presentation are the views of the author/s and do not necessarily reflect the views or policies of the AsianDevelopment Bank, or its Board of Governors, or the governments they represent. ADB does not guarantee the accuracy of the data includedin this presentation and accepts no responsibility for any consequence of their use. The countries listed in this presentation do not imply anyview on ADB's part as to sovereignty or independent status or necessarily conform to ADB's terminology.
Leveraging Learning
in Evaluation
Olivier Serrat
2010
-
8/8/2019 Leveraging Learning in Evaluation
2/32
Overview
Introduction to monitoring and evaluation, and itsresults chain and life cycle
Introduction to independent evaluation
Distinction between evaluation for accountabilityand evaluation for learning
Considerations in evaluation capacitydevelopment
Description of areas of competence for knowledge
management and learning Overview of tools for evaluation for learning
-
8/8/2019 Leveraging Learning in Evaluation
3/32
Monitoring is the continuous collection of data andinformation on specified indicators to assess the
implementation of a development intervention in relationto activity schedules and expenditure of allocated funds,and progress and achievements in relation to its
intended outcome. It involves day-to-day follow-up of activities during
implementation to measure progress and identifydeviations.
It requires routine follow-up to ensure activities areproceeding as planned and are on schedule.
It needs continuous assessment of activities andresults answers the question, "what are we doing?"
Monitoring and Evaluation
-
8/8/2019 Leveraging Learning in Evaluation
4/32
Evaluation is the periodic assessment of the design,implementation, outcome, and impact of a development
intervention. It should assess the relevance andachievement of the intended outcome, the performance ofimplementation in terms of effectiveness and efficiency,and the nature, distribution, and sustainability of impact.
It is a systematic way oflearning from experience toimprove current activities and promote better planningfor future action.
It is designed specifically with the intention to attributechanges to the intervention itself.
It answers the question, "what have we achieved andwhat impact have we had?"
Monitoring and Evaluation
-
8/8/2019 Leveraging Learning in Evaluation
5/32
The Results Chain Outputs
Outcome
Impact
ActivitiesInputs
Monitoring and Evaluation
-
8/8/2019 Leveraging Learning in Evaluation
6/32
The Results Chain Explained
Needs
Objective I nputs Activities Outputs
Outcome
Impact
Monitoring and Evaluation
Re l e v a n ce Ef f i ci e n cy
E f f e c t i v e n e s s
S u s t a i n a b i l i t y
-
8/8/2019 Leveraging Learning in Evaluation
7/32
-
8/8/2019 Leveraging Learning in Evaluation
8/32
Challenges and Limits to Management
LogicDegree ofControl
Challenge ofMonitoring
and
Evaluation
Impact
What the
development
intervention is
expected to
contribute to
OutcomeWhat the
developmentintervention
can be
expected toachieve and
be
accountablefor
Outputs What is withinthe direct
control of the
developmentintervention'smanagement
Activities
Inputs
De
creasing
Control
Inc
reasing
Difficulty
Monitoring and Evaluation
-
8/8/2019 Leveraging Learning in Evaluation
9/32
-
8/8/2019 Leveraging Learning in Evaluation
10/32
1. Adhere to strategic principles.
2. Sharpen evaluation strategies.
3. Distinguish recommendation typologies.
4. Make better recommendations.5. Report evaluation findings.
6. Track action. on recommendations.
Increasing Value Added
from Independent Evaluation
-
8/8/2019 Leveraging Learning in Evaluation
11/32
Provides for control of the organization.
Implies lack of utility.
Diverts resources.
Focuses on justification rather than improvement. Distorts program activities.
May promote incentive to lie, cheat, and distort.
Misplaces accent on control.
Emphasizes results orientation while maintaining
traditional checks on use of inputs andcompliance with procedures.
Some Criticisms of
Evaluation for Accountability
-
8/8/2019 Leveraging Learning in Evaluation
12/32
Learning is the acquisition of know ledge or skills
through instruction, study, and experience.
Learning is driven by organization, people, know ledge,
and technology working in harmonyurging better and
faster learning, and increasing the relevance of an
organization.
Learning is an integral part of know ledge management
and its ultimate end.
Data WisdomInformation Know ledge
Know WhyKnow HowKnow What
Reductionist Holistic
Learning
Systemic
-
8/8/2019 Leveraging Learning in Evaluation
13/32
Learning from Experience
Evaluation fromlearning canserve as animportantfoundation block
of a learningorganization.
Researchers
now recognize itas the greatestneed today and
tomorrow.
-
8/8/2019 Leveraging Learning in Evaluation
14/32
Item Evaluation forAccountability
Evaluation for Learning
Basic Aim The basic aim is to find outabout the past.
The basic aim is to improvefuture performance.
Emphasis Emphasis is on the degreeof success or failure.
Emphasis is on the reasonsfor success or failure.
Favored by Parliaments, treasuries,media, pressure groups
Development agencies,developing countries,research institutions,
consultantsSelection ofTopics
Topics are selected basedon random samples.
Topics are selected fortheir potential lessons.
Status ofEvaluation
Evaluation is an endproduct.
Evaluation is part of theproject cycle.
Evaluation for Accountability and
Evaluation for Learning
-
8/8/2019 Leveraging Learning in Evaluation
15/32
Item Evaluation forAccountability
Evaluation for Learning
Status ofEvaluators
Evaluators should beimpartial and independent.
Evaluators usually includestaff members of the aidagency.
Importance
of Data fromEvaluations
Data are only one
consideration.
Data are highly valued for
the planning andappraising of newdevelopment activities.
Importance
of Feedback
Feedback is relatively
unimportant.
Feedback is vitally
important.
Source: Adapted from Cracknell, B. 2000. Evaluating Development Aid: Issues, Problems, and Solutions. East Sussex: Sage Publications.
Evaluation for Accountability and
Evaluation for Learning
-
8/8/2019 Leveraging Learning in Evaluation
16/32
Stage Category
Preparation Failures of intelligence: not knowing enough at the earlystages of project formulation, resulting in crucial aspects of
the project context being ignored.
Failures of decision making: drawing false conclusions ormaking wrong choices from the data that are available, andunderestimating the importance of key pieces ofinformation.
Implementation Failures of implementation: bad or inadequate managementof one or more important aspects of the project.
Failures of reaction: inability or unwillingness to modify theproject in response to new information or changes in
conditions that come to light as the project proceeds.
Evaluation Failures of evaluation: not paying enough attention to theresults.
Failures of learning: not transferring the lessons into future
plans and procedures.
Source: Adapted from Nolan, R. 2002. Development Anthropology: Encounters in the Real World. Boulder, Colorado: Westview Press.
Types of Learning Failure
-
8/8/2019 Leveraging Learning in Evaluation
17/32
Who Should BeLearning?
What ShouldThey Be Learning?
Field Staff Participation in practice
Effective empowerment
Local-level collaboration with government and othernongovernment organizations
Gender dimensions of local developmentTechnicalSpecialists
Good practice in their area of expertise
Ways of integrating with other disciplines
How to improve cost-effectivenessHow existing internal and external policies affectperformance
Operational
Managers
What factors make interventions and projects work well or
badly, for example, funding conditionsHow to be more cost-effective
How to coordinate internally and externally
Learning in Nongovernment
Organizations: Food for Thought
-
8/8/2019 Leveraging Learning in Evaluation
18/32
Who Should BeLearning?
What ShouldThey Be Learning?
Fund-RaisersandDevelopmentEducationalists
Principles and insights to be used in negotiation withprofessional donorsNew messages to get across to private contributorsExamples of impact and what made things work or fail
Leaders How policy choices and strategies work out in practice
How to make external relationships more effective
How best to exert influence
What environmental factors have had unforeseen effects andmust be taken into account
Governors The quality and costs of donors
The degree of stakeholder satisfaction
Consistency between mission, strategy, and impact
Improving social standing and credibility of the organizationSource: Adapted from Fowler, A. 1997. Striking a Balance: A Guide to Enhancing the Effectiveness of Non-Governmental Organizations in InternationalDevelopment. London: Earthscan.
Learning in Nongovernment
Organizations: Food for Thought
-
8/8/2019 Leveraging Learning in Evaluation
19/32
-
8/8/2019 Leveraging Learning in Evaluation
20/32
Evidence PersuasiveArgument
Clear purposeCohesive argumentQuality of evidence
Transparency of evidence underpinning policyrecommendations (e.g., a single study or asynthesis of available evidence)
Authority Clear purpose
Cohesive argumentQuality of evidence
Transparency of evidence underpinningrecommendations (e.g., a single study or asynthesis of available evidence)
Context AudienceContextSpecificity
Addresses the specific context (e.g., national,local)
Addresses the needs of target audience (e.g.,social, economic policy)
ActionableRecommendations
Information linked to specific processesClear and feasible recommendations on steps tobe taken
Making Evaluation Reports
Effective
-
8/8/2019 Leveraging Learning in Evaluation
21/32
Engagement Presentationof Evidence-Informed
Opinions
Presentation of author's own views about theimplications of findings
Clear identification of argument components that
are opinion based
ClearLanguageand Writing
Style
Easily understood by educated nonspecialists
Appearanceand Design
Visually engaging
Presentation of information through charts,graphs, and photographs
Source: Adapted from Jones, N., and C. Walsh. 2008. Policy Briefs as a Communication Tool for Development Research. Overseas DevelopmentInstitute Background Note. May. Available: www.odi.org.uk/publications/background-notes/0805-policy-briefs-as-a-communication-tool.pdf
Making Evaluation Reports
Effective
http://www.odi.org.uk/publications/background-notes/0805-policy-briefs-as-a-communication-tool.pdfhttp://www.odi.org.uk/publications/background-notes/0805-policy-briefs-as-a-communication-tool.pdfhttp://www.odi.org.uk/publications/background-notes/0805-policy-briefs-as-a-communication-tool.pdfhttp://www.odi.org.uk/publications/background-notes/0805-policy-briefs-as-a-communication-tool.pdf -
8/8/2019 Leveraging Learning in Evaluation
22/32
Capacity is the ability of people, organizations, andsociety to manage their affairs successfully.
Capacity to undertake effective monitoring andevaluation is a determining factor of aideffectiveness.
Evaluation capacity development is the process ofreinforcing or establishing the skills, resources,structures, and commitment to conduct and usemonitoring and evaluation over time.
Evaluation Capacity Development
-
8/8/2019 Leveraging Learning in Evaluation
23/32
Stronger evaluation capacity will help development agencies to
Develop as a learning organization
Take ownership of their visions for poverty reduction, if theevaluation vision is aligned with that
Profit more effectively from formal evaluations
Make self-evaluations an important part of their activities Focus on quality improvement efforts
Increase the benefits and decrease the costs associated withtheir operations
Augment their ability to change programming midstream andadapt in a dynamic, unpredictable environment
Build evaluation equity, if they are then better able to
conduct more of their own self-evaluation, instead of hiringthem out
Shorten the learning cycle
Evaluation Capacity Development
-
8/8/2019 Leveraging Learning in Evaluation
24/32
In starting to develop evaluation capacity internally,consider key decisions in
Architecture Locating and structuring evaluationfunctions and their coordination.
Strengthening Evaluation Demand Ensuringthat there is an effective and well-manageddemand for evaluations.
Strengthening Evaluation Supply Makingcertain that the skills and competencies are in placewith appropriate organizational support.
Institutionalizing Evaluations Buildingevaluation into policy-making systems.
Evaluation Capacity Development
-
8/8/2019 Leveraging Learning in Evaluation
25/32
Strategy Development
A strategy is a long-term plan of action to achieve aparticular goal.
Management Techniques
Leadership is the process of working out the right thingsto do. Management is the process of doing thingsright.
Collaboration Mechanisms
When working with others, efforts sometimes turn out tobe less than the sum of the parts. Too often, not enoughattention is paid to facilitating effective collaborative
practices.
Competencies for Know ledge
Management and Learning
-
8/8/2019 Leveraging Learning in Evaluation
26/32
Know ledge Sharing and Learning
Two-way communication that take placesimply and effectively build knowledge.
Know ledge Capture and Storage
Know ledge leaks in various ways at various times.
Competencies for Know ledge
Management and Learning
-
8/8/2019 Leveraging Learning in Evaluation
27/32
-
8/8/2019 Leveraging Learning in Evaluation
28/32
Tools for Evaluation for Learning
Conducting After-Action Reviews
Posting Research Online
Storytelling Identifying and Sharing Good
Practices
Disseminating Knowledge
Products
Learning from Evaluation
Asking Effective Questions
Embracing Failure
Enriching Policy with Research
Know ledge Sharing andLearning
Conducting Exit Interviews
Monthly Progress Notes
Assessing the Effectiveness ofAssistance in CapacityDevelopment
Showcasing Knowledge
Harvesting Knowledge
The Critical Incident Technique
Know ledge Capture andStorage
-
8/8/2019 Leveraging Learning in Evaluation
29/32
Level StrategyDevelopment
ManagementTechniques
CollaborationMechanisms
KnowledgeSharing and
Learning
KnowledgeCapture and
Storage
5
4
3
2
1
Organizational Competencies for
Know ledge Management and Learning
-
8/8/2019 Leveraging Learning in Evaluation
30/32
Source: Adapted from ScanteamInternational. 1993. Internal Learningfrom Evaluations and Reviews. ReportNo. 193. Oslo: Royal NorwegianMinistry of Foreign Affairs.
Learning from Evaluation
How is learninggenerated in the
organization?
Are users
capable of usingthe information?
Is theinformation
easily accessible?
Are usersinvolved in
generating theinformation?
Is the information
used in decisions?
Is the information channeledto the right people
at the right time?
Is the information relevant
and useful?
How is the need for theinformation generated?
Are users opento the
information?
-
8/8/2019 Leveraging Learning in Evaluation
31/32
Monitoring and evaluation are closely interrelated.
The role of independent evaluation has grown inrecent years. It is now the focus of efforts to raise itscontribution to development effectiveness.
Evaluation for accountability should be supplemented
by evaluation for learning. Many shortcomings stem from learning failures.
Evaluation capacity development is an intuitive area
for investment. Notions of knowledge management have a role to play
in evaluation.
Key Learning Points
-
8/8/2019 Leveraging Learning in Evaluation
32/32
Olivier D. Serrat
Principal Knowledge Management Specialist
Knowledge Management Center
Regional and Sustainable Development Department
Asian Development [email protected]
www.adb.org/knowledge-management/
Know ledge Management Center
mailto:[email protected]://www.adb.org/knowledge-management/http://www.adb.org/knowledge-management/mailto:[email protected]