evaluating impact: lessons learned from measure evaluation
DESCRIPTION
During a September presentation at South Africa’s Department of Planning, Monitoring and Evaluation, Dr. Jason Smith shared experiences and lessons learned on evaluating impact from MEASURE Evaluation Phase III implementationTRANSCRIPT
![Page 1: Evaluating Impact: Lessons Learned from MEASURE Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022081401/559eb4701a28abf16a8b485d/html5/thumbnails/1.jpg)
Evaluating Impact: Lessons Learned
from MEASURE Evaluation
Jason B Smith, PhD, MPHDirector for Health Systems Strengthening
![Page 2: Evaluating Impact: Lessons Learned from MEASURE Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022081401/559eb4701a28abf16a8b485d/html5/thumbnails/2.jpg)
Presentation Overview
• What is Evaluation?
• What is MEASURE Evaluation?
• Lessons learned from Phase 3
• Special topics:
– Strategic Information for South Africa (MEval-SIFSA)
– GEMNET-Health and University of Pretoria
– Evaluation Resources
• Questions, Comments and Dialogue
![Page 3: Evaluating Impact: Lessons Learned from MEASURE Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022081401/559eb4701a28abf16a8b485d/html5/thumbnails/3.jpg)
What is Evaluation anyway?
• evaluation [e-val″u-a´shun]
– a critical appraisal or assessment;
– a judgment of the value, worth, character, or effectiveness of
something;
– measurement of progress.
From : The Free Dictionary (online)
![Page 4: Evaluating Impact: Lessons Learned from MEASURE Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022081401/559eb4701a28abf16a8b485d/html5/thumbnails/4.jpg)
Evaluation: USAID Policy
• “Evaluation is the systematic collection and analysis of
information about the characteristics and outcomes of
programs and projects as a basis for judgments, to
improve effectiveness, and/or inform decisions about
current and future programming”
USAID Evaluation Policy, January 2011
“The two purposes of evaluation are to provide information
for decision-making and contextual learning, and to
demonstrate accountability for resources”
Evaluation of USAID, November 2013
![Page 5: Evaluating Impact: Lessons Learned from MEASURE Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022081401/559eb4701a28abf16a8b485d/html5/thumbnails/5.jpg)
Why Evaluate Programs?
• Determine if an intervention was delivered as per plan
• Discover if the intervention had the desired effect(s), and
• Explore whether there are ways that the intervention
could be delivered faster or better or at lower cost
![Page 6: Evaluating Impact: Lessons Learned from MEASURE Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022081401/559eb4701a28abf16a8b485d/html5/thumbnails/6.jpg)
Basic Components
The basic components of evaluation are:
• identifying the parameters
• developing criteria
• data gathering
• measuring the data against the criteria, and
• employing the results
From : The Free Dictionary (online)
![Page 7: Evaluating Impact: Lessons Learned from MEASURE Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022081401/559eb4701a28abf16a8b485d/html5/thumbnails/7.jpg)
Types of Evaluation Research
• Impact
• Outcome
• Public Health
• Methodological Studies
• Population-based Surveys and Surveillance
![Page 8: Evaluating Impact: Lessons Learned from MEASURE Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022081401/559eb4701a28abf16a8b485d/html5/thumbnails/8.jpg)
What is Measure Evaluation?
• MEASURE Evaluation is the USAID Global Health
Bureau’s primary vehicle for supporting improvements in
monitoring and evaluation in population, health and
nutrition worldwide.
![Page 9: Evaluating Impact: Lessons Learned from MEASURE Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022081401/559eb4701a28abf16a8b485d/html5/thumbnails/9.jpg)
MEASURE Evaluation Timeline
• 1991-1997 - The Evaluation Project
• 1997-2003 - MEASURE Evaluation Phase 1
• 2003-2008 - MEASURE Evaluation Phase 2
• 2008-2014 - MEASURE Evaluation Phase 3
• 2014-2019 - MEASURE Evaluation Phase 4
![Page 10: Evaluating Impact: Lessons Learned from MEASURE Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022081401/559eb4701a28abf16a8b485d/html5/thumbnails/10.jpg)
The MEASURE Evaluation Team
![Page 11: Evaluating Impact: Lessons Learned from MEASURE Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022081401/559eb4701a28abf16a8b485d/html5/thumbnails/11.jpg)
![Page 12: Evaluating Impact: Lessons Learned from MEASURE Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022081401/559eb4701a28abf16a8b485d/html5/thumbnails/12.jpg)
MEASURE Evaluation Phase III
Evaluation Activities
North Atlantic Ocean
Indian Ocean
Arctic Ocean
Arctic Ocean
North Pacific Ocean
Nepal
Nigeria
Senegal
Ghana
Ethiopia
South Africa
Madagascar
Mali
Tanzania
Benin
Mozambique
Kenya
India Bangladesh
MalawiAngola Zambia
Rwanda
Uganda
Guatemala
MexicoHaiti
Ukraine
![Page 13: Evaluating Impact: Lessons Learned from MEASURE Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022081401/559eb4701a28abf16a8b485d/html5/thumbnails/13.jpg)
MEASURE Evaluation
Guiding Principles for Evaluations
• Planning in parallel with program implementation.
• Work with country programs as well as global partners
• Partnership with local institutions to build their capacity
• Where appropriate, cost components are included
• Dissemination and use of study results incorporated from
the outset.
![Page 14: Evaluating Impact: Lessons Learned from MEASURE Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022081401/559eb4701a28abf16a8b485d/html5/thumbnails/14.jpg)
MEASURE Evaluation
Criteria for prioritizing evaluation activities (1)
• Comparative advantage
• Theory driven
• Clearly adds to evidence base
• Produces actionable finding
• Reasonable relative cost
• Balance of cost with ability to achieve objectives
![Page 15: Evaluating Impact: Lessons Learned from MEASURE Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022081401/559eb4701a28abf16a8b485d/html5/thumbnails/15.jpg)
MEASURE Evaluation
Criteria for prioritizing evaluation activities (2)
• Incorporates capacity building, data use, and knowledge
management principles
• Follows protection of human subjects principles
• Generalizability of results
• Other pragmatic elements
![Page 16: Evaluating Impact: Lessons Learned from MEASURE Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022081401/559eb4701a28abf16a8b485d/html5/thumbnails/16.jpg)
![Page 17: Evaluating Impact: Lessons Learned from MEASURE Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022081401/559eb4701a28abf16a8b485d/html5/thumbnails/17.jpg)
Lessons Learned: Ongoing Challenges
• Centralized evaluation agendas hard to implement
• Evaluations are slow – lose momentum and visibility
• Need to improve publication, communication and data use
• Designs are compromised by field realities – “real world” evaluation
• “Evaluation” means different things to different people
![Page 18: Evaluating Impact: Lessons Learned from MEASURE Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022081401/559eb4701a28abf16a8b485d/html5/thumbnails/18.jpg)
Lessons Learned: Implementing Evaluation
• Begin with the end in mind (but easier said than done)
• Timing is everything
• Size matters
• Pressure for rapid results to inform programs now
• Be prepared to manage expectations
• The need for ownership
Archive of Phase 3 End of Project Event Sessions
www.measureevaluation.org/eop
![Page 19: Evaluating Impact: Lessons Learned from MEASURE Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022081401/559eb4701a28abf16a8b485d/html5/thumbnails/19.jpg)
A Message of Hope
![Page 20: Evaluating Impact: Lessons Learned from MEASURE Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022081401/559eb4701a28abf16a8b485d/html5/thumbnails/20.jpg)
Special Topics
• Strategic Information for South Africa (MEval – SIfSA)
• GEMNet-Health and University of Pretoria
• Evaluation Resources
![Page 21: Evaluating Impact: Lessons Learned from MEASURE Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022081401/559eb4701a28abf16a8b485d/html5/thumbnails/21.jpg)
Current Associate Awards
![Page 22: Evaluating Impact: Lessons Learned from MEASURE Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022081401/559eb4701a28abf16a8b485d/html5/thumbnails/22.jpg)
MEASURE Evaluation in South Africa
Strategic Information for South Africa (MEval–SIFSA)
Supports the Department of Health (DoH) at national,
provincial and district levels to improve strategic
information for evidence-based management of HIV and
related health programs.
MEval-SIFSA seeks to ensure that the health system in
South Africa produces useful and high quality information
that contributes to sustainable policy, planning, and
programmatic decision making.
![Page 23: Evaluating Impact: Lessons Learned from MEASURE Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022081401/559eb4701a28abf16a8b485d/html5/thumbnails/23.jpg)
SIfSA Strategic Approaches
• System-wide approach
• Leveraging technology
• Data use
• Commitment to sustainability
![Page 24: Evaluating Impact: Lessons Learned from MEASURE Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022081401/559eb4701a28abf16a8b485d/html5/thumbnails/24.jpg)
![Page 25: Evaluating Impact: Lessons Learned from MEASURE Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022081401/559eb4701a28abf16a8b485d/html5/thumbnails/25.jpg)
University of Pretoria
• MEASURE Evaluation Regional Training
Partner since 1999
• Pre-service Education in M&E at MPH level
• In-Service Workshops
• Provision of Technical Assistance to others
• Support for Capacity-Building under
MEval-SIfSA
• Founding member of GEMNet-Health
![Page 26: Evaluating Impact: Lessons Learned from MEASURE Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022081401/559eb4701a28abf16a8b485d/html5/thumbnails/26.jpg)
![Page 27: Evaluating Impact: Lessons Learned from MEASURE Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022081401/559eb4701a28abf16a8b485d/html5/thumbnails/27.jpg)
Acknowledgements
This presentation is not all of my own original thinking. A
number of my colleagues have contributed to this
presentation in person, in print or by providing presentation
materials. These colleagues include Sian Curtis, Jim,
Thomas, Heidi Reynolds, Tara Nutley, Peter Lance,
Gustavo Angeles, Derek Kunata and Hemali Kulatilaka. My
apologies to anyone I may have forgotten.
![Page 28: Evaluating Impact: Lessons Learned from MEASURE Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022081401/559eb4701a28abf16a8b485d/html5/thumbnails/28.jpg)
MEASURE Evaluation is funded by the U.S. Agency for
International Development (USAID) through cooperative
agreement GPO-A-00-03-00003-00 and is implemented
by the Carolina Population Center at the University of
North Carolina at Chapel Hill, in partnership with Futures
Group, ICF International, John Snow, Inc., Management
Sciences for Health, and Tulane University. The opinions
expressed in this presentation are those of the authors
and do not necessarily reflect the views of USAID or the
U.S. government.
Visit us online at http://www.cpc.unc.edu/measure
![Page 29: Evaluating Impact: Lessons Learned from MEASURE Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022081401/559eb4701a28abf16a8b485d/html5/thumbnails/29.jpg)
Questions, Comments and Dialogue
Visit our website at:
www.measureevaluation.org