rotary foundation cadre training: monitoring and evaluation
TRANSCRIPT
Monitoring and EvaluationTRF Cadre Training 12 January 2015
CADRE TRAINING | 2
PRESENTERS
Philip Silvers•Cadre Chairman 2014-15•Past RI Director and board liaison to the Trustees•44-year career as a program evaluation and survey research professional•10 years professor of graduate-level Research Methods
James Robinson•Director, Strategy Innovation and Evaluation•Former Division Manager Humanitarian Grants and Future Vision Pilot•RI-USAID Steering Committee
Korhan Atilla•Cadre Technical Coordinator 2008-11•Civil Engineer•Conducted sustainability check site visits•District Governor Elect 2014-15, D-2430
CADRE TRAINING | 3
Cadre’s Efforts to Support Sustainability Phil Silvers
CADRE TRAINING | 4
The New Grants Model Is Working Fairly Well– TRF Manuals Are Helpful– Clubs and Districts are working together– The Cadre Is an Important Resource
• Technical assistance• Stewardship (accountability)
Proposal Writers and Sponsors are struggling– Online application system is a challenge– Effective needs assessments are not being done– Sustainability—still an elusive concept– Monitoring and Evaluation (M&E) Who will do it?– Average time to grant approval: 3 months
YEAR ONE – WHAT DID WE LEARN?
CADRE TRAINING | 5
Core Effects:Sustainable
Change/Lasting Results
ProcessResults
(‘Doing Good’)
Training
Cultural Experiences
Adventure
Travel
Friendships
HumanitarianSupplies
Equipment
Externalities(The ‘Sizzle’)
Challenges
Rotary“Credits”
CADRE TRAINING | 6
• Real sustainability begins with the Community Needs Assessment—by talking to the grass-roots people before you come up with the solution.– Need to comprehend their view of the world—
their fears, their challenges, their dreams for their kids
• Needs Assessment, Sustainability, and M&E are not independent concepts—they must be integrated into one piece by one team.
• Depending on what you are trying to accomplish, the following slide gives you some options on measuring Sustainability (or ‘red flags’ if you omit them)
BASIC PRINCIPLES OF SUSTAINABILITY
CADRE TRAINING | 7
Security—•People
•Premises•Equipment
Dimensionsof
Sustainability
Ecological
Human Capital OrganizationalSystems
Behavioral Change MechanicalSystems
New Technologies(Tools) Cultural Ethos
Legacy
DIMENSIONS OF SUSTAINABILITY
Economic New Policy/Legislation
Cultural Ethos/Legacy
CADRE TRAINING | 8
Security refers to the protection against harm to people, to premises, and to equipment. Other than a good needs assessment, this is the most fundamental issue in Sustainability.
Are the participants secure from coercion, threats to their reputation, privacy, and physical/mental well-being?Are the premises secure from unwanted intruders?Are the equipment and materials safe from theft and vandalism?
CADRE TRAINING | 9
Economic refers to the financial viability of the project over the long haul.
Are revenue streams planned to support the program after the grant funding ends?
Operating costs? Equipment maintenance and replacement?
Who will be responsible for managing the revenue streams?Are contingency plans in place if the planned funding streams fail to deliver?
Economic
CADRE TRAINING | 10
Cultural ethos refers to what is acceptable or not acceptable behavior within society, e.g., smoking in common areas, young girls’ access to education, women’s role in government, getting tested for HIV/AIDS, tribal enmities, xenophobia.
Have project planners done a ‘force-field’ analysis of the factors inhibiting and fostering change?What rewards, incentives, modeling, or strategies might help shift prevailing attitudes?What various measures will reveal whether a change has happened?
Cultural Ethos/Legacy
CADRE TRAINING | 11
SUSTAINABILITY
How Else Can You Say It?
Game changer Systemic change Lasting results A permanent difference Self-perpetuating Continuity Can continue without outside help Can exist on its own indefinitely The ‘buck’ does not stop here!
CADRE TRAINING | 12
Evaluation Plan OverviewJames Robinson
CADRE TRAINING | 13
EVALUATION PLAN OVERVIEW
M & E Activities13-14
14-15
15-16
16-17
17-18
18-19
19-20
20-21
21-22
Areas of Focus X X X
Strategic Partners X X X
Grant Model X X X
Sustainability Assessments X X X
Multi-year evaluation Site visits for 5-10% of Global Grants Study common project types in each area of focus Replicate sustainability check methodology
CADRE TRAINING | 14
M & E Framework
CADRE TRAINING | 15
TERMINOLOGY
Monitoring typically refers to the routine and ongoing process of collecting data as agreed upon before project implementation.
Evaluation generally involves a post-project assessment of the degree to which the project goals were met and the impact the project activities had on beneficiaries.
A Measure is a quantitative or qualitative description of a project’s activities or impact.
Slide 15
CADRE TRAINING | 16 Slide 16
ONLINE REPORTING
CADRE TRAINING | 17 Slide 17
ONLINE REPORTING
CADRE TRAINING | 18 Slide 18
ONLINE REPORTING
CADRE TRAINING | 19 Slide 19
Is monitoring and evaluation data required to close a global grant?
Yes. Sponsors must provide the minimum standard (at least one measure per AoF) for M & E data in the final report. This information is required to close the grant.
If Sponsors promised more M & E data than they can provide at the time of reporting, can close the grant if the minimum standard has been met.
How should M & E funds budgeted in a global grant be processed as part of grant closure?
Sponsors can utilize up to 10% of the grant budget for M & E.
Provide accounting of expenditures at time of final report. If applicable, sponsors may provide a plan to expend remaining funds post closure and report M & E data into online system
GRANT REPORTING REQUIREMENTS
CADRE TRAINING | 20
CHARACTERISTICS OF GOOD M & E PLANS
1. Less is More
Rotarians should be modest in the number of measures they will track…if it’s in the application, we’ll expect to see it in the report.
Advise Rotarians to limit the number of areas of focus they select in the application as this will multiply the number of measures required.
Slide 20
CADRE TRAINING | 21
2. Measures match core project components
Rotarians do NOT need to measure every project component. Instead, they should select the most important components and focus on measuring their success in those.
Measures should be relevant to the main goals of the project and should indicate success in reaching the goals.
Slide 21
CHARACTERISTICS OF GOOD M & E PLANS
CADRE TRAINING | 22
3. Mixture of output & outcome measures
Counting activities and participants is a good place to start when developing an M/E Plan, but falls short of demonstrating quality.
Strong M/E plans include measures that demonstrate the quality of the project in terms of changes in the behavior or knowledge of the beneficiaries.
Slide 22
CHARACTERISTICS OF GOOD M & E PLANS
CADRE TRAINING | 23
4. Reasonable attribution
– Measures should be closely and obviously connected to activities conducted and the project participants.
– Grant officers can advise Rotarians on what is reasonable attribution of project outputs to outcomes.
Slide 23
CHARACTERISTICS OF GOOD M & E PLAN
CADRE TRAINING | 24
5. Resources are dedicated to M/E activities Eligible items include:– Contracted labor of a local org or paying stipends to community members to
collect data.– Consulting costs to technical experts to review and validate data.– To purchase data sets and other information necessary to establish a baseline. – Purchasing equipment such as phones, GPS equipment, and laptops to
monitor activities.– Local travel costs, such as car rental, fuel, and train fare to conduct site visits.– Cost of electronic surveys; meeting costs for focus groups for evaluation
purposes.– Printing costs for manuals and other materials needed for data collection.
Slide 24
CHARACTERISTICS OF GOOD M & E PLAN
CADRE TRAINING | 25
5. Resources are dedicated to M/E activities
Ineligible items include:– International travel to visit the project site.– Accommodations, meals and per diems.– Producing project promotional or recognition materials.– Any measurement expenses that are incurred prior to the grant approval
date, such as the costs of conducting a community assessment.
Slide 25
CHARACTERISTICS OF GOOD M & E PLAN
CADRE TRAINING | 26
Sustainability
CADRE TRAINING | 27
SUSTAINABILITY
For The Rotary Foundation, sustainability means: ensuring that grants provide long-term solutions to community needs that the benefiting community can maintain after grant funding ends.
CADRE TRAINING | 28
Legacy Grants; Giving A Man A Fish
SUSTAINABILITY
CADRE TRAINING | 29
Rotary Foundation
ensuresSustainable
impact ofgoods or services.
Clubs and Districts ensure sustainable
impact of goods and services
Individuals & organizations benefit
from sustainable impact of goods and
services
Community or organization ensures sustainable impact of goods and services
TRF ensures sustainable impact of goods and services
SUSTAINABILITY
Global Grants; Teaching A Man to Fish
CADRE TRAINING | 30
Sustainability Framework
CADRE TRAINING | 31
SUSTAINABILITY FRAMEWORK
CommunityNeeds/
Strengths
Materials/Technology Funding
• Community Needs Assessment
• Meeting beneficiary needs
• Involving community partners
• Confirm local funding for long-term support
• Compensate project participants to ensure continuity of work
• Purchase equipment locally
• Involve community in selecting technology
• Train community to operate and maintain equipment
CADRE TRAINING | 32
SUSTAINABILITY FRAMEWORK
Knowledge Motivation Monitoring/Evaluation
• Provide training to beneficiaries
• Create plan to transfer knowledge to new beneficiaries
• Collaborate with local organizations with expertise
• Give incentives to beneficiaries to provide ongoing support
• Identify leaders committed to outcomes
• Prepare the community to own the project when funds expended
• Develop measurable objectives
• Identify ways to collect data
• Establish baseline data to track the impact of the project
CADRE TRAINING | 33
SUSTAINABILITY FRAMEWORK
CommunityNeeds/
Strengths
Materials/Technology
Funding
Knowledge
Motivation
Monitoring/Evaluation
High incidence of waterborne illness among children under 5 in a village
CADRE TRAINING | 34
Abt Sustainability Check
CADRE TRAINING | 35
Average Lifecycle Scores For MGs & GGs
1.001.502.002.503.003.504.00
Application &Proposal
Contract Point InTime/ Final
Lifecycle Stage
Avera
ge Sc
ore (
1-4 P
ossib
le)
Average Of All GrantsAverage Of All GGsAverage of All MGs
Mid-Point
ABT SUSTAINABILITY CHECK
CADRE TRAINING | 36
Lifecycle Sustainability Scores; All Grants
1.00
1.50
2.00
2.50
3.00
3.50
4.00
1 2 3Lifecycle Stage
Scor
e (1-
4 Pos
sible)
India GG 25034
Kenya GG 25081
India GG 25027
Honduras GG 25065
Honduras GG 25363
India MG 74944
India MG 74942
Kenya MG 70520
Honduras MG 71320
Honduras MG 71446
Honduras MG 71940
Mid-Point
Cooperating Organizations
ABT SUSTAINABILITY CHECK
CADRE TRAINING | 37
Sustainability Matrix
CADRE TRAINING | 38
SUSTAINABILITY MATRIX
Eight Factors of Sustainability Evaluated
Grant ManagementOutcomesFinancialCommunityBeneficiary OrganizationsCooperating OrganizationsTrainingSoftware/Equipment
CADRE TRAINING | 39
SUSTAINABILITY MATRIX
Methodology
Assessment is comprised of eight sectionsEach section has 15-30 indicator questions Each grant is assigned a score for each indicator within
the applicable sections Score is between 1-4
Each grant is evaluated independently
CADRE TRAINING | 40
SUSTAINABILITY MATRIX
Grant ManagementThe Grant Management section focuses on overall project management
Key indicators include: Does the project have precise and finite objectives? Is there an implementation workplan including milestones and
benchmarks? Were sustainable activities incorporated into the original project
design? Were the required progress reports submitted on time? Was a sustainability plan included in the final report?
CADRE TRAINING | 41
SUSTAINABILITY MATRIX
Measuring OutcomesThe Measuring Outcomes section focuses on the measurability of a project, and whether or not appropriate M&E was conducted during and after the project
Key indicators include: Does the project have clear and measurable outcomes? Does the monitoring system include community involvement? Are the project outcomes shared with the community and other
stakeholders in ways meaningful for sustained project support?
CADRE TRAINING | 42
SUSTAINABILITY MATRIX
FinancialThe Financial section focuses on budget planning and the financial role of the cooperating organization
Key indicators include: Does the projected budget indicate sufficient revenue to fund the
workplan? Does the project Spending Plan (Budget) align with the
implementation workplan line items (Activity Plan)? Does the project budget indicate sufficient revenue to fund the
sustainability activities beyond the end of the grant?
CADRE TRAINING | 43
SUSTAINABILITY MATRIX
CommunityThe Community section focuses on community involvement and buy-in for the project
Key indicators include: Was a community needs assessment completed? Were
recommendations made? Does the workplan include activities to create or promote local
community involvement and ownership? Do grant reports describe community relationships developed which
support the continuation of project outcomes beyond the life of the grant?
Does the grant identify community champions willing to assume leadership roles for sustaining project outcomes after the life of the grant?
CADRE TRAINING | 44
SUSTAINABILITY MATRIX
Beneficiary OrganizationThe Beneficiary Organization section focuses on the project’s relationship with the beneficiary organization and the organization’s capacity to take ownership of the project
Key indicators include: Do written agreements define the terms of cooperation, roles and
responsibilities of the beneficiary organizations? Does the documentation identify personnel willing to assume
leadership roles for sustaining project outcomes? Does the local Club/District demonstrate strong partnerships with
beneficiary organizations as appropriate to objectives?
CADRE TRAINING | 45
SUSTAINABILITY MATRIX
Cooperating OrganizationThe Cooperating Organization section focuses on the project management plan and structure of the relationship between the project sponsors and cooperating organization
Key indicators include: Does the documentation include analysis and/or justification of
cooperating organization(s) selection based on demonstrated expertise or experience?
Does the design integrate into the cooperating organization(s) regular systems or operations?
Does the cooperating organization agree in writing to participate in program financial reviews?
Does the documentation include a succession plan to hand over the project to the partner organization(s) at the end of the grant lifecycle?
CADRE TRAINING | 46
SUSTAINABILITY MATRIX
TrainingThe Training section focuses on the training and follow up associated with the project
Key indicators include: Does the training plan address gaps identified in the assessment of
beneficiaries targeted for training (doctors, teachers, unskilled labor)? Are the training course design (delivery and materials) locally and
culturally relevant (e.g., language, practical examples)? Is there a measurement process to assess learner outcomes after
training? Is there a plan for keeping skills up-to-date? Has the project resulted in demand for training outside the targeted
beneficiary group (organization, individuals)?
CADRE TRAINING | 47
SUSTAINABILITY MATRIX SECTIONS
Software/EquipmentThe Software/Equipment section assesses whether culturally appropriate technology was used during the project
Key indicators include: Does the documentation include a justification of why equipment is
needed? Does the proposed equipment respect local social and cultural
norms? Was the availability of equipment on the local market considered? Is there an inventory management system? Do user guidelines exist for equipment use? Are local spare parts and replacement equipment available? Is equipment being used as intended?
CADRE TRAINING | 48
Questions?
CADRE TRAINING | 49
Areas of FocusJames Robinson
CADRE TRAINING | 50
AREA OF FOCUS EVALUATIONS – FUTURE VISION PILOT
Global Grants:Sample of all Global Grants approved during pilot, 35 site visits across 5 areas of focus Nov 2013- Feb 2014Nine countries on four continentsFocused on VTT and Humanitarian Projects, not Scholarships10 Cadre and 4 Area of Focus Managers
CADRE TRAINING | 51
Pilot Evaluation - Area of Focus Site VisitsKorhan Atilla
CADRE TRAINING | 52
WHAT IS SUSTAINABILITY?(is it QUALİTY?)
AND WHAT IT IS NOT?(is it only QUALİTY?)
CADRE TRAINING | 53
MATTERS OF SUSTAINABILITY
1- Financial Sustainability2- Maintenance3- Proper Operation4- Precautions during installation5- Rotarian Involvement6- Benefitting Community Involvement
CADRE TRAINING | 54
CADRE TRAINING | 55
CADRE TRAINING | 56
CADRE TRAINING | 57
CADRE TRAINING | 58
CADRE TRAINING | 59
CADRE TRAINING | 60
CADRE TRAINING | 61
CADRE TRAINING | 62
CADRE TRAINING | 63
CADRE TRAINING | 64
CADRE TRAINING | 65
CADRE TRAINING | 66
CADRE TRAINING | 67
CADRE TRAINING | 68
CADRE TRAINING | 69
CADRE TRAINING | 70
CADRE TRAINING | 71
CADRE TRAINING | 72
CADRE TRAINING | 73
CADRE TRAINING | 74
CADRE TRAINING | 75
CADRE TRAINING | 76
CADRE TRAINING | 77
CADRE TRAINING | 78
Questions?