state of sustainability planning & evaluation in community health sustainability framework (sf)...
TRANSCRIPT
![Page 1: State of Sustainability Planning & Evaluation in Community Health Sustainability Framework (SF) 101 April 21, 2011 IntraHealth with the SHOUT Group](https://reader035.vdocuments.site/reader035/viewer/2022062515/56649ce15503460f949abdd7/html5/thumbnails/1.jpg)
State of Sustainability Planning & Evaluation in Community Health
Sustainability Framework (SF) 101
April 21, 2011IntraHealth with the SHOUT Group
![Page 2: State of Sustainability Planning & Evaluation in Community Health Sustainability Framework (SF) 101 April 21, 2011 IntraHealth with the SHOUT Group](https://reader035.vdocuments.site/reader035/viewer/2022062515/56649ce15503460f949abdd7/html5/thumbnails/2.jpg)
ACKNOWLEDGEMENTS
IntraHealthRenée Kotz, Sara Margolis-Pacqué, Frances Thornhill, Kris Horvath
ICF MacroJennifer Yourkavitch, Ilona Varallyay, Kiersten Johnson, Leo Ryan
![Page 3: State of Sustainability Planning & Evaluation in Community Health Sustainability Framework (SF) 101 April 21, 2011 IntraHealth with the SHOUT Group](https://reader035.vdocuments.site/reader035/viewer/2022062515/56649ce15503460f949abdd7/html5/thumbnails/3.jpg)
SF 101 - Outline
• Part 1: Introduction--“It’s complicated”
• Part 2: Practical Recent M&E Experience
• Part 3: SF 101: – Planning: Small but Critical Steps– M&E: Do we count all trees to measure a forest?
• Conclusion
![Page 4: State of Sustainability Planning & Evaluation in Community Health Sustainability Framework (SF) 101 April 21, 2011 IntraHealth with the SHOUT Group](https://reader035.vdocuments.site/reader035/viewer/2022062515/56649ce15503460f949abdd7/html5/thumbnails/4.jpg)
PART 1 - INTRODUCTIONIt’s not complicated; it’s complex
![Page 5: State of Sustainability Planning & Evaluation in Community Health Sustainability Framework (SF) 101 April 21, 2011 IntraHealth with the SHOUT Group](https://reader035.vdocuments.site/reader035/viewer/2022062515/56649ce15503460f949abdd7/html5/thumbnails/5.jpg)
Current Context Exhibit A: Global Health Initiative
• “The challenge of the next decade and beyond is to take these impressive accomplishments to the next level by helping countries achieve long-term sustainability in their health services.”
• “Building on a long tradition of U.S. government global health leadership and the unprecedented level of commitment manifested in recent years, the Obama Administration’s Global Health Initiative has the opportunity to move global health to a new level of effectiveness, with a vision of long-term sustainability led by partner countries.”
GHI— [Implementation of the Global Health Initiative: Consultation Document]
![Page 6: State of Sustainability Planning & Evaluation in Community Health Sustainability Framework (SF) 101 April 21, 2011 IntraHealth with the SHOUT Group](https://reader035.vdocuments.site/reader035/viewer/2022062515/56649ce15503460f949abdd7/html5/thumbnails/6.jpg)
Do we need a system perspective?E.g. Urban Health System, Bangladesh (Concern)
Health Platform (MESPCC)
NGO Health
Providers
ADB, DfID, USAID, etc.
Govt & Private Health Facilities
Ward Health Committee
Community organizations
Traditional health providersYouth volunteers
Teachers
Private pharmacists
Social & religious leaders
Commissioner
City Government
MOLGRD
Chairman
Health Department
Health Inspector (in absence of Med
Off.)
MOHFW District & Sub-District
MOHFW
![Page 7: State of Sustainability Planning & Evaluation in Community Health Sustainability Framework (SF) 101 April 21, 2011 IntraHealth with the SHOUT Group](https://reader035.vdocuments.site/reader035/viewer/2022062515/56649ce15503460f949abdd7/html5/thumbnails/7.jpg)
Sustainability in HSS: a Complex Equation?
Illustration inspired by Samir Rihani: “Complex Systems: Theory and Development Practice. Understanding Non-Linear Realities.”
“Health -programme sustainability is the ultimate manifestation of a complex web of inter-relations between health concerns, stakeholders, resources, and actions analogous to an ecosystem.”
Gruen et al. The Lancet. 2008
![Page 8: State of Sustainability Planning & Evaluation in Community Health Sustainability Framework (SF) 101 April 21, 2011 IntraHealth with the SHOUT Group](https://reader035.vdocuments.site/reader035/viewer/2022062515/56649ce15503460f949abdd7/html5/thumbnails/8.jpg)
Common Evaluation Questions about Sustainability
Questions I don’t understand:• Is it sustainable?
• Can you ensure it will be sustained?
• Who will take over after you leave?
Questions I understand: Will you leave partners with a better chance of
facing what will come after you leave? Are you at least improving expectations for
continued progress in the context? Are you thinking “within a system?” [a.k.a. is the
“Sustainability Scenario” coherent from an internal development logic?]
Do you have information about where your efforts are heading?
Are your partners mastering the processes which produce the results?
Do the M&E systems inform the local actors about their progress?
![Page 9: State of Sustainability Planning & Evaluation in Community Health Sustainability Framework (SF) 101 April 21, 2011 IntraHealth with the SHOUT Group](https://reader035.vdocuments.site/reader035/viewer/2022062515/56649ce15503460f949abdd7/html5/thumbnails/9.jpg)
Evolving Understanding of Sustainability— A Suggestion:
• An (emerging?) property within a local system which allows interdependent actors to maintain and improve the health status of the system’s (vulnerable) population through negotiated and coordinated social interactions, allowing the expression of their respective and collective capabilities.
![Page 10: State of Sustainability Planning & Evaluation in Community Health Sustainability Framework (SF) 101 April 21, 2011 IntraHealth with the SHOUT Group](https://reader035.vdocuments.site/reader035/viewer/2022062515/56649ce15503460f949abdd7/html5/thumbnails/10.jpg)
Implications of this definition1. Other definitions (i.e. financial viability, organizational
capacity, policy alignment), identify potential determinants of sustainability
2. Projects can contribute to the process, or hinder it, but ultimately neither control nor define it.
3. We can, however, measure progress on hypothesized determinants of sustainability during our projects
4. More than one configuration can support the same sustainable outcome
5. Allowing systems to find their own new equilibrium might be more important than enforcing the best approach
![Page 11: State of Sustainability Planning & Evaluation in Community Health Sustainability Framework (SF) 101 April 21, 2011 IntraHealth with the SHOUT Group](https://reader035.vdocuments.site/reader035/viewer/2022062515/56649ce15503460f949abdd7/html5/thumbnails/11.jpg)
Understanding the Role of External Agents in Building Sustainability
In a Linear Process• Attribution:
Input->Process->Output->Outcome->Impact
• Direct Accountability– “ensure sustainability in X years”
• Fits Top-Down Planning– Caveat:“involve stakeholders”– pre-determined “models”
• “We come, we fix, we go. Hey! They ‘got it’!”
• Evidence? – Partial, sub-systems, very thin.
In a Complex Process
• Contribution:Ownership? Negotiation? Alignment?
• Shared Accountability– “advance capacity & processes”
• Role of “Agents”– Agents have strategies– “models” emerge / constructed
• “Donors and Implementers as Types of Agents.”
• Evidence?– Challenging ; Growing Interest;
More robust?
![Page 12: State of Sustainability Planning & Evaluation in Community Health Sustainability Framework (SF) 101 April 21, 2011 IntraHealth with the SHOUT Group](https://reader035.vdocuments.site/reader035/viewer/2022062515/56649ce15503460f949abdd7/html5/thumbnails/12.jpg)
PART 2 – THREE RECENT EXAMPLES
1. Prospective (USAID/NEPAL)
Point in time evaluation of progress toward sustainability by supported districts
2. Prospective & Retrospective (USAID/CSHGP/Concern Bangladesh)
Planning for sustainabilityWhat has been sustained?How much has been sustained?
3. Modeling Investments (USAID/CSHGP/Save the Children Guinea)
What could we achieve if…?
![Page 13: State of Sustainability Planning & Evaluation in Community Health Sustainability Framework (SF) 101 April 21, 2011 IntraHealth with the SHOUT Group](https://reader035.vdocuments.site/reader035/viewer/2022062515/56649ce15503460f949abdd7/html5/thumbnails/13.jpg)
The Sustainability Framework –Time and Perspective in Evaluation
Prospective: What are the long-term odds based on the results, processes, and systems being built?
Retrospective: Were gains maintained?
Did progress continue? How did system adapt?
Constraints & Black Swans
Implement HSS
Intervention
Intervention Ends
Stakeholders Adapt & Organize
Improvements continue and/or
Achievements are maintained
SF
Source: Taking the Long View: A Practical Guide to Sustainability Planning and Measurement in Community-Oriented Health Programming. 2008. http://www.mchipngo.net/controllers/link.cfc?method=tools_sustain. And: Black Swan, Grey Swan, Sustainability. Or, the difference between Planning and Predicting. Available at: http://cedarscenter.com/resdetail.cfm?resid=125
![Page 14: State of Sustainability Planning & Evaluation in Community Health Sustainability Framework (SF) 101 April 21, 2011 IntraHealth with the SHOUT Group](https://reader035.vdocuments.site/reader035/viewer/2022062515/56649ce15503460f949abdd7/html5/thumbnails/14.jpg)
E.g. 1- Measuring sustainability as a programming tool for health sector investments – USAID Nepal
Using Composite / Dashboard Measures
Intermediate
Promising
Poor
Nascent
Strong
Compo 1 index
Compo 4 index
Compo 5 index
Compo 6 index2
3
Source: Measuring sustainability as a programming tool for health sector investments—report from a pilot sustainability assessment in five Nepalese health districts. Int J Health Plann Manage. 2009 Oct-Dec;24(4):326-50 http://www3.interscience.wiley.com/journal/118903030/abstract
![Page 15: State of Sustainability Planning & Evaluation in Community Health Sustainability Framework (SF) 101 April 21, 2011 IntraHealth with the SHOUT Group](https://reader035.vdocuments.site/reader035/viewer/2022062515/56649ce15503460f949abdd7/html5/thumbnails/15.jpg)
Can districts sustain the health of their mothers & children?
Kanchanpur and Rasuwa Districts, Nepal - 2006
![Page 16: State of Sustainability Planning & Evaluation in Community Health Sustainability Framework (SF) 101 April 21, 2011 IntraHealth with the SHOUT Group](https://reader035.vdocuments.site/reader035/viewer/2022062515/56649ce15503460f949abdd7/html5/thumbnails/16.jpg)
E.g. 2- Sustained Results 5 Years Post-Project. Urban Health / Concern Bangladesh: 1999-2004↨-2009
• From Urban Child Survival Project to Urban Health System Strengthening
– Defining a Sustainability Scenario consistent with national policy
– Defining ±consistent measures (incl. outcomes and repeated capacity assessments)
– Repeated assessments & collective learning
• 18 months post-project coaching (minimal)
→ 3-year and 5-year post-project sustainability assessments
Source: Sustainability of the Saidpur and Parbatipur Urban Health Model (Bangladesh) Five Years After the End of Concern’s Child Survival Project . Final Evaluation Report – January 10, 2010. USAID, Concern Worldwide. ICF Macro/. Available at: http://cedarscenter.com/resdetail.cfm?resid=106
![Page 17: State of Sustainability Planning & Evaluation in Community Health Sustainability Framework (SF) 101 April 21, 2011 IntraHealth with the SHOUT Group](https://reader035.vdocuments.site/reader035/viewer/2022062515/56649ce15503460f949abdd7/html5/thumbnails/17.jpg)
CWI Bangladesh: 1999-2004↨-2009How Much Sustainability in Health Outcomes?
Project Ends
![Page 18: State of Sustainability Planning & Evaluation in Community Health Sustainability Framework (SF) 101 April 21, 2011 IntraHealth with the SHOUT Group](https://reader035.vdocuments.site/reader035/viewer/2022062515/56649ce15503460f949abdd7/html5/thumbnails/18.jpg)
E.g.3- Does it matter? HSS at District Level: Sustainability Investments and Child Deaths Averted
Source: Pro-sustainability choices and child deaths averted: from project experience to investment strategy Health Policy and Planning 2010;1–12. ; doi: 10.1093/heapol/czq042
Save the Children US in Guinea
![Page 19: State of Sustainability Planning & Evaluation in Community Health Sustainability Framework (SF) 101 April 21, 2011 IntraHealth with the SHOUT Group](https://reader035.vdocuments.site/reader035/viewer/2022062515/56649ce15503460f949abdd7/html5/thumbnails/19.jpg)
A Simplified Traditional Scenario for Investment
Total Investment:$7.5 millions
U5 Lives Saved:2,530 [1,569 – 6,167]
![Page 20: State of Sustainability Planning & Evaluation in Community Health Sustainability Framework (SF) 101 April 21, 2011 IntraHealth with the SHOUT Group](https://reader035.vdocuments.site/reader035/viewer/2022062515/56649ce15503460f949abdd7/html5/thumbnails/20.jpg)
Investment Scheme for Maximizing Sustainability – Phase 1
![Page 21: State of Sustainability Planning & Evaluation in Community Health Sustainability Framework (SF) 101 April 21, 2011 IntraHealth with the SHOUT Group](https://reader035.vdocuments.site/reader035/viewer/2022062515/56649ce15503460f949abdd7/html5/thumbnails/21.jpg)
Investment Scheme for Maximizing Sustainability – Phase 2
![Page 22: State of Sustainability Planning & Evaluation in Community Health Sustainability Framework (SF) 101 April 21, 2011 IntraHealth with the SHOUT Group](https://reader035.vdocuments.site/reader035/viewer/2022062515/56649ce15503460f949abdd7/html5/thumbnails/22.jpg)
Investment Scheme for Maximizing Sustainability – Phase 3
![Page 23: State of Sustainability Planning & Evaluation in Community Health Sustainability Framework (SF) 101 April 21, 2011 IntraHealth with the SHOUT Group](https://reader035.vdocuments.site/reader035/viewer/2022062515/56649ce15503460f949abdd7/html5/thumbnails/23.jpg)
Investment Scheme for Maximizing Sustainability – Phase 4
![Page 24: State of Sustainability Planning & Evaluation in Community Health Sustainability Framework (SF) 101 April 21, 2011 IntraHealth with the SHOUT Group](https://reader035.vdocuments.site/reader035/viewer/2022062515/56649ce15503460f949abdd7/html5/thumbnails/24.jpg)
Investment Scheme for Maximizing Sustainability – Phase 5
Total Investment:$7.5 millions
U5 Lives Saved:8,485 [4,169 – 8,909]
![Page 25: State of Sustainability Planning & Evaluation in Community Health Sustainability Framework (SF) 101 April 21, 2011 IntraHealth with the SHOUT Group](https://reader035.vdocuments.site/reader035/viewer/2022062515/56649ce15503460f949abdd7/html5/thumbnails/25.jpg)
Testable Hypothesis: three-time the impact for the same investment
![Page 26: State of Sustainability Planning & Evaluation in Community Health Sustainability Framework (SF) 101 April 21, 2011 IntraHealth with the SHOUT Group](https://reader035.vdocuments.site/reader035/viewer/2022062515/56649ce15503460f949abdd7/html5/thumbnails/26.jpg)
PART 3SF 101 - From Planning to M&E
![Page 28: State of Sustainability Planning & Evaluation in Community Health Sustainability Framework (SF) 101 April 21, 2011 IntraHealth with the SHOUT Group](https://reader035.vdocuments.site/reader035/viewer/2022062515/56649ce15503460f949abdd7/html5/thumbnails/28.jpg)
![Page 29: State of Sustainability Planning & Evaluation in Community Health Sustainability Framework (SF) 101 April 21, 2011 IntraHealth with the SHOUT Group](https://reader035.vdocuments.site/reader035/viewer/2022062515/56649ce15503460f949abdd7/html5/thumbnails/29.jpg)
‘Sustainability Manual’ Tools
• Any-Stage Rapid Tool– Sustainability Planning Checklist
• Step 1– Local system mapping and stakeholder analysis– Visioning activity
• Step 2– From Visioning to Developing a “Sustainability
Scenario”
![Page 30: State of Sustainability Planning & Evaluation in Community Health Sustainability Framework (SF) 101 April 21, 2011 IntraHealth with the SHOUT Group](https://reader035.vdocuments.site/reader035/viewer/2022062515/56649ce15503460f949abdd7/html5/thumbnails/30.jpg)
Stakeholders’ Analysis for Sustainability Planning
POWER /
INFLUENCE Low-----------------------------------------High
INFORM & ANSWER CONCERNS
Core of the SystemChange Agents
MINIMAL INVOLVEMENT
KEEP INFORMED & FIND OPPORTUNITIES FOR COOPERATION WITH LOW
TRANSACTION COST
INTERESTLow---------------------------------------------------------------------------------High
![Page 31: State of Sustainability Planning & Evaluation in Community Health Sustainability Framework (SF) 101 April 21, 2011 IntraHealth with the SHOUT Group](https://reader035.vdocuments.site/reader035/viewer/2022062515/56649ce15503460f949abdd7/html5/thumbnails/31.jpg)
![Page 32: State of Sustainability Planning & Evaluation in Community Health Sustainability Framework (SF) 101 April 21, 2011 IntraHealth with the SHOUT Group](https://reader035.vdocuments.site/reader035/viewer/2022062515/56649ce15503460f949abdd7/html5/thumbnails/32.jpg)
‘Sustainability Manual’ Tools for M&E
• Facilitating a detailed planning workshop with local stakeholders
• Measurement tools & Environmental Scan • Notes on basing an evaluation on the
Sustainability Framework
![Page 33: State of Sustainability Planning & Evaluation in Community Health Sustainability Framework (SF) 101 April 21, 2011 IntraHealth with the SHOUT Group](https://reader035.vdocuments.site/reader035/viewer/2022062515/56649ce15503460f949abdd7/html5/thumbnails/33.jpg)
Local actors revise sustainability vision
and scenario
Local actors are supported in
developing a (draft) sustainability vision
and scenario
Project designs interventions to
strategically CONTRIBUTE in the
most effective manner possible to the achievement of
the sustainability scenario
Projects develops its M&E plan, in
order to be accountable for its
progress. Project also adds
element of information in its M&E plan, to help
the local actors measure progress
toward their sustainability
scenario.
Participatory evidence-based
M&E activities are implemented to
help answer: [1] is the project
effective in achieving its objectives?
[2] are there indications of
progress toward the sustainability scenario of local
actors?[3] what lessons
are being learned about the
sustainability scenario (still
valid?), the value of the project’s
contribution (still strategic); and the
role played by local actors (consistent
with their own sustainability
scenario?)
Corrective measures are taken by the project and
by local actors
![Page 34: State of Sustainability Planning & Evaluation in Community Health Sustainability Framework (SF) 101 April 21, 2011 IntraHealth with the SHOUT Group](https://reader035.vdocuments.site/reader035/viewer/2022062515/56649ce15503460f949abdd7/html5/thumbnails/34.jpg)
First Steps in Sustainability Planning– Who Does What?
![Page 35: State of Sustainability Planning & Evaluation in Community Health Sustainability Framework (SF) 101 April 21, 2011 IntraHealth with the SHOUT Group](https://reader035.vdocuments.site/reader035/viewer/2022062515/56649ce15503460f949abdd7/html5/thumbnails/35.jpg)
THE BEAUTY AND DANGERS OF M&E FRAMEWORKS
![Page 36: State of Sustainability Planning & Evaluation in Community Health Sustainability Framework (SF) 101 April 21, 2011 IntraHealth with the SHOUT Group](https://reader035.vdocuments.site/reader035/viewer/2022062515/56649ce15503460f949abdd7/html5/thumbnails/36.jpg)
Component 1—Health Outcomes
Component 2—Services
Component 3—Regional/District MOH Capacity & Viability
Component 4—Local NGO Capacity & Viability
Component 5—Community Capacity
Component 6—“Enabling” Environment
Proportion of pregnant women in targeted households who slept under an ITN the previous night
Quality of existing services
DPHO capacity and viability
Capacity of NRD (local NGO)
Community Health Competency in Malaria
Environment Scan
Proportion of children under five in targeted households who slept under an ITN the previous night
Coverage of health services – facilities + community based
Viability of NRD (local NGO)
Proportion of targeted households with a pregnant woman or children under 5 with at least one ITN
Community-based treatment policy
Proportion of women in targeted households who received two or more doses of IPTp during their last pregnancy in the last two years
Proportion of women in targeted households that cite IPTp as a necessary treatment during pregnancy.
National ACT + IPT supply
Proportion of children under five years old in targeted households with fever in the last two weeks who received treatment with an antimalarial according to national policy within 24 hours of onset of fever
Coverage of community groups
Functionality / Capacity of community groups
Example / Community-Based Malaria Project
![Page 37: State of Sustainability Planning & Evaluation in Community Health Sustainability Framework (SF) 101 April 21, 2011 IntraHealth with the SHOUT Group](https://reader035.vdocuments.site/reader035/viewer/2022062515/56649ce15503460f949abdd7/html5/thumbnails/37.jpg)
Results Framework
IR1 IR2 IR3 IR4
Key Elements for Sustainability in the Results Framework of the Project
Potentially Key Elements for Sustainability in the Larger Environment of the Project and National Stakeholders
Elements in the Results Framework Sustainability Framework
Component 1
Component 2
Component 3
Component 4
Component 1
Component 2
Component 3Component 4
Component 5
Baseline 18 months
Project Objective
Sustainability Scenario
Assessing Progress Toward Sustainability
Project
Accountability
Shared
Accountability
![Page 38: State of Sustainability Planning & Evaluation in Community Health Sustainability Framework (SF) 101 April 21, 2011 IntraHealth with the SHOUT Group](https://reader035.vdocuments.site/reader035/viewer/2022062515/56649ce15503460f949abdd7/html5/thumbnails/38.jpg)
Results Framework Aligned to the SF (Taking the Long View. CARE Nepal)
Dimension 1
IR1: Improved MN outcomes esp amongst marginalized populations
IR2:Improved
maternal & neonatal services
Dimension 2
IR3:Improved
capacity of MOH and
other sectors for MN care
IR4:Improved viability of
maternal and neonatal services
Strategic Objective: Sustained and equitable improvements in maternal and newborn health in the district of Doti and Kailali
especially amongst marginalized populations
IR5:Improved
community commitment for maternal & neonatal
care
IR6:Improved
environment for maternal & neonatal
care
![Page 39: State of Sustainability Planning & Evaluation in Community Health Sustainability Framework (SF) 101 April 21, 2011 IntraHealth with the SHOUT Group](https://reader035.vdocuments.site/reader035/viewer/2022062515/56649ce15503460f949abdd7/html5/thumbnails/39.jpg)
The SF “Dashboard”A Picture Is Worth 1,000 Words But Can Sometimes Tell a Lie
![Page 40: State of Sustainability Planning & Evaluation in Community Health Sustainability Framework (SF) 101 April 21, 2011 IntraHealth with the SHOUT Group](https://reader035.vdocuments.site/reader035/viewer/2022062515/56649ce15503460f949abdd7/html5/thumbnails/40.jpg)
Conclusion• Some Practical Tool and Guidance; No Insurance Policy• Nothing (Really) New Under the Sun, Except Getting
Serious:– Implications for Health and Community Systems– Implications for Scale Up Strategies– Implications for Integration of Health with Sustainable
Development in view of Global Challenges– Implications for Evaluation +++
• Don’t Lose Sight of the Forest for the Trees• The [Sustainability] Questions of Donors are also the
Questions for Countries Themselves
![Page 41: State of Sustainability Planning & Evaluation in Community Health Sustainability Framework (SF) 101 April 21, 2011 IntraHealth with the SHOUT Group](https://reader035.vdocuments.site/reader035/viewer/2022062515/56649ce15503460f949abdd7/html5/thumbnails/41.jpg)
THANK YOU.