program design: integrating research and...
TRANSCRIPT
Program Design: Integrating Research and Evaluation
Session A7: 9:30-10:45
Abiyou KiflieAbiy Seifu,Gareth Parry
Faculty
Abiyou Kiflie, MD,
MPH
Deputy Country
Director, IHI Ethiopia
Gareth Parry, PhD
Senior Scientist,
IHI
@GJParry03
Abiy Seifu, MPH
Lecturer
School of Public Health,
Addis Ababa University
@AbiySe
Success!
Conclusions: Implementation of surgical safety checklists in Ontario, Canada, was not associated
with a significant reductions in operative mortality or complications.
Fernando Althabe
A population-based, multifaceted strategy to implement antenatal corticosteroid treatment versus
standard care for the reduction of neonatal mortality due to preterm birth in low-income and middle-
income countries: the ACT cluster randomized trial
Fernando Althabe et alThe Lancet. 2015 Feb 20;385(9968):629-39
Interpretation Despite increased use of antenatal corticosteroids in low-birthweight infants in the intervention groups,
neonatal mortality did not decrease in this group, and increased in the population overall. For every 1000 women
exposed to this strategy, an excess of 3·5 neonatal deaths occurred, and the risk of maternal infection seems to have
been increased
“..described in the 1980s by American program
evaluator Peter Rossi as the “Iron Law” of … arguing
that as a new model is implemented widely across a
broad range of settings, the effect will tend toward zero.”
Innovation
sample
Evaluation
sample
Immediate wide-scale
implementation
Reduction in Effectiveness from Applying Same
Fixed-Protocol Program in Different Contexts
Parry GJ, et al (2013).
Where Can a Model Be Amended to Work?
Identify contexts in which it can be amended to work as we move
from Innovation to Prototype to Test and Spread
Innovation
sample
Parry GJ, et al (2013).
Fernando Althabe
Implementation science at the crossroads Richard J Lilford
BMJ Qual Saf. 2017 Nov 28:bmjqs-2017
“What we need to come up with is interventions that will do more good than
harm, without requiring a degree of specification that would be hard to
replicate and/or that only works when implemented by a research team
committed to the cause”
Fernando Althabe
Increasing the generalisability of improvement research
with an improvement replication programmeJohn Øvretveit et al
Quality and Safety in Health Care. 2011 Apr 1;20(Suppl 1):i87-91
“If a quality improvement is found effective in one setting, would the same
effects be found elsewhere? Could the same change be implemented in
another setting?”
Core Concepts & Detailed Tasks
MEWS >=5Use a reliable
method to identify
deteriorating
patients in real
time.
When a patient is
deteriorating,
provide the most
appropriate
assessment and
care as soon as
possible
MEWS >=4
2 Nurses
1 Physician
1 Nurse
1 Physician
1 Physician
ActionTheory
Core Concepts Detailed Tasks
and Local
Adaptations
1) Generating the pressure (will) for ICUs to take part
2) A networked community
3) Re-framing BSIs as a social problem
4) Approaches that shaped a culture of commitment
5) Use of data as a disciplinary force
6) Hard edges
(4) Milbank Quarterly, 2011
From an Improvement Perspective:
Initial
Concepts
Concepts rather than fixed protocols are a good
starting point for people to test and learn whether
improvement interventions can be amended to their
setting.
Social ChangeImprovement requires social change and that people
are more likely to act if they believe.
Context
MattersInterventions need to be amended to local settings
(contexts).
Learning Learn what is takes to bring about improvement.
We need: ‘Theory-driven rapid-cycle formative evaluation’
Salzburg Global Seminar Session 565
M. Rashad Massoud University Research Co., LLC
Framework for learning about improvement
Objective for
stakeholders
Theory of
change
Context
HowWhat
Improvement
Program DesignEvaluation Design
Evaluation Continuum
Pure
External
Evaluation
Highly
Embedded
Evaluation
Improvers and Evaluators as best friends
Evaluability Assessment
With all key stakeholders:
Agree the Theory of Change
– Five Core Design Components
Agree the evaluation design, including:
– Agreeing on the evaluation questions
– Formative and/or summative approaches
– Availability and Use of Data
– Available human and financial resources
Leviton LC et al Evaluability assessment to improve public health policies,
programs, and practices. Annual Review of Public Health. 2010 Apr
21;31:213-33.
Evaluation Design 15
The What The How The Context
Innovation phase: Model development typically takes place in a small number
of settings, and evaluation questions should focus largely on The What
Testing Phase: The aim is to identify where a model works, or can be amended
to work. Hence, although refining The What will occur, developing The How and
The Context will also be important.
Spread and Scale-up Phase: The aim is to spread or scaling up the model in
contexts earlier work has indicated it is likely to work or be amended to work.
Here, the What and the Context should be well developed, and the focus will be
primarily on The How.
What is the overall impact of the
model on patient outcomes?
Which elements of the model
had the greatest impact on
patient outcomes?
To what extent can all the
changes be implemented?
What are barriers and facilitators
to implementing the changes
locally?
Within what settings does the
model work, or can be amended
to work?
To what extent did the
implementation of the model
vary across settings?
To what extent can all the
changes be implemented?
What are barriers and facilitators
to implementing the changes?
Approaches to assess attribution 16
http://www.academyhealth.org/evaluationguide
Ethiopia Health Care Quality Initiative17
Ethiopia Country Background
Total Population,
2017: 94,228,000
GDP per capita, 2017:
US$660
Neonatal Mortality
Rate, 2016: 29 per
1000 live births
Maternal Mortality
Ratio, 2016: 412 per
100,000 live births
Facility delivery: 26%
Sources: World Population Review, WHO, EDHS 2016
Ethiopian Health Care System
Ministry of Health
•Agencies
Regional Health Bureaus
Zonal Health Department
Woreda Health Office
Improvements in Maternal and Child
Health
Ambitious initiatives of the FMoH led to a two-thirds decrease in
child mortality between 1990 and 2012, thus achieving Millennium
Development Goal 4 three years before the target year (2015)
Critical progress in access and coverage
However, rates of neonatal and maternal mortality remain
unacceptably high
Further progress will require more system-level change
Across health system levels
Across the MNH continuum of care
Move beyond coverage high quality, patient-centered, equitable care
Program Components
Creation of Ethiopian National Health Care Quality Strategy with
the Ethiopian FMOH
– Aligned with the Ethiopian Health Sector Transformation Plan
– Builds on the existing quality and equity initiatives in the country
Activate a culture of continuous improvement at all levels of the
healthcare system
– Multi-level QI capability building training activities
Launch and test large-scale results-focused collaboratives in
maternal and neonatal health
– Demonstrate impact of QI methods to accelerate change in key priority area
– Create scalable woreda-wide model for operationalizing QI for national scale-up
Overall Project Driver Diagram
Improved Healthcare
Outcomes and Improved Quality of
Health Services
Institutionalized Culture of Quality
Development and Implementation of Unified
Quality Strategy
Government Ownership
QI Capability Building across the Ethiopian Health
Sector at All Levels
Demonstration of Use of QI on One Priority Agenda
(MNH-focused Collaboratives to Reduce Maternal and Newborn
Mortality)
All Other Drivers of Quality Services
(WHO Building Blocks)
MNH Collaborative Aims
Reduce maternal and
neonatal facility-based
mortality in participating
sites by 30% over a
period of 30 months
Improve quality of antenatal
care, delivery management,
and postnatal care
Reduce
maternal and
neonatal
mortality across
Ethiopia by 30%
over a period of
5 years
5 years
Improve management of
complications related to
leading causes of maternal and
neonatal death
Improve demand for care
services through reduced
delays in seeking and reaching
quality care
30 months
Habits of Continuous Improvement Culture of Continuous Improvement
Short-term Aim
End of Prototype Phase
Medium-term Aim
End of Test-of-Scale Phase
Long-term Aim
End of 5 Years
MNH Collaborative Driver Diagram
Reduce maternal and
neonatal facility-based mortality in
participating sites by 30% over a period of 30 months
Increased Health Seeking Behavior
Optimize the ability of the HEW to educate the community
Community Engagement for awareness creation and positive influence
Utilize the Health Development Army structure to reach the house hold
Use culturally acceptable strategies to improve dissemination and uptake of key
health messages
Use schools as a dissemination mechanism
Use multimedia for Health education activities
Create positive experiences through every health encounter
Use each facility visit to educate/counsel mothers towards raising their health seeking behavior
Improved experience at care
Improved mechanisms to reach appropriate level of health care facility
Improved referral network
Improving transportation mechanisms (ambulance and others) for immediate
response
Maximizing the potential of nearby health facilities to avoid unnecessary referral
Improved quality of care at health
institutions (safe, effective, patient-centered, timely,
efficient, equitable)
Create a culture of QI and leadership
Create structure (QI teams, committees, plan) to facilitate and execute work
Improve data quality through DQA’s
Create a learning platform for collaboration and routine use of data for improvement
Increase the skills of health professionals and health managers to use QI methods and tools
Organize learning collaborative among health facilities serving the same geographic areas (full Woreda
Coverage)
Availability of skilled and respectful health personnel
Training in key MNH national protocols
Onsite mentorship to maintain skills and address skills gaps
Maximize efficiency of existing facility staff
Professionals get regular updates on the management and prevention of key causes of mortality
Improve the reliability of the supply chain management system to deliver essential
commodities all the timeAddress gaps in essential commodities as defined in
baseline assessment
Availability of national guidelines, clinical protocols and job aids
Dissemination of existing protocols and support for local development when necessary
Timely identification, prevention and management of life threatening conditions
to mothers and newborns
Fast tracking/triaging/follow-up mechanism
Reliable implementation of labor and delivery bundle
Reliable implementation of the "MNH" checklists/relevant guidelines
Support for a care delivery system that ensures respectful care for patients
Incorporation of compassionate and respectful care (CRC) change ideas and training in learning sessions
Clean, safe, comfortable spaces for patients and staff
PHCU + Hospital Unit (“scalable unit”)
HP HP HP HP HP HP HP HP HP HP HP HP HP HP HP HP HP HP HP HP HP HP HP HP HP
Health Center Health Center Health Center Health Center Health Center
Primary
HospitalReferral
Hospital
Woreda Health
Office
Referral Hospitals send 2 teams:
-Neonatal (5 people)
-Maternal (5 people)
Primary Hospitals (when
present) send 1 team
HC and linked HP send 1 team:
-3 people from HC
-1 from each HP
WHO sends 1-2
officials to participate
in LS
X2X1
1 Collaborative
includes 7-11 QI teams
(depending on # of
participating hospitals)
HP = Health Post
Learning Collaborative Design
Learning
Session 3
Learning
Session 2
Intensive coaching to support teams to improve system and skills gaps
(visits, phone calls, engagement of program and supervisory
managers, data collation & interpretation)
12-18 months
Learning
Session 1
Address
gaps in
clinical
and QI
skills and
supplies
(training
and
procure-
ment of
essential
supplies)
Finalize
change
package,
publicize
& spread
Learning
Session 4
Action
Period 2
Action
Period 3
Action
Period 1Identify
focus area
and core
indicators
Conduct
Baseline
Assess-
ment
Learning Collaborative Core IndicatorsProcess
Measure
Data
Source*
ANC
(coverage) Antenatal care coverage – four visits
HMIS
ANC
(quality) Percentage of pregnant women tested for syphilis during ANC1
HMIS
Delivery Management
(coverage) Proportion of births attended by skilled health personnel
HMIS
Delivery Management
(quality)* Proportion of deliveries with 100% compliance to ‘on admission’ bundle
Chart review
with checklist
Delivery Management
(quality)* Proportion of deliveries with 100% compliance to ‘before pushing’ bundle
Chart review
with checklist
Delivery Management
(quality)* Proportion of deliveries with 100% compliance to ‘soon after birth’ bundle
Chart review
with checklist
Delivery Management
(quality-sick
newborns) Proportion of neonates treated for birth asphyxia
HMIS or
Delivery
register
PNC (coverage)Percentage of women who attended postnatal care 48 hrs after delivery
HMIS
PNC
(quality-sick
newborns) Proportion of neonates treated for sepsis
HMIS or
Delivery
register
PNC
(quality-
preterm/LBW) Percentage of preterm or low birth weight infants put on KMC
Delivery
register
Clinical Outcome/Impact Measures
Institutional and community level (when available)
measures of:– Maternal Mortality
– Neonatal Mortality
– Stillbirths
– Perinatal Mortality
Phased Design
Prototype: – Design and refine district-wide QI approach to catalyze change in
the key priority area of maternal newborn health
– Gain experience in different regions
– Gain experience in different geographic archetypes (agrarian, urban, pastoralist)
– Build will for change and QI capability at all levels
– Produce contextualized change packages
Test of Scale:– Test the scalability of the prototype-designed approach by
integrating more completely into the routine system
– Examine role of LEAD and university hospitals in scale-up design
– Prime the system for full-scale up and explore structures for scale-up (RHB, ZHB) and lay the plan for scale-up (phase 3)
IHI MNH Initiative Regions
IHI MNH Initiative Regions
= Prototype Phase Woreda
HC + HPs in Zana Woreda, Tigray
(with Last 10 Kilometers)
(pop. 144,246)
Launched November 2016
Fogera Woreda, Amhara
(pop. 264,512)
Launched April 2017
Progress to Date: Learning Collaboratives
Limu Bilbilu/Bekoji Woreda,
Oromia (pop. 237,820)
Launched April 2016
Tanqua Abergele Woreda,
Tigray (pop. 107,081)
Launched August 2016
Duguna Fango Woreda,
SNNPR (pop. 118,051)
Launched October 2016
Amibara, Afar
(pop. 94,718)
Launching January 2018
Prototype Phase (18 months)
= Agrarian
= Urban
= Pastoral
Key
Test-of-scale Phase (18 months)
= Agrarian
= Urban
= Pastoral
Key
Reduce
maternal
and
neonatal
facility-
based
mortality in
participatin
g sites by
30% over a
period of
30 months.
External factors - What factors outside of the project may be a barrier or
facilitator to reaching your desired outcomes?
Turnover in health facilities, low health-seeking behaviors, low rates of facility
deliveries, shifting baseline due to pastoral communities, political stability in
regions of implementation
Assumptions – What is necessary in order for this project to proceed and see results
as planned?
FMoH, Regional, and woreda-level leadership, woreda-level change agent for joint
coaching with IHI PO in prototype phase, engaged woreda-level coach in TOS phase,
will for improvement at health facilities
Improving Maternal and Neonatal Health in EthiopiaInputs Activities Outputs
Short term
OutcomesMedium term
Outcomes
Long term
Outcomes
Improve reliability of care
processes for maternal health:
Antenatal care
- Promote early registration of
pregnant mothers
-Increase subsequent ANC visits
- Screen, prevent and treat
pregnancy-related conditions and
complications e.g. APH,
hypertension, HIV, Anemia,
Malaria etc.
Labour & delivery
-Increase % of skilled deliveries
-AMTSL
-provide compassionate and
respectful care
-Screen, prevent and treat L&D
conditions/complications e.g.
obstructed labor, ruptured uterus,
pre-eclampsia/eclampsia, PPH,
PROM,
Postnatal Care
-Immediate breastfeeding
-Early postnatal care
-Routine subsequent postnatal
care
Improve reliability of care
processes for newborn health:
-Prevention of prematurity
-Routine care of newborn
-Screen and manage
complications i.e. Pre-term care,
Sepsis care, Asphyxia care etc.
-Routine postnatal care including
vaccinations
Community engagement
-postnatal follow ups of
mother/baby pair in the community
-compassionate and respectful
care at all levels of facility-based
care
Referral systems
-Strengthen referral and
transportation system
National Quality Strategy
-Conduct assessment of current health system with regard to:
data systems: leadership and functionality; existing QI
initiatives
-Co-facilitate stakeholder sessions for syndication to gain buy-
in
-Co-develop strategy with FMOH, inclusive of implementation
guidelines and evaluation metrics and development of a
Patient Rights Charter
Improvement Collaboratives
Set-Up: Define clinical bundles and select core indicators with
MOH approval; create all QI coaching/clinical mentorship
tools, create program monitoring and implementation tools;
analysis of existing strategies; baseline data collection;
identify early adopters
Prototype Phase: Test promising changes with
representative slice of health system via collaborative
woredas in 3 regions (Oromia, Tigray, Amhara); test
measurement system, leadership engagement, data system;
test integrated clinical and QI mentoring
Test of Scale Phase: Expand to 21 woredas in 5 regions
(Oromia, Tigray, Amhara, SNNPR, Afar); continued mentoring
of prototyping sites; test and further develop data systems and
other infrastructure needs required for scale-up; engage test-
of-scale implementing partner (TBD); test model of leveraging
existing health structures for QI approach; begin integration
with NQS
Go to Full Scale: Fully leverage existing structures and
meetings, add more scalable units within each of the existing
5 regions; expand to 3 new regions in first year; expanding to
the remaining 4 regions in the second year; fully integrate with
NQS
Capacity Building
In each phase, build leadership, managerial, and point of care
capacity needed for scale up to next phase via:
- 9 L&F course waves for WoHO, and facility-level staff
- 5 QILM courses for FMoH, RHB, and WoHO staff for each
region
- 1-2 waves of IHI’s IA Course for coaches at the national and
regional level, IHI senior project officers, M&E officers
- 3 Senior Leaders’ QI Courses for leaders at the national level
- 5 Data Quality Trainings at each prototype LS2 for facility-
level staff working with data.
Community Engagement And Education
-Work with public education company to develop educational
radio or TV dramas
-Engage HEWs to register pregnancies, promote ANC skilled
deliveries, and PNC; build data collection and QI skills
-Develop client satisfaction feedback mechanism
Measurement And Evaluation
Internal and external evaluation with development of
operational research agenda to optimize local engagement
Individual health
worker:
-Increase knowledge
and skills in QI,
testing change
ideas, collecting real
time data for
improvement and
using data for
decision making.
-Increase clinical
knowledge and skills
-Learn promising
practices from peers
and other change
packages
Team:
-QI team formation
-Ongoing QI team
meetings and team
activities e.g. testing
of ideas, data
collection etc.
Community:
-Increase care-
seeking behavior for
preventative and
curative
maternal/newborn
care services (pre-
conception, ANC,
delivery, and PNC)
-HEWs increase
tracking of data
Health System:
-National Quality
Strategy to
institutionalize
sustainable QI
-Integration of
quality structures
and quality body
-Improved data
quality
National Quality Strategy
-Assessment for strategy
-Co-developed NQS document
Improvement Collaboratives
Set-Up: Initial bundles ready for testing and
core indicators selected; clear roles for
stakeholders; early adopters engaged
Prototype Phase: Locally developed/tested
change package; 150 health staff across three
regions engaged in QI teams participating in
collaboratives; learning shared across groups;
methods for building needed infrastructure
identified; full FMoH ownership
Test of Scale Phase: Standardized process
for integrating initiative into existing systems;
standardized materials, including manual for
coaching QI team meeting, reporting template;
locally adapted QI training and reference
materials printed for distribution
Go to Full Scale: How-to guide for
implementing change package nationally; QI
fully integrated into health system structures
Capacity Building
-20 FMoH and RHB QI coaches trained to
support prototype
-15-30 IAs to lead and monitor quality activities
nationally
-150 senior leaders with enhanced
understanding of QI in healthcare
-Up to 180 skilled improvement coaches to
lead QI teams
-Up to 150 regional, zonal, and district staff
with bolstered QI leadership skills
-Up to 1,500 regional, district, and point-of-
care staff with working knowledge of QI and
capable of infusing it into standard review
meetings
-150 trained data quality experts
Community Engagement And Education
-Radio or TV program to spread messages
related to Maternal Newborn health and/or
respectful maternal care.
-Client satisfaction feedback standard
materials
Measurement And Evaluation
Baseline data, regular opportunities to reflect
on progress toward aims
Office
Space:
-Central
office in
Addis
Staff:
-Staff at
central Addis
Office
-International
and US-
based staff
and faculty
Operations:
-Registration
in Ethiopia
Partners:
-FMOH,
RHBs,
ZHB’s,
WoHO’s
-L10K
-Evaluation
partners:
CPC, IDEAS
Tools
QI How to
Guides (co-
developed by
IHI and
Aurum
Institute
South Africa
QI
methodology
tools
harnessed
from other
projects
Context
Ethiopia has
made great
progress
reducing child
mortality,
however
neonatal
mortality rate
and maternal
mortality ration
remain high
In partnership
with the
FMOH, we
plan to use QI
to accelerate
improvement
building on
assets and
strengths of
Ethiopian
health care
system and
working with
partner
organizations.
Real-time coaching
Reporting to inform
mentorship
Monitoring of program
activities (M/E)
Formal evaluation and implementation
research
Efficient Tool Development and Use for
Action
Tools for onsite clinical/QI
mentorship
Tools for data extraction
and collection
Tools that aggregate
data
Collaborative-level
reports • Collaborative Dashboard
• IHI Program Dashboard
• Facility Workbooks
• Program Monitoring Tool
• Clinical bundle collection tool
• Quarterly complications deep dive tool
• Medication and Equipment Survey
• Safe Childbirth Checklist and Clinical
mentorship checklists (ANC, delivery,
PNC)
• Aim Statement Template
• Driver Diagram Template
• Fishbone Template
• Measures Template
• PDSA Template
• Run chart Worksheet
• Learning Session assessments
IHI Tool and
Data Flow
Guiding Documents:• M/E Framework (indicator
definitions)
• Implementation Manual
Routine Programmatic Data Use
Supervise program and support technical staff (ie, monitor program delivery, gaps, challenges, and new approaches)
Develop change package and anticipate areas in need of further testing
Direct QI coaching to facilities with greatest performance gap (ie, QI activity engagement, indicator performance)
Document critical externalities – data quality, changes in leadership and governance, massive flooding in Amhara (*can add photo to slide), cholera outbreaks
36
Routine Programmatic Data Use
Example – baseline assessment revealed 0 neonatal
deaths in past year in 1 district
– Response: Engaged district political leadership to lead
discussion on underreporting pressures and create commitment
and ‘safety’ in honest reporting moving forward
Track progress in core quantitative measures, address
challenges with participants and facilitating leaders, and
celebrate successes
37
Evaluation38
Evaluation Aims
Mixed Method evaluation encompasses “prototype” and “test of scale” phases with following aims:
1. Describe the intervention as implemented in each phase and changes as approach is scaled
2. Understand the mechanisms of action of the QI approach, how individuals and teams change over time and how teams function within the QI approach, and health care worker motivation
3. Evaluate the impact of the intervention on facility level MNH quality of care and outcomes during the ‘prototype’ and test of scale phases and compare findings
4. Assess the cost-effectiveness of the intervention
39
Leverage Partnerships
FMoH – lead implementer and primary end-user of results
IHI – co-lead implementer, overseeing overall evaluation design and coordination
Addis Ababa University – key local research partner to support design, data collection, and analysis
University of North Carolina – key international partner with experience in complex QI program evaluation (mixed methods)
London School of Hygiene and Tropical Medicine (IDEAS) - key international partner with experience in assessing the how and why of QI (mixed methods)
40
Four Evaluation ComponentsDifferent institutions lead sub components, with IHI overseeing all, and AAU supporting all:
1. Quantitative Impact Analyses:
– to determine whether the intervention is leading to improved MNH processes and outcomes
– to understand what program and facility level factors lead to improved program implementation and quality of care
2. Qualitative assessment of maternal Perception of Care and utilization of Services (experiential quality)
3. Mixed-methods assessment of Individual Change and Team Functioning:
– the change in knowledge and attitudes among LS participants (quantitative)
– the functioning of QI teams (quantitative)
– health worker motivation as it relates to QOC (quantitative)
– how QI leads to change (qualitative)
4. Cost-Effectiveness Analysis
Relevance
Results will help determine:
– ultimate degree of institutionalization of the approach
– appropriateness/readiness for full scale-up locally and globally
– contribute to the limited literature on the mechanism of action of
QI and program effectiveness at scale
42
Operational Research Agenda
Evaluation linked with Research Capability Building
program, to ensure local implementers and leadership
are deeply involved in evaluation and publication – operational research agenda supports development of local
implementation scientists and data use for decision-making
Ministry partners in program design and implementation
also prioritized understanding impact of initiative
Development of dissemination plan in advance with open
dialogue
Example Dissemination Plan44
Conclusions
Integration of evaluation into program allows for the most intentional high quality data collection that can serve:– Program implementation needs
– Routine internal evaluation (M/E) to adapt implementation in real-time to meet the program aims and patient needs by rigorously documenting process
– Formal evaluation (generalizable learning, local and global publications and policy implications)
Less is more– Death by documentation can lead to poor quality data – think
intentionally about every single data element to be collected (purpose – program only, program + internal evaluation, program + internal evaluation + formal evaluation)
Implementation science and research requires rigorous implementation and rigorous evaluation married together in harmony
45
Recommendations
Complex programs require complex evaluation designs
Create a rigorous design and plan, build in times for data/experience review and course correction
“Optimistic realism” to the realities of program implementation
Be intentional and document adaptations as you go to allow for the most rigorous analysis and data interpretation
Maximize partnerships and have open dialogue on roles and expectations for research outputs (paper list, authorship teams)– Allows for building in meaningful learning and experience
46
47
Questions?