Implementation Research: What is it? Am I doing it?

How can I do it better?

Olakunle Alonge, Assistant Scientist, Health Systems Program, JHBSPH Jim Ricca, MCSP Learning and Implementation Science Team Leader Stefan Baral, JHBSPH

Learning Objectives

By the end of this session, participants will be able to: • Explain what Implementation Research (IR) is and how it

related to other forms of research • Explain three commonly studied Implementation Research

outcomes • Design answerable Implementation Research questions of

importance to key stakeholders

2

Implementation Practice and Research

• “Implementation [practice] is the act of carrying an intention into effect, which in health can be policies, programmes, or individual practices (collectively called interventions)” e.g. the act of administering misoprostol for AMTSL

• “Implementation research is the scientific inquiry into questions concerning implementation” e.g. study on the acceptability of misoprostol among pregnant women in an area

3 Source: Peters, Adam, Alonge, Agyepong and Nhan, BMJ 2013; Gilbert, JRScT 1991

Implementation Research

Translational Research

Implementation Research and Other Common Studies in Public Health

4

T4 studies

T3 (D&I)

studies

Intervention studies

T2 studies

T1 studies

QI studies

Operation Research

M&E

Source: Lobb and Colditz, Annual Rev Pub Health 2013; IRDS Consultative Meeting Report, 2014

IMPLEMENTATION OUTCOMES: FRAMEWORK AND DEFINITIONS

5

Implementation Outcomes

• Implementation variables: – Constructs whose vector properties (magnitude and direction)

changes with an implementation process

• Implementation outcomes: – Implementation variables that describe effects (results) of a deliberate

and purposive action to implement an intervention – Useful for measuring success or failure of an implementation process – Intermediate outcomes of the effectiveness of an intervention (on

individual/population health outcomes and quality of life) – Underdeveloped constructs; operates across socio-ecological levels

6 Source: Peters, Adams, Alonge et al 2013; Proctol et al. 2011; Fixsen et al. 2005

Conceptual Framework for Implementation Outcomes

Clients Outcome Satisfaction

Symptomatology Function

Population-

Based Incidence of

diseases Morbidity Mortality

DALYs

Health Outcomes

Efficiency Coverage

Equity Responsiveness

Services Outcomes**

Acceptability Adoption

Appropriateness Costs

Feasibility Fidelity

Penetration Sustainability

Implementation Outcomes I*

7

• *measurable both at the individual and aggregated level • **measurable mostly at an aggregated level

Source: Proctor et al 2011

Implementation Outcomes - Definitions

Implementation Outcome

Working Definition* Related terms**

Acceptability The perception among stakeholders (e.g. consumers, providers, managers, policy-makers) that an intervention is agreeable

Related factors: (e.g. Comfort, Relative advantage, Credibility)

Adoption The intention, initial decision, or action to try to employ a new intervention

Uptake, Utilization, Intention to try,

Appropriateness The perceived fit or relevance of the intervention in a particular setting or for a particular target audience (e.g. provider or consumer) or issue

Relevance, Perceived fit, Compatibility, Perceived usefulness or suitability

Feasibility The extent to which an intervention can be carried out in a particular setting or organization

Practicality, Actual fit, Utility, Trialability

8 Source: Proctor et al 2011; Peters, Adams, Alonge et al 2013

Implementation Outcomes - Definitions Implementation Outcome

Working Definition* Related terms**

Fidelity The degree to which an intervention was implemented as it was designed in an original protocol, plan, or policy

Adherence, Delivery as intended, Integrity, Quality of delivery, Intensity or dosage of delivery

Implementation cost

The incremental cost of the implementation strategy (i.e. how the services are delivered in a particular setting); including the cost of the intervention itself.

Marginal cost, Total cost***

Penetration/ Coverage

Extent to which a practice or program is integrated within a service settings and system (niche saturation). The degree to which the population that is eligible to benefit from an intervention actually receives it.

Reach, Access, Service Spread or Effective Coverage, Penetration

Sustainability The extent to which an intervention is maintained or institutionalized in a given setting.

Maintenance, Continuation, Routinization, Institutionalization,

9 Source: Proctor et al 2011; Peters, Adams, Alonge et al 2013

Measuring Implementation Outcomes

10 Source: Proctor et al 2011

Measuring Implementation Outcomes

11 Source: Proctor et al 2011

FIDELITY

12

Fidelity - Definition

• Degree to which an intervention was implemented as prescribed by the original protocol or program developers

• Adherence to the core program components – Core components: elements without which the intervention will not

have its intended effect – Balancing fidelity and adaptation: 100% fidelity may be unrealistic

13 Source: Dusenbury et al 2003, Rabin et al. 2008

Fidelity – Domains of Interest

• May consist of the following domains:

14 Source: Dane & Schneider, 1998; Dusenbury et al 2003; Carroll et al. 2007; Durlak and Dupre 2008; Proctor et al 2011

Dane & Schneider, 1998

Carroll et al. 2007 Proctor et al. 2011 Durlak and DuPre 2008

Adherence Adherence Adherence Adherence

Dose Exposure or dose Dose

Quality of delivery Quality of delivery Quality of delivery

Participant responsiveness

Participant responsiveness

Program differentiation

Program differentiation

Intervention complexity

Facilitation strategies

Fidelity – Domains of Interest

• Domains of interest: – Adherence – an intervention is being delivered as it was designed or written – Dosage or exposure - frequency and duration of the intervention is as full as

prescribed by its designers. Could mean amount of intervention that the recipient received

– Quality of delivery - the manner in which an intervention worker (e.g. teacher, volunteer, or staff member) delivers a program

• Based on techniques prescribed by developer • Based on a theoretical ideal

– Participant responsiveness - how far participants respond to, or are engaged by, an intervention

– Program differentiation - identifying unique features of different components of an intervention, and identifying the core components

• Perspectives in measurement: – Representative (commonest approach) – Comprehensive – Complex

15 Source: Mihalic 2004, Carroll 2007

ACCEPTABILITY

16

Acceptability

• Perception among stakeholders (e.g. consumers, providers, managers, policy-makers) that an intervention is agreeable

– Based on direct knowledge of or experience with specific aspects of the intervention, e.g. content, comprehensiveness, ease of use, comfort, etc.

– Distinguished from service satisfaction which often focuses on general service experience

– Dynamic and changes with experience

• Often conceptualized as consisting a single domain

17

FEASIBILITY

18

Feasibility

• The extent to which an intervention can be carried out in a particular setting or organization

– Typically assessed retrospectively to explain success or failure of an implementation process (actual fit)

– Closely related to appropriateness which looks at the perceived fit of an intervention for a setting (i.e. prospectively assessed)

• Has multiple domains and similar measurement perspectives as in fidelity (i.e. representative, comprehensive, complex)

19

Feasibility – Domains of Interest

20 Source: Bowen et al. AJPM 2009

Feasibility – Domains of Interest

21 Source: Bowen et al. AJPM 2009

Summary – What is IR? / IR Outcomes

• Implementation research is the scientific inquiry into questions concerning implementation

• Such research could be conducted by observing changes in implementation research outcomes (IRO) during the implementation process

• IRO describe effects (results) of a deliberate and purposive action to implement an intervention

• IRO are mediators of individual/population health outcomes • IRO could comprise multiple domains or a single domain, are

defined and measured in context across single or multiple socioecological levels

• They could be measured using quantitative, qualitative and mixed method approach

• Some common examples of IRO are fidelity, acceptability and feasibility 22

FORMULATING IMPLEMENTATION RESEARCH PROBLEMS AND QUESTIONS

23

24

Addressing Public Health Problems

Define the problem Measure the magnitude of the problem

Develop a conceptual framework for the understanding of the key

determinants of the problem

Identify and develop strategies for intervention (program)

Set priorities and recommend interventions (policy)

Implement interventions and evaluate outcomes

Communication strategy

25

Problem Definition Creating a problem definition is an iterative

process • Non-linear process • Changes in the definition are expected • Requires informed discussion • Tests team skills

– Key difference with IR is the involvement of end users in defining the public health problem

Defining a problem is an iterative process and is often the

most difficult step in the problem solving methodology, but a good problem definition goes a long way towards successfully solving the problem

http://www.notablebiographies.com/De-Du/Dewey-John.html

Prioritization of Health Issues for IR

• Priority Rating Formula (Hanlon) – Component A = Size of the problem – Component B = Seriousness of the problem – Component C = Estimated effectiveness of the solution

– Basic Priority Rating

• BPR = (A + 2B) * C

– +/- Component D = PEARL factors (propriety, economic, acceptability, resource availability, legality)

– Overall Priority Rating

• OPR = [(A + 2B) * C]D

27

Specificity of the Problem

• What is the specific dimension or aspect of interest?

• If you think you are being too specific, you probably are not

• Avoid broad, general topics as they run the risk of containing competing concepts

– Eg. How do improve health systems in LMIC? – How to improve pediatric polio vaccine delivery among

rural populations born in 2015 in Northern Nigeria?

Key Aspects of a Problem Definition

• Nature of the problem/focus

• Framing of the problem – Person – Place – Time

• Specificity of the problem

• Causality

• Language considerations

28

29

Framing of the Problem Person • Who is affected by this problem?

• Be as specific as possible…

– …using basic demographics: • Gender • Age • Race/ethnicity • Socioeconomic status (education, income, etc)

– …and using problem-specific demographics • Eg. HIV

– Sexual practices – Drug use – Housing situation – Occupation

Estimates of new HIV Infections in the US, by Transmission Category, 2006

Hall HI, Song R, Rhodes P, et al. Estimation of HIV Incidence in the US. JAMA 2008;300: 520–529.

Place

• Where is the problem located?

• Be as specific as possible: – Geographical – Cultural – Social – Political – Economic – Historical – Ethnic

30

Framing of the Problem

Cholera cases (dots) and outbreaks (orange), 2007-2009

http://gamapserver.who.int/mapLibrary/Files/Maps/Global_CholeraCases0709_20091008.png

Time

• What is the specific time frame for the problem?

• Be as specific as possible – Dates (1990-2000) – Time since a specific event (post 9/11)

31

Framing of the Problem

http://www.cdc.gov/Features/dsSpinaBifida/

Summary Problem Statement

• Summarizes the purpose of the study • Establishes the direction and captures the essence

of the study • Reflects the research design of the study • Leads logically to the implementation research

objectives/questions/specific aims

32

IR Questions

• Of interest to stakeholders – Government, community, academia, implementing partners

• Answerable • Shaped by the problem and in turn shape the design of the

research • Clear, specific, and feasible

• Three Broad Areas of IR Questions

– Describe Health Situation or Interventions – Provide Data to Evaluate Ongoing Interventions or Information

Needed to Adjust Interventions – Analyze missed targets and potential solutions

33

34

Language Considerations

• Be clear and concise • Define and reference technical terms

• Define local terms

• Generally a paragraph rather than a single statement • Incorporate general objectives and use action words to

succinctly outline the purpose of the study – Purpose, Intent, Objective

35

Causal Claims

• Are etiologic factors or causes part of your problem statement?

• AVOID causal claims

• Be short and clear – this is the problem definition, not the answer to the problem

Small Group Assignment

• Statement of the problem for your IR proposal – 1-2 paragraphs

• Formulate one specific IR objective (question) – Your IR objective should include one of the IR outcomes discussed

36