Core Components for Program SuccessFrank Federico, RPh

This presenter has nothing to disclose

March 4, 2017

Objectives

List the core components necessary to support the success of a program

Discuss how to use the core components as a foundation for improvement

work

Examine one of the programs you are leading in your organization to

determine which components may need further development

3

1) Goals

Aim Statement

2) Content Theory

Driver Diagram or Change

Package

3) Execution Theory

Logic Model

4) Data Measurement & Learning

Measurement Plan

5) Dissemination

Dissemination & Spread Plan

What are we trying to accomplish?

What changes will we make that

will result in improvement?

How will we know that a change is

an improvement?

Five Core Components:

The Model for Improvement

Kirkpatrick Framework

1) Experience What was the participants’ experience? Did the

participants have an excellent experience working on

the improvement project?

2) Learning What did participants learn? Did they learn

improvement methods and begin testing?

3) Process/Behavior Did participants modify their behavior? Did they work

differently and see change in their process

measures?

4) Outcomes Did the organization improve its performance (via

outcome measures)?

4

Goals: Aim Statement

How much, by when, for whom?

• An aim statement describes what we expect to

achieve in the timeframe of the project, taking the form

of “how much, by when, for whom”. The system or

bounds of the project are also defined.

• What is your aim statement?

What Is A Program Theory?

A conceptual explanation of how the program, as designed,

will theoretically result in the desired improvement.

Rapid Response Teams (RRTs)

RRTs aim to provide care to deteriorating patients, to reduce the likelihood of ICU

admission and mortality.

Rapid Response Team 1 Rapid Response Team 2

Monitor all patients in the acute care

setting using the Modified Early

Warning Score (MEWS).

When a patient scores 5 or more on

MEWS, this indicates the patient is

deteriorating.

Call for a Medical Emergency Team

(MET) consisting of two ICU nurses

and a physician to examine the

patient within 10 minutes.

Monitor all patients in the acute care

setting using the Modified Early

Warning Score (MEWS).

When a patient scores 4 or more on

MEWS, this indicates the patient is

deteriorating.

Call for a Medical Emergency Team

(MET) consisting of one ICU nurse

and a physician to examine the patient

within 20 minutes.

Core Concepts & Detailed Tasks

MEWS >=5Use a reliable

method to identify

deteriorating

patients in real

time.

When a patient is

deteriorating,

provide the most

appropriate

assessment and

care as soon as

possible

MEWS >=4

2 Nurses

1 Physician

1 Nurse

1 Physician

1 Physician

ActionTheory

Core Concepts Detailed Tasks

and Local

Adaptations

Program Theory

What is the theory of the program you are working on?

What evidence will you need to prove your theory?

Program Theory

Content theory

Execution theory

Content Theory (The What)

What changes will teams make that will result in improved

outcomes?

• Content theory describes the processes or behaviors that, if

adopted, we predict will improve patient outcomes.

• A driver diagram is a visualization of this shared theory,

depicting areas in the system that improvement teams can

modify to drive improvement.

Inputs

Resources

invested and

activities

performed by

each

improvement

agent/entity

Level 1

Participant

Experience

Level 2

Learning

Level 3

Process/

Behavior

Changes

Level 4

Clinical, personal,

organizational

outcomes

Content Theory:What changes will teams make that will result in improved outcomes?

Execution Theory:

What will the improvement initiative do that will lead teams to adopt the

process changes?

Parry et al. Recommendations for Evaluation of Health Care Improvement Initiatives, 2013,

Acad Peds.

Driver Diagram

• QI tool used to outline the system that underlies the process or

outcome you want to improve.

• This ‘map’ helps improvement teams identify change concepts

that are likely to impact the result of interest.

• Three components:

• Aim

• Primary Drivers

• Secondary Drivers

PRIMARY DRIVERS SECONDARY DRIVERSAIM

Increased use of tele-health services

Reduce death,

disability, or

other

preventable

harm among

newborns and

mothers across

six-district

service area

Staff clinical knowledge and skills

Adequate supply of necessary

materials

Use of data for decision making

Expand access

to pre-natal

care

Activate

community

members to

promote health

and connect

peers with

health services

Reliable

delivery of pre-

natal care

bundle

Supportive community structures

Attractiveness of Health Services

Communication & transportation

Remove cost barriers

Spread information to communities

Identify and train community leaders

Open lines of communication

Community-based education

Driver Diagram: Example

Execution Theory (The How)

What will the improvement initiative do that will lead teams to adopt

process changes?

• Execution theory describes the rationale for how the experience provided by

the improvement initiative, the improvement methods taught and other

activities delivered, and the learning applied leads to improvement in the

process or outcome measures

Why Describe Your Execution Theory?

Clarifies theory and strategy

Gets everyone on the same page

Increases intentionality and purpose, sets priorities

Identifies measures that matter to us

Helps identify standard work

Allows for comparison across programs

Many funders require this

Execution Theory

“.. explicit application of theory could shorten the time needed to develop

improvement interventions, optimise their design, identify conditions of

context necessary for their success, and enhance learning from those

efforts.”

A programme theory-driven evaluation approach was used to describe the

processes that might lead to the programme outcomes, and the conditions

under which these processes were believed to operate.

Davidoff et al, Demystifying theory and its use in improvement, BMJ Quality and Safety, http://qualitysafety.bmj.com/content/early/2015/01/23/bmjqs-2014-003627.full

What Is Not an Effective Execution Theory: the Audit

What’s wrong with junior doctor led audit?

A number of studies have shown that the traditional “clinical audit” as

expected in medical curriculums is an ineffective way of improving practice or

changing process—the very thing that it is intended to do.[2] [3] [4] In one

study only 27% of audits were considered to be complete, and only 22%

were reaudited. The steps studied in that paper may lead us to conclude that

only 5% of audits led to any change in the practice or process studied.[2]

This is a worrying finding as it implies that all the other audits were simply

data collection exercises.

Hillman and Roueche, BMJ Careers , 08 Apr 2011 http://careers.bmj.com/careers/advice/Quality_improvement

19

Marian Bihrle Johnson and Pedro Delgado IHI

Example of an Execution Theory

Execution Theory

What was/is your theory for the work you are leading?

Why did you select that method to execute on your theory?

Logic Models

1) What are they?

2) How do I develop one for my improvement project?

3) How can I use one to evaluate my work?

What Are They?

Diagram connection between what we want to accomplish and the

way we plan to do that

Teams use them to:

– Lay out plan

– Expose gaps

– Get multiple parties on the same page

– Increase intentionality

Long-Term Outcomes

Mid-Term Outcomes

Inputs Activities OutputsShort-Term Outcomes

What resources will

be used to support

the project?

What are the main

things the project will

do/provide?

How many and what

sort of observable/

tangible results will be

achieved?

What will occur as a

direct result of the

activities & outputs?

(typically, changes in

knowledge, skills,

attitudes—LEVEL 2)

• Enrollment fees

• Human resources

• Partnership contract

• Grant funding

• Industry partners

• In-kind contributions

• Monthly 1:1 calls

• Quarterly community

calls

• Develop how-to-

guides

• Build change

packages

• Learning sessions

• On-site coaching

• Use web

• Engage participants

(providers, partners,

stakeholders)

• Strengthen leadership

• Build shared vision

• Curriculum materials

developed

• Establish best practice

• Help implementation

• Participants improve

QI theory knowledge

• Running PDSA cycles

• Brainstorming tests of

change

What results should

follow from the initial

outcomes? (typically

changes in behavior,

policies, practice—

LEVEL 3)

• Improved retention

• More effective

classroom instruction

• Increased number of

job placements in

technical fields

• Increased employer

satisfaction

• Increased regional

economic vitality

• Increased diversity

in the technical

workforce

• A more highly skilled

and adaptable

workforce

What long-term

results should follow

from the mid-term

outcomes (typically,

changes in broader

conditions)

Logic Model Template (Created by Lori Wingate • www.evalu-ate.org)

For additional guidance on developing logic models, see:

University of Wisconsin-Extension’s Logic Model Resources: http://www.uwex.edu/ces/pdande/evaluation/evallogicmodel.html

W.K. Kellogg Foundation’s Logic Model Development Guide: http://www.wkkf.org/~/media/475A9C21974D416C90877A268DF38A15.ashx

Examples:

**NOTES HERE

Logic Model

Which of the components of the logic model did you include in

your work?

Which were missing?

Measurement Plan

How will we know that a change is an improvement?

A measurement plan outlines:

• What data we will collect

• How it will be collected

• How it will be analyzed

• Who will receive the feedback

Ideally, a measurement plan identifies one primary outcome metric and up to

four process measures aligned with your aim statement.

Small Multiples

Dig Deeper

We saw the data move in the right direction. Why?

• New hospital

We saw the process measures move in the right direction, but the

related outcomes haven’t moved. Why?

• Carrying out the processes, but want to look into the quality of the

processes

• Lag time between process changes and impact on outcomes

Qualitative Follow-up

“Our surgeons definitely see how this impacts their work. And they

definitely don’t have time for our hourly team meeting and the

webinars, I think, go a little too much outside of their work. So

they tried at first, but have kind of dropped off.” – surgical nurse

from rural hospital

“We could all the see the importance of doing this work, however,

the hospital had signed on to another program that took the

attention of senior leadership.” – senior nurse from small

community hospital

Measurement

What measures have you adopted?

Do you have qualitative measures to understand the impact of

your work on staff?

Dissemination Plan

How will we spread what we are learning?

• Central to our ability to spread our work is a set of outputs that

summarize what was done, where it was done, what impact it had and

how it was achieved.

• With these outputs, our partners, and others are better informed in how to

spread the work in their settings.

• The Dissemination Core Component is focused on developing a

resourced plan to increase the chances that learning from your project

can be spread widely.

Take a moment to reflect

on your own work.

What will you incorporate from

this session into your plans?