orchestrated testing aggregate data€¦ · through collaborative use of improvement...

Through collaborative use of improvement science methods, reduce preterm births &

improve perinatal and preterm newborn outcomes in Ohio as quickly as possible.

Orchestrated Testing Aggregate Data

Michele Walsh, MD, MSE

Percent of Infants with Greater than 10% Weight Loss from Birth Weight in First Seven Days De-Identified Formula Groups

Number of Finnegan Scores Greater than 12 in 24 Hours Prior to Starting Pharmacologic Treatment De-Identified Formula Groups

Highest Finnegan Score in 24 Hours Prior to Starting Treatment De-Identified Formula Groups

Average Length of Stay for All Opioid Exposed Infants De-Identified Formula Groups

Average Length of Opiate Treatment De-Identified Formula Groups

Average Length of Stay for Pharmacologically Treated Babies De-Identified Formula Groups



NAS-Orchestrated TestingGroup 1

Theresa Ruby, MSN, RNC, IBCLCSouthern Ohio Medical Center

Group 1: 22 kcal /low lactose formulaAkron Children’s St Elizabeth/ Mahoning Valley/St Joseph

Cincinnati Children’s Hospital Medical Center

Kettering Soin Medical CenterNCH Dublin Methodist Southern Ohio Medical CenterNCH Mt Carmel St Ann’s Southview Medical CenterNCH Riverside St Rita’s Medical CenterProMedica Bay Park Trumbull Memorial Rainbow Babies & Children’s UC Cincinnati

Akron Children’s St. Elizabeth

Mahoning ValleySt Joseph’s

Kettering

Rainbow Babies & Children’s

NCH Dublin Methodist

NCH Mt. Carmel St. Ann’s

NCHRiverside

ProMedica Bay Park

Cincinnati Children’s Soin Medical Center

Southern OhioMedical Center

Southview Medical Center

St. Rita’s Medical Center

Trumbull Memorial

UC Cincinnati




Gail Bagwell, DNP, APRN Nationwide Children’s Hospital

Group 2: 22 kcal /not low lactose formulaBethesda North Licking Memorial HospitalDayton Children’s Mercy Children’s HospitalGood Samaritan Tri-Health Nationwide Children’s Hospital

NationwideChildren’s

Bethesda North

Good Samaritan Tri-Health Hospital

Mercy Children’s Hospital

Dayton Children’s

Licking Memorial




Mischel Balazs, NNP, CNP-BC Toledo Children’s Hospital

Group 3: 19 kcal /low lactose formulaAkron Children’s Hospital Medical Center Mercy AndersonAkron Children’s Summa Mercy Medical CantonAkron Children’s General Miami Valley HospitalAtrium Hospital Mt Carmel EastAultman Hospital Mt Carmel WestCleveland Clinic NCH Doctors HospitalFairview Hospital NCH Grant MedicalFort Hamilton Hospital OSU Wexner Well Baby/OSU Wexner NICUGenesis Healthcare ProMedica Toledo Normal NewbornHillcrest Medical Center ProMedica Toledo Children’s Hospital

Toledo Children’sToledo Normal Newborn

OSU Well BabyOSU NICU

NCH Grant

NCH Doctors

Mt Carmel WestMt Carmel East

Miami Valley

Mercy Medical CantonHillcrest

Genesis

Fairview Cleveland Clinic

Aultman

Atrium

Fort Hamilton

Mercy Anderson

Akron Children’sACH SummaACH General




Tonia Deacon

Group 4: 19 kcal /not low lactose formulaAdena Regional Medical Center Mercy Health FairfieldElyria UHCMC Mercy Health WestGood Samaritan Dayton Mercy Regional LorainLima Memorial Hospital Springfield Medical CenterMarion General Hospital The Christ HospitalMetroHealth Upper Valley Medical Center

MetroHealthMercy Lorain

Good SamaritanDayton

Elyria

Upper Valley Medical Center

Lima Memorial

Marion

AdenaMercy Fairfield

Mercy West

Springfield

ChristHospital



Data Analysis of Orchestrated Testing

Heather Kaplan, MD

OPQC Phase IIOrchestrated Testing

OT Analysis Team

Heather Kaplan, MD, MSCE

Maurizio Macaluso, MD, DPH Pierce Kuhnell, MS

55

Quality Improvement using the Model for Improvement

Hunches Theories

Ideas

Changes That Result in

Improvement

A PS D

A PS D

Very Small Scale Test

Follow-up Tests

Wide-Scale Tests of Change

Implementation of Change

What are we trying toaccomplish?

How will we know that achange is an improvement?

What change can we make thatwill result in improvement?

Model for Improvement

Sequential building of knowledge under a wide

range of conditions

Orchestrated Testing• OT involves planned testing across multiple sites

(within or across institutions)• Can use factorial design to…

– Be more systematic about simultaneous testing of different change ideas

– Look at the independent and combined effects of different changes

• Standardization of practices and reliable implementation is necessary

• Can result in faster and more efficient learning

OPQC OT Phase II

• Wide scale test of change examining the role of formula in non-pharmacologic care across 54 NICU/SCN sites

• Two change ideas (factors):– Type of formula– Calorie content of formula

• Two “levels” of each factor– Standard Lactose vs. Low-Lactose – Standard Calorie vs. Higher Calorie

Factorial Design

OPQC Factorial Design (22)

Group Calorie Content Formula Type

A 19 kcal/oz Low-Lactose

B 22 kcal/oz Standard

C 19 kcal/oz Standard

D 22 kcal/oz Low-Lactose

The Model for Improvement

Measures: Outcomes

• Primary Outcome– LOS (pharmacologically treated infants)

• Secondary Outcomes– LOS (all infants)– LOTx– Percent pharmacologically treated– Percent infants requiring dose escalation– Weight Loss >10%

These are similar outcomes to the outcomes examined in the RCT of standard vs. 24 kcal formula posted on clinicaltrials.gov (weight loss, LOS, LOTx, Finnegan scores)

Measures: Background Variables• Pharmacologic treatment approach*

– OCHA Morphine– Cincinnati Methadone– Other Morphine– Other Methadone

• Rates of breastmilk feeding• Percent outborn infants+

• Formula compliance• Time of formula initiation (early vs. time of treatment)• NAS Severity (Max Finnegan Score, Scores>12)• Change in typical formula treatment from Phase I to II

*Centers with extreme deviation from these protocols will not be included in analysis+Infants transferring between sites in different groups will be excluded

Analysis

• Main analyses are graphical– Run and control charts

• Explore differences on background variables

• Examine impact according to groups

– Response plots

Run and Control ChartsExplore Background Variables

1 2 3 4 5 6

Background Variable

Groups: y, c, z, x

Group X Group Y Group Z Group C

Run and Control ChartsExamine Impact

Factor 1 High High Low Low

Factor 2 Low High Low High

Low HighFactor 1

Low HighFactor 1

Low HighFactor 1

F2 High

F2 Low

F2 High

F2 High

F2 Low

Low HighFactor 1

F2 High

F2 Low

F2 Low

Response Plots: InteractionsNo Interaction Moderate Interaction

Moderate Interaction Strong Interaction

Replication

• Ultimate confirmation of findings from OT comes from replication across sites over time

• If positive results are seen for a change, process modification at additional sites with subsequent improvement further validates findings

• This is a critical step because conditions in the future (when the findings will be used) are likely different than the conditions during OT

How this works in “real life”• SLUG Bug QI Project with OT

– 17 participating centers– Examine 4 CLABSI prevention strategies

• Tubing change technique (sterile vs. clean)• Hub care monitoring (yes vs. no)• CVC access limitation (limitation vs. no limitation)• CVC removal monitoring (tracking policy vs. no policy)

– Used factorial design (24-1 8 groups)– Focused on reliability…16/17 centers achieved >75%

reliability on these processes

Piazza AJ, et al. Pediatrics. 2016

4 Factors

8 GroupsVariation in

Baseline Rates


Run and Control Charts

3/4 Sterile Tubing Change **Greatest decrease with addition of HC Monitoring+

** **** **+ + + +


Response Plots: Impact

Sterile Tubing Change Decreased CLABSI Rates by an average of 0.51


Response Plots: Interaction

The addition of hub care compliance monitoring with sterile tubing change created the greatest

impact…showing evidence of an interaction


Replication

Courtesy of Lloyd Provost, data unpublished

How this might look for OPQC

Group Calorie Content

Formula Type

#Hospitals

Baseline LOS

Phase II LOS

A 22 kcal LLF 11 17.36 ?

B 22 kcal Standard 5 20.51 ?

C 19 kcal LLF 17 20.00 ?

D 19 kcal Standard 12 21.71 ?

Hypothetical Control Chart

Type LLF Std LLF Std

Calorie 22 kcal 22 kcal 19 kcal 19 kcal

Hypothetical Response Plots

Std LLF19 kcal 22 kcal

Std LLF

19 kcal

22 kcal

LLF Std

19 kcal 18 21 19.5

22 kcal 16 16 16

17 18.5

Formula TypeC

alor

ies

LOS:

Keep in mind• We are examining the effect of 2 factors in a complex

system…there will always be the possibility that other variables (known background variables or unknown variables) account for some of what we see

• BUT, the complexity is also a strength—wide range of clinical conditions improves generalizability

• The strength of this OT approach is that is allows us to examine factors in real settings in a range of conditions (54 sites!)

• This is an new method of learning for OPQC and we have to figure out some things as we go (e.g., best approach to adjust for differences in baseline LOS)

• After June, we need to commit to implement the findings to achieve the full benefit of the OT approach

References for other geeks

Pallotto EK, et al. “Orchestrated Testing: An Innovative Approach to a Multicenter Improvement Collaborative.” American Journal of Medical Quality. 2015 Oct 19, [Epubahead of print]

Piazza AJ, et al. “SLUG Bug: Quality Improvement with Orchestrated Testing Leads to NICU CLABSI Reduction.” Pediatrics. 2016 Jan;13(1):1-12.

Moen RD, Nolan TW, Provost LP. Quality Improvement Through Planned Experimentation. 2nd ed. New York: McGraw-Hill, 1999

Polaris C Polaris F Hallway 1RB&C MetroHealth Dayton Children’sAtrium Mercy Anderson GenesisMt Carmel East Springfield Mercy FairfieldElyria The Christ Hospital LickingKettering NCH Riverside NCH Mt Carmel St Ann’sUC Health St Rita’s Medical Center Marion GeneralTrumbullHallway 2 Hallway 3Nationwide Children’s Bethesda NorthAultman Hospital Fairview HospitalToledo Children’s OSU NICU/OSU Well BabyMercy Lorain Mercy Children’sSoin Medical Center SouthviewMercy West Akron Children’s/Summa

Gemini B Gemini CMt Carmel West Good Samaritan Tri-HealthHillcrest Mercy CantonGood Samaritan Dayton NCH GrantCCHMC Akron St E/Mahoning ValleyDublin Methodist Southern Ohio Medical CenterMiami Valley Hospital Fort Hamilton

**LUNCH is 12:30 – 1:15PM after Storyboard Rounds**

Steal Shamelessly;

Share Seamlessly

Storyboard Rounds

orchestrated testing aggregate data€¦ · through collaborative use of improvement...

Documents