health it summit beverly hills 2014 – case study “the progression of predictive analytics: the...

THE EVOLUTION OF

PREDICTIVE ANALYTICS The Rothman Index

- A Case Study -

Presented by:

Mark Headland Vice President and CIO

Children’s Hospital of Orange County

November 4, 2014

OVERVIEW

• Organizational Bio

• Predictive analytics – defined and progress

• The Rothman Index – history and overview

• Case study – Children’s Hospital of OC

DISCLOSURES

• No personal interest or relationship with PeraHealth other than CHOC’S use of their product.

• All copyrighted slides reproduced and used with the permission of PeraHealth.

CHOC’s Bio and History 1960’s Champion

the need for a pediatric hospital in Orange County

Agree to lease land to CHOC

Sisters of St. Joseph

1991

1964 CHOC opens doors with 62 beds

CHOC North opens

2013

1993

CHOC Children’s at Mission opens

The Bill Holmes Tower opens

Four centers of excellence opened

•Heart •Neuroscience •Orthopedic •Hyundai Cancer

KID PROGRAMS

PatientConnect Program

Partial Snapshot

Tertiary Care 279 Beds

PICU NICU CVICU Hem/Onc

30 67 12 28 l

Med / Surg NeuroScience 82 24

1

5 Primary Care Clinics

30 Specialty Care Clinics

500 residents, fellows and med

students UCI affiliation

Research – 375 active studies

UCI affiliation

Turtle Talk

Seacrest Studio

RECOGNITIONS FOCUS ON EXCELLENCE

Leapfrog Safe Hospital

Beacon Gold Level Award For Critical Care Excellence

Magnet Designation – Nursing Excellence

Cape Award Gold Level for Performance Excellence

Ranked Nationally in Seven Specialties

HIMSS EHR Adoption Model: Stage 6; Site Visit for Stage 7 on 12/3/14

http://www.choc.org/pressroom/index.cfm?id=P00296&nid=720&section=pressroom&keyword=pressroom

http://www.choc.org/pressroom/index.cfm?id=P00296&nid=599

http://www.nursecredentialing.org/Magnet.aspx

http://www.choc.org/pressroom/index.cfm?id=P00296&nid=594

http://health.usnews.com/best-hospitals/children's-hospital-of-orange-county-6932287

Predictive Analytics Defined

Wikipedia: …encompasses a variety of statistical techniques from modeling, machine learning, and data mining that analyze current and historical facts to make predictions about future, or otherwise unknown, events….

… clinical decision support systems incorporate predictive analytics to support medical decision making at the point of care.

…in health care primarily to determine which patients are at risk of developing certain conditions, like diabetes, asthma, heart disease, and other lifetime illnesses.

“…prescriptive analytics”: includes evidence,

recommendation and actions for each predicted category

or outcome . (David Crockett, Health Catalyst)

http://en.wikipedia.org/wiki/Predictive_modelling

http://en.wikipedia.org/wiki/Machine_learning

http://en.wikipedia.org/wiki/Data_mining

http://en.wikipedia.org/wiki/Prediction

http://en.wikipedia.org/wiki/Clinical_decision_support_system

http://en.wikipedia.org/wiki/Clinical_decision_support_system

THE CHANGING NATURE OF INFORMATICS AND PREDICTIVE ANALYSIS

• Moving away from single points of data in episodes of care to incorporating time series data into predictive modeling

• “Changes in vital signs over time are better predictors of cardiac arrest than a snapshot”

Dr. Curtis Kennedy, Asst. Professor of Critical Care, Baylor University

“Ignorance and Blindness as a Strategy to Provide Just-in-Time Life Saving Care to Critically Ill Patients”

Presented at The Pediatric Data / Intelligence Forum 10/13/14

9

APGAR Score

• Newborns

• Simple, Repeatable Assessment

• Manual Calculation

• Criteria: 5 Observations

• Used Widely Today

APACHE II • Adult Patients

Admitted to ICU

• Admission Score Only

• Criteria: 12 Physiological Measures

• In Use Today, along with APACHE III and SAPS II

Braden Scale

• Adult Patients

• Risk of Pressure Ulcers

• Criteria: 6 Observations

• In Use Today

MEWS • Adult Patients


• Criteria: 4 Physiological Measures and 1 Observation

• Built on Expert Opinion

• Limited Use Today

PEWS • Pediatric

Population (up to age 18)


• Criteria: Originally 20 physiological and observation measurements- most hospitals use 4-7

• Built on Expert Opinion

• In Use Today

RI Score

1952 1985

2001

2010

1987

2005

• All Patients • Automated Calculation • Real Time • Disease Agnostic • No Manual Data Entry • Common Clinical Language • Integrated with EHR • Criteria: 50+ Measures

• Physiological • Clinical Assessments • Lab Results • Includes measures of previous

score algorithms • Built on Heuristic Modeling

Leveraging Data to Predict Outcomes….

The Story: The Rothman Index is Created

• Florence Rothman: Avoidable death from undetected complication

• Michael and Steven Rothman: Engineers with expertise in big data analysis and statistics

• EHR Data – Available but untapped

• Heuristic modeling techniques create a universal patient score

10

Turning loss into meaning

“We didn’t want this to happen to someone else’s family.” Michael Rothman, PhD., Co-Founder

Florence Rothman

WHAT IS THE ROTHMAN INDEX?

• Composite score of 50 measures Physiologic data

Clinical assessments

Lab results

• Used to assess a patient’s condition and potential decline

• Real time and automated

• Index is 1 – 100 Significance: low absolute numbers and trending

• EHR agnostic

11

© 2014 PeraHealth, Inc. All Rights Reserved

EXISTING

EHR DATA

Clinical

Assessments

Labs Vitals

Visualizing Patient Condition

Opportunities for Earlier Intervention

PeraTrend Graph

Ro

thm

an In

dex

12

50+ Measures

“ROTHMAN INDEX”

Single Numeric Score

© 2014 PeraHealth, Inc. All Rights Reserved US Patent Nos. 8,092,380; 8,100,829; 8,355,925; 8,403,847 and 8,454,506; and other foreign patents pending

RI Score is Rooted in Proven Science

13

11 Peer Reviewed Studies; Over 12 Oral and Poster Presentations

Validation of RI for Predicting 30 day readmissions

Predicting ICU readmissions

RI Outperforms existing early warning systems (MEWS)

Validity of Clinical Assessments for measuring patient condition

Palliative care trigger

Predicting Surgical Complications

* www.PeraHealth.com/publications

Key Studies*

http://www.perahealth.com/publications

http://www.perahealth.com/publications


The Pediatric Rothman Index (pRI): Development Background

• Development Partners – Children’s Hospital of Pittsburgh: Dr. Jim Levin, CMIO

– Yale New Haven Children’s Hospital: Dr. Allen Hsiao, CMIO

– University of Florida Health: Dr. Joseph Tepas, Chair of Pediatric Surgery

– PeraHealth, Dr. Michael Rothman, Chief Science Officer

• Data Source – 80,000 patient visits

– Children’s Hospital of Pittsburgh data from 2009-2012

– Yale New Haven Children’s Hospital data from 2010-2012

….leverages EHR data to improve safety.

14

Hospitals, Principal Investigators, and Underlying Data


The Pediatric Model

• Index score calculated from vitals, labs1 and nursing assessments2

• Variables are consistent with different criteria associated for age – Gastrointestinal Assessment may have different criteria for a minimum standard for a

newborn and a 3-year old, but we still compute “met” and “not met”

• Of the existing continuous variables we found 5 variables with significant age dependencies

– Heart rate

– Respiration rate

– Systolic blood pressure

– Diastolic blood pressure

– Serum creatinine

• For the five variables above, there are solid biophysical reasons for variation in values for healthy children with age, relating to total mass, muscle mass and surface area

15

Children Present Unique Characteristics

1 Development and validation of a continuous measure of patient condition using the Electronic Medical Record, Michael J. Rothman, Steven I. Rothman, Joseph Beals IV. Journal of Biomedical Informatics, 2013 Oct;46(5):837–48.

2 - Clinical Implications and Validity of Nursing Assessments: A Longitudinal Measure of Patient Condition from Analysis of the Electronic Medical Record – Michael J Rothman, Alan B Solinger, Steven I Rothman, G Duncan Finlay, BMJ Open 2(4) 2012.


Pediatric Rules Engine: Deterioration Rule

16

Base mortality at Hospital A is 0.7%

Rule Definition Mortality (PPV)

Visits Flagged

Very High Acuity

Hits RI<=30 in the past 24 hours

13.7% 4.6%

High Acuity sensitive to rapid changes

Falls 40% within 6 hours

6.6% 9.5%

Medium Acuity sensitive to slower changes

Falls 30% within 24 hours

3.5% 19.9%

1. Rules are hospital- population specific

2. Targeted to specific clinical teams

3. Tied to hospital-defined clinical escalation protocols


A hospitalist rounds. Sees ONE moment in time. Not concerned.

Normally you only see one moment.

17


But she looks at the prior clinical notes… and sees improvement.

Normally you only see one moment.

18


If she had looked back 2 days, she would have seen a sharp decline

pRI would have revealed 2 escalating alerts.

19


The next day she should be concerned. Poor vitals… HR=148, RR=38

But she probably doesn’t know that the patient has also failed 8 of 11 nursing assessments

20


Here’s the big picture… escalating alerts days prior to RRT, ICU and death

Medium Alert

RRT

10-year old, died after an 8-day stay

21

Medium Alert

High Alert

Very High Alert

Patient expired

General decline shown in nursing data. PEWS improves.


Pediatric Rothman Index Rules yield early warnings for interventions

22

80% fire more than 1-day ahead of death

Very High Acuity Rule Triggers Five Days Prior to RRT

24

Primary Diagnosis: Dermatitis due to substances taken internally

1. pRI rule fires 2. RRT Called

Very High Acuity Rule Fires 21 Hours Before RRT

25

PEWS of 4 appears to initiate RRT

1. pRI rule fires

2. RRT Called

Clear deterioration 48 hours prior to Code Blue

26

Medium Alert

Code Blue

6-year old, discharged to a skilled nursing facility. LOS following code blue was 2 months


RI Protocol: Reducing Unplanned 30 Day Readmissions

28

Patients discharged with an RI Score below 70 are 2.7x more likely to readmit within 30 days*

NOTE: Assume use of RI discharge rule can conservatively move the readmission rate for the population with RI<70 to the average. *SOURCE: Identifying Patients at Increased Risk for Unplanned Readmission, Elisabeth H. Bradley, PhD, et al, Medical Care Volume 51, Number 9, September 2013

Discharge Rule Typical Health System Annual Savings: $4.9M

Discharge Rule with PeraTrend

As shown at left, readmission rates increase dramatically for patients released with RI Scores <70. Discharged patients with very low scores expire rather than readmit, so the rate begins to decrease.

Improvement Opportunity

Reduction in Readmission Rate by 8.2%

Assumptions

Average cost of readmission = $13,228

Average health system readmission rate = 16%

Benchmark

Average cost savings for typical health system

$4.9M

(Does not include avoidance of CMS penalties for readmissions)


Multiple Ways to View Pediatric Rothman Index

29

Embedded EHR

Tablet View

PeraHealth Web Portal

Mobile Phone App

PeraHealth Secure Server

Unit Monitor View


PeraTrend Quilt View - 22 patients on a Med-Surg unit

THE CHOC STUDY

31

CHALLENGE TO PERIHEALTH

• Evaluate one year of CHOC data retrospectively

• Validate pRI’s ability to accurately reflect and predict patient condition and risk

• Demonstrate ability to trigger warnings on key cases with good specificity and sensitivity

• Compare pRI to PEWS and demonstrate pRI is a more robust early warning system

• Demonstrate cost benefit of pRI

Data Set Parameters

• 11,467 patient discharges from 7/2012 – 6/2013

• 18,453,698 observations

• 1,052,217 pRI values generated

33

• The Pediatric Rothman Index (pRI) accurately reflects patient condition and patient risk at CHOC

• pRI Rules Engine is a highly accurate tool for warning clinicians in an appropriate manner of patient decline

• pRI significantly outperforms PEWS in providing early warning for critical events, thus providing an opportunity for earlier intervention

• Several PeraHealth identified cases in the data file would provide immediate ROI on the PeraTrend investment

Our Key Findings

34


Mortality Results Comparable

35

Consistent evidence of the pRI to reflect patient risk

CHOC SCH* CHP*

* Seattle Children’s Hospital, Children’s Hospital of Pittsburgh


Mortality Accuracy is Consistent Across Sites

36

CHOC – AUC=0.95

CHP – AUC=0.96

SCH – AUC=0.94


pRI at Admission is Predictive of LOS

CHOC – LOS v pRI at admission

37

Note: acuity increases with decreasing pRI

SCH – LOS v pRI at admission


pRI vs PEWS as Predictor of LOS at Admission

• CHOC – LOS v pRI at admission

38


• CHOC – LOS v PEWS at admission


pRI vs PEWS as Predictor of Mortality at Admission

CHOC – Mortality v pRI at admission

39


CHOC – Mortality v PEWS at admission

A Word About PEWS CHOC Non-ICU Code Rate

1.07

0.48

0.79

0.31

0.10 0.11 0.10

0.03

0.12

0.00

0.20

0.40

0.60

0.80

1.00

1.20

2005 2006 2007 2008 2009 2010 2011 2012 2013

Codes

Per

Thousa

nd P

atient D

ays

Fiscal Year

RRT

PEWS


24 Hour Prediction of Mortality and Code Whites

41

Evidence supports pRI providing early warning; pRI = 30: 3% Code White and 6% Mortality


Very High Acuity

42

Very high acuity rule: patient below 30 in the last 24 hours

Rule fires? Visits % of Visits Mortality Number of Expired

No 11,159 97% 0.1% 13

Yes 310 3% 18.1% 56

TOTALS 11,469 100% 1.7% 69


pRI Warnings Fire Within a Reasonable Timeframe

43

Two thirds fire more than 24 hours before expiration

• The Pediatric Rothman Index (pRI) accurately reflects patient condition and patient risk at CHOC

• pRI Rules Engine is a highly accurate tool for warning clinicians in an appropriate manner of patient decline

▪ 98% of Code White patients

▪ 70% of unplanned transfer patients

▪ 42% of ICU patients

▪ 13% of non-ICU patients

• pRI significantly outperforms PEWS in providing early warning for critical events, thus providing an opportunity for earlier intervention

• Several PeraHealth identified cases in the data file would provide immediate ROI on the PeraTrend investment

Our Key Findings at CHOC

44

pRI is a useful tool in predicting decline in clinical condition and provides opportunity

for early intervention.

CONCLUSION

45

health it summit beverly hills 2014 – case study “the progression of predictive analytics: the...

Education