how to predict overdose death with pdmp data and advanced … · 2018-12-03 · disclaimer •dr....

How to Predict Overdose Death with PDMP Data and Advanced Analytics

Live Webinar

Thursday, March 16, 2017

Sponsored by:

Live Webinar3/16/17 | 1:00 p.m. ET

Live Webinar3/16/17 | 1:00 p.m. ET

Q+A – Submit a question, located below the slides

Resources List – Access website links and download slides

Help – Submit any technical issues, located below the slides

TwitterJoin the discussion on Twitter!

Live tweet using the hashtag #RXLiveWebinar

Live Webinar3/16/17 | 1:00 p.m. ET

How to Predict Overdose Death with PDMP Data and Advanced Analytics

A cooperative effort between OARRS and Appriss Health

Speakers

Chad Garner, MS• OARRS Director

Jim Huizenga, MD• Emergency Physician, BCEM• Chief Clinical Officer for Appriss Health

David Speights, PhD• Ph.D. Biostatistics• Chief Data Scientist for Appriss

Disclaimer

• Dr. Huizenga and Dr. Speights are both employees of Appriss Health.

• Mr. Garner is the PMP director for the Ohio Automated Rx Reporting Service (OARRS).

• He certifies that he has NO affiliations with or involvement in any organization or entity with any financial interest (such as honoraria; educational grants; participation in speakers’ bureaus; membership, employment, consultancies, stock ownership, or other equity interest; and expert testimony or patent-licensing arrangements), or non-financial interest (such as personal or professional relationships, affiliations, knowledge or beliefs) in the subject matter or materials discussed in this manuscript.

Objectives

• Explain how PDMP data and advanced analytics can impact detection of unintentional overdose deaths

• Identify comprehensive data results, from a variety of complex data patterns, for early detection of overdose risk

• Review the early identification process of prevention and management of substance use disorders in the U.S. and OARRS

Overview

• Data Overview

• Initial Analysis

• Secondary Analysis

• Advanced Data Analysis

• Summary and Future Directions

Perspective

“Treatment without prevention is simply unsustainable”

Bill Gates

Data Overview

OARRS Operational Overview

• Operational since October 2, 2006

• Collects information regarding all Schedule II-V controlled substances dispensed by Ohio-licensed pharmacies

• 25 million prescriptions, 4 million patients per year

• Patient-identifiable data kept for 3 years

• Patient-deidentified data kept indefinitely

Data Overview - Decedents

2014 Death Data

• Data Sample

• 2,482 unintentional overdose deaths

• Inclusion Criteria

• Department of Health determined death due to unintentional overdose between Jan 1 – Dec 31, 2014

• Matching ID within OARRS

• Overdose death data not available until June 2015

• OARRS keeps 3 years of patient-identifiable data

• Selection results

• 1,687 decedents

Initial OARRS Analysis

Microsoft SQL Server Analysis Services was used to create predictive data models across 12 data measures using 4 different algorithms.

1. Microsoft Decision Trees*

2. Microsoft Clustering

3. Microsoft Naïve Bayes

4. Microsoft Logistics

Output was used to trim the data model to 4 data measures showing strong association with overdose death.

Initial Analysis

Four risk factors strongly associated with OD death

Risk Factor Odds Ratio

Pharmacies ≥ 4 3.7

Benzo / Opioid overlap ≥ 35 days 2.4

Max MED ≥ 100 2.3

Cash Payment ≥ 1 2.2

All 4 present 10.3

Odds and Odds Ratios

What are odds?

• Chance of an event occurring divided by the chance that the event won’t occur

• If the chance of something is small then it is approximately equal to the probability

What are odds ratios?

• Odds ratios are a ratio which compares one group to another group and is used to express relative risk

Odds and Odds Ratios - Example

Example

Odds Ratio of group 2 compared to group 1

1.52 / 0.07 = 21.71

Group 2 is more than 20 times as likely to suffer the outcome as Group 1

Group Chance Of Outcome Odds

1 0.07% 0.07% / 99.93% = .07%

2 1.50% 1.50% / 98.50% = 1.52%

Initial Analysis

Four risk factors strongly associated with OD death

Risk Factor Odds Ratio

Pharmacies ≥ 4 3.7

Benzo / Opioid overlap ≥ 35 days 2.4

Max MED ≥ 100 2.3

Cash Payment ≥ 1 2.2

All 4 present 10.3

Secondary Analysis

Using Narx Scores as a predictor of overdose death

• Type specific use indicators for narcotics, sedatives and stimulants

• Range from 000-999

• As the score increases, so does the presence of:

• Providers

• Pharmacies

• MME

• Overlaps

Narx Scores in vivo

Narx Scores in vivo

Narx Scores In Workflow

Narx Scores

Narx Score Distribution

Scores

< 200 200-499 500-650 >650

Narx Scores Summary

Narx Scores are

• Numerical representations of PDMP data that capture “use”.

• Information at a glance.

Narx Scores are not

• Rules (they are tools).

• Are not synonymous with abuse.

Secondary Analysis

Methodology – Narx Scores as a predictor of overdose risk

• 100:1 case / control study

• Determined the highest narcotic score in the year prior to death for each decedent and for 100 date-matched living controls

• Calculated Odds Ratios

Secondary Analysis

Results

Narcotic Score Odds Ratio

0-199 1

200-299 6.4

300-399 7.4

400-499 10.2

500-599 15.5

600-699 23.3

700-799 29.8

800-899 37.7

900-999 63.4

Primary and Secondary Analysis Summary

Using traditional techniques that include both red flags and a composite use indicator, we were able to determine significant associations with unintentional overdose death

• Initial analysis identified 4 red flags strongly associated with overdose death risk

• Secondary analysis strongly associated Narx Scores with overdose death risk

Machine Learning ApproachData Overview

Used the same data from the secondary analysis

• 1:100 case to controls

• Artificial resultant 1% incidence of disease

Applied machine learning and other predictive techniques to develop a 3-digit score similar to Narx Scores, termed an Overdose Risk Score

• Range from 000-999

• Risk doubles approximately every 100 points

• Similar distribution to Narx Scores

Advanced Data Analysis – General Method

Drug Dispensation History and Trends

Types of Narcotics/Sedatives

Prescriber/Pharmacy Visit History and Acceleration

Narcotics to Morphine Equivalencies

Data Inputs

Literature Defined Red Flags

High Risk Behavioral Patterns

Inputs processed through predictive models to determine

the composite risk

Machine Learning Models Optimized

by Simulated Annealing

Decision Engine

Overdose Risk Score

(000 to 999)

Machine Learning ApproachVariable Derivation

Variable Determination• Hypothesize a variable and the expected effect• Develop variable for case and controls• Determine independent predictive ability

More than 70 variables were evaluated using this approach

Examples• Amount of narcotics (in MME) used in the prior 365 days• Amount of sedatives (in MME) used in the prior 60 days

From the 70 variables, approximately one dozen chosen for final model• Some that are used for Narx Scores• Some that are used for Red Flags• Some new variables that look at change over time

Model Validation During Development

• For each decedent and matched control 4 random dates were chosen in the one year prior to the date of death for the decedent producing 4 separate modeling sets to use in model fitting and evaluation.

• Each Set was further split into a 75% training sample and 25% validation sample

Final Model Validation

• After model completion, we used death data from 2013 and 2015 to validate the final model and compare to the 2014 results

Machine Learning ApproachVariable Derivation

Machine Learning Approach Model Evaluation with the KS Statistic

• Kolmogorov-Smirnov (KS) Statistic measures the maximum difference between

the cumulative percentage of two populations (Non-Decedents vs Decedents) by

score.

• Standard metric used in statistics to evaluate models.

0%

100%

Non-Decedents

Decedents

KS

Low HighSCORE

CU

MU

LATI

VE

%

0%

5%

10%

15%

20%

25%

30%

35%

40%

0 to 50 51 to100

101 to150

151 to200

201 to250

251 to300

301 to350

351 to400

401 to450

451 to500

501 to549

550 to599

600 to649

650 to699

700 to749

750 to799

800 to849

850 to899

900 to949

950 to999

Perc

ent

of

Pop

ula

tio

n

Overdose Model Score

Model Scores for Decedents and Non-Decedents

Non-Decedents Decedents

Model Score Distribution for Decedents and Non-Decedents

DecedentsMean: 505Median: 50795th %tile: 83599th %tile: 938

Non-decedentsMean: 209Median: 20995th %tile: 56999th %tile: 730

Decedents have higher risk scores

Model KS Sample 1 KS Sample 2 KS Sample 3 KS Sample 4 Avg. KS

Overdose Risk Score 75%/25% train/test 47.32 48.80 46.23 47.87 47.56

Overdose Risk Score 100%/100% train/test 48.34 49.62 47.57 48.27 48.45

• KS was evaluated on all four test samples (25% holdout group)

• During final testing, models were fit/tested against the full sample

• Many commercial models have KS scores in the 35 to 50+ range

KS Score was 49.62

Cu

mu

lati

ve P

erce

nt

of

Pop

ula

tio

n

KS Plot from Sample 2

Machine Learning ApproachModel Evaluation with the KS Statistic

Machine Learning ApproachSome Key Model Variables

The chance of overdose is 48 times higher when visiting 7+ pharmacies compared to no pharmacies

Group visiting 7+ pharmacies has 9% decedents

(compared to average of 1%)

Advanced Data Analysis

The chance of overdose is 44 times higher when visiting 11+ providers compared to no providers (in the last 2 years)

Group visiting 11+ providers has 9% decedents

(compared to average of 1%)

0%

1%

2%

3%

4%

5%

6%

7%

0 to 483 mg 484 to 966 mg 967 to 1449 mg 1450 to 1932 mg 1933 mg +

Perc

ent

Dec

eden

ts

mg of Sedative Prescribed in the Past Year

Some Key Model Variables

Group having 1933+ mg of sedatives in the last year has 6% decedents (compared to average of 1%)

Machine Learning ApproachFinal Score Construction

• Core model output is converted to a scaled score from 0 to 1000

• Low values = low risk

• High values = high risk

• Risk of overdose death doubles approximately every 100 points

0%

5%

10%

15%

20%

25%

0-99 100-199 200-299 300-399 400-499 500-599 600-699 700-799 800-899 900-999

Perc

ent

Dec

eden

ts

Overdose Risk Score

Predictive Power of Overdose Risk Score

Overdose risk doubles every 100

points (approx)

Individuals in the last bin are 329 times more likely to die of overdose than

individuals who score 0-199

Comparison of Odds Ratio for Drug Overdose Death

329 times more likely to die due to

drug overdose than people with score < 200

OD RiskPERCENT OF DECEDENTS

(1:100 Ratio Overall)ODDS RATIO

0-199 0.1% 1

200-299 0.7% 10

300-399 0.9% 12

400-499 1.9% 25

500-599 3.3% 44

600-699 6.3% 85

700-799 10.0% 141

800-999 13.2% 194

900-999 20.5% 329

0

50

100

150

200

250

300

350

Od

ds

Rat

io

Score

Overdose Risk Model Score strongly associated with risk

of overdose death

Machine Learning ApproachValidation of Overdose Risk Model Scores

Testing on years not used in the modeling process

shows stability of the model

NarxCare Example

Future Directions

1. Include More Data Types• Demographics

• Non-fatal overdose

• Criminal justice (CJ) data where appropriate

• Claims data

• CCD

• Mandatory reporting (NAS)

2. Include More Data• Invite more states

Future Directions

3. Study more outcomes• Non-fatal overdose

• Misuse and substance use disorders

• Arrest, Injury, etc.

Goal is to eventually detect risk at the earliest possible point to provide an optimal intervention point.

Questions?

Chad Garner, MS• Email: chad.garner@pharmacy.ohio.gov

Jim Huizenga, MD• Email: jhuizenga@apprisshealth.com

David Speights, PhD• Email: dspeights@appriss.com

Contact Information

General Webinar Information:

webinars@vendomegrp.com