how to predict overdose death with pdmp data and advanced … · 2018-12-03 · disclaimer •dr....
TRANSCRIPT
How to Predict Overdose Death with PDMP Data and Advanced Analytics
Live Webinar
Thursday, March 16, 2017
Sponsored by:
Live Webinar3/16/17 | 1:00 p.m. ET
Live Webinar3/16/17 | 1:00 p.m. ET
Q+A – Submit a question, located below the slides
Resources List – Access website links and download slides
Help – Submit any technical issues, located below the slides
TwitterJoin the discussion on Twitter!
Live tweet using the hashtag #RXLiveWebinar
Live Webinar3/16/17 | 1:00 p.m. ET
How to Predict Overdose Death with PDMP Data and Advanced Analytics
A cooperative effort between OARRS and Appriss Health
Speakers
Chad Garner, MS• OARRS Director
Jim Huizenga, MD• Emergency Physician, BCEM• Chief Clinical Officer for Appriss Health
David Speights, PhD• Ph.D. Biostatistics• Chief Data Scientist for Appriss
Disclaimer
• Dr. Huizenga and Dr. Speights are both employees of Appriss Health.
• Mr. Garner is the PMP director for the Ohio Automated Rx Reporting Service (OARRS).
• He certifies that he has NO affiliations with or involvement in any organization or entity with any financial interest (such as honoraria; educational grants; participation in speakers’ bureaus; membership, employment, consultancies, stock ownership, or other equity interest; and expert testimony or patent-licensing arrangements), or non-financial interest (such as personal or professional relationships, affiliations, knowledge or beliefs) in the subject matter or materials discussed in this manuscript.
Objectives
• Explain how PDMP data and advanced analytics can impact detection of unintentional overdose deaths
• Identify comprehensive data results, from a variety of complex data patterns, for early detection of overdose risk
• Review the early identification process of prevention and management of substance use disorders in the U.S. and OARRS
Overview
• Data Overview
• Initial Analysis
• Secondary Analysis
• Advanced Data Analysis
• Summary and Future Directions
Perspective
“Treatment without prevention is simply unsustainable”
Bill Gates
Data Overview
OARRS Operational Overview
• Operational since October 2, 2006
• Collects information regarding all Schedule II-V controlled substances dispensed by Ohio-licensed pharmacies
• 25 million prescriptions, 4 million patients per year
• Patient-identifiable data kept for 3 years
• Patient-deidentified data kept indefinitely
Data Overview - Decedents
2014 Death Data
• Data Sample
• 2,482 unintentional overdose deaths
• Inclusion Criteria
• Department of Health determined death due to unintentional overdose between Jan 1 – Dec 31, 2014
• Matching ID within OARRS
• Overdose death data not available until June 2015
• OARRS keeps 3 years of patient-identifiable data
• Selection results
• 1,687 decedents
Initial OARRS Analysis
Microsoft SQL Server Analysis Services was used to create predictive data models across 12 data measures using 4 different algorithms.
1. Microsoft Decision Trees*
2. Microsoft Clustering
3. Microsoft Naïve Bayes
4. Microsoft Logistics
Output was used to trim the data model to 4 data measures showing strong association with overdose death.
Initial Analysis
Four risk factors strongly associated with OD death
Risk Factor Odds Ratio
Pharmacies ≥ 4 3.7
Benzo / Opioid overlap ≥ 35 days 2.4
Max MED ≥ 100 2.3
Cash Payment ≥ 1 2.2
All 4 present 10.3
Odds and Odds Ratios
What are odds?
• Chance of an event occurring divided by the chance that the event won’t occur
• If the chance of something is small then it is approximately equal to the probability
What are odds ratios?
• Odds ratios are a ratio which compares one group to another group and is used to express relative risk
Odds and Odds Ratios - Example
Example
Odds Ratio of group 2 compared to group 1
1.52 / 0.07 = 21.71
Group 2 is more than 20 times as likely to suffer the outcome as Group 1
Group Chance Of Outcome Odds
1 0.07% 0.07% / 99.93% = .07%
2 1.50% 1.50% / 98.50% = 1.52%
Initial Analysis
Four risk factors strongly associated with OD death
Risk Factor Odds Ratio
Pharmacies ≥ 4 3.7
Benzo / Opioid overlap ≥ 35 days 2.4
Max MED ≥ 100 2.3
Cash Payment ≥ 1 2.2
All 4 present 10.3
Secondary Analysis
Using Narx Scores as a predictor of overdose death
• Type specific use indicators for narcotics, sedatives and stimulants
• Range from 000-999
• As the score increases, so does the presence of:
• Providers
• Pharmacies
• MME
• Overlaps
Narx Scores in vivo
Narx Scores in vivo
Narx Scores In Workflow
Narx Scores
Narx Score Distribution
Scores
< 200 200-499 500-650 >650
Narx Scores Summary
Narx Scores are
• Numerical representations of PDMP data that capture “use”.
• Information at a glance.
Narx Scores are not
• Rules (they are tools).
• Are not synonymous with abuse.
Secondary Analysis
Methodology – Narx Scores as a predictor of overdose risk
• 100:1 case / control study
• Determined the highest narcotic score in the year prior to death for each decedent and for 100 date-matched living controls
• Calculated Odds Ratios
Secondary Analysis
Results
Narcotic Score Odds Ratio
0-199 1
200-299 6.4
300-399 7.4
400-499 10.2
500-599 15.5
600-699 23.3
700-799 29.8
800-899 37.7
900-999 63.4
Primary and Secondary Analysis Summary
Using traditional techniques that include both red flags and a composite use indicator, we were able to determine significant associations with unintentional overdose death
• Initial analysis identified 4 red flags strongly associated with overdose death risk
• Secondary analysis strongly associated Narx Scores with overdose death risk
Machine Learning ApproachData Overview
Used the same data from the secondary analysis
• 1:100 case to controls
• Artificial resultant 1% incidence of disease
Applied machine learning and other predictive techniques to develop a 3-digit score similar to Narx Scores, termed an Overdose Risk Score
• Range from 000-999
• Risk doubles approximately every 100 points
• Similar distribution to Narx Scores
Advanced Data Analysis – General Method
Drug Dispensation History and Trends
Types of Narcotics/Sedatives
Prescriber/Pharmacy Visit History and Acceleration
Narcotics to Morphine Equivalencies
Data Inputs
Literature Defined Red Flags
High Risk Behavioral Patterns
Inputs processed through predictive models to determine
the composite risk
Machine Learning Models Optimized
by Simulated Annealing
Decision Engine
Overdose Risk Score
(000 to 999)
Machine Learning ApproachVariable Derivation
Variable Determination• Hypothesize a variable and the expected effect• Develop variable for case and controls• Determine independent predictive ability
More than 70 variables were evaluated using this approach
Examples• Amount of narcotics (in MME) used in the prior 365 days• Amount of sedatives (in MME) used in the prior 60 days
From the 70 variables, approximately one dozen chosen for final model• Some that are used for Narx Scores• Some that are used for Red Flags• Some new variables that look at change over time
Model Validation During Development
• For each decedent and matched control 4 random dates were chosen in the one year prior to the date of death for the decedent producing 4 separate modeling sets to use in model fitting and evaluation.
• Each Set was further split into a 75% training sample and 25% validation sample
Final Model Validation
• After model completion, we used death data from 2013 and 2015 to validate the final model and compare to the 2014 results
Machine Learning ApproachVariable Derivation
Machine Learning Approach Model Evaluation with the KS Statistic
• Kolmogorov-Smirnov (KS) Statistic measures the maximum difference between
the cumulative percentage of two populations (Non-Decedents vs Decedents) by
score.
• Standard metric used in statistics to evaluate models.
0%
100%
Non-Decedents
Decedents
KS
Low HighSCORE
CU
MU
LATI
VE
%
0%
5%
10%
15%
20%
25%
30%
35%
40%
0 to 50 51 to100
101 to150
151 to200
201 to250
251 to300
301 to350
351 to400
401 to450
451 to500
501 to549
550 to599
600 to649
650 to699
700 to749
750 to799
800 to849
850 to899
900 to949
950 to999
Perc
ent
of
Pop
ula
tio
n
Overdose Model Score
Model Scores for Decedents and Non-Decedents
Non-Decedents Decedents
Model Score Distribution for Decedents and Non-Decedents
DecedentsMean: 505Median: 50795th %tile: 83599th %tile: 938
Non-decedentsMean: 209Median: 20995th %tile: 56999th %tile: 730
Decedents have higher risk scores
Model KS Sample 1 KS Sample 2 KS Sample 3 KS Sample 4 Avg. KS
Overdose Risk Score 75%/25% train/test 47.32 48.80 46.23 47.87 47.56
Overdose Risk Score 100%/100% train/test 48.34 49.62 47.57 48.27 48.45
• KS was evaluated on all four test samples (25% holdout group)
• During final testing, models were fit/tested against the full sample
• Many commercial models have KS scores in the 35 to 50+ range
KS Score was 49.62
Cu
mu
lati
ve P
erce
nt
of
Pop
ula
tio
n
KS Plot from Sample 2
Machine Learning ApproachModel Evaluation with the KS Statistic
Machine Learning ApproachSome Key Model Variables
The chance of overdose is 48 times higher when visiting 7+ pharmacies compared to no pharmacies
Group visiting 7+ pharmacies has 9% decedents
(compared to average of 1%)
Advanced Data Analysis
The chance of overdose is 44 times higher when visiting 11+ providers compared to no providers (in the last 2 years)
Group visiting 11+ providers has 9% decedents
(compared to average of 1%)
0%
1%
2%
3%
4%
5%
6%
7%
0 to 483 mg 484 to 966 mg 967 to 1449 mg 1450 to 1932 mg 1933 mg +
Perc
ent
Dec
eden
ts
mg of Sedative Prescribed in the Past Year
Some Key Model Variables
Group having 1933+ mg of sedatives in the last year has 6% decedents (compared to average of 1%)
Machine Learning ApproachFinal Score Construction
• Core model output is converted to a scaled score from 0 to 1000
• Low values = low risk
• High values = high risk
• Risk of overdose death doubles approximately every 100 points
0%
5%
10%
15%
20%
25%
0-99 100-199 200-299 300-399 400-499 500-599 600-699 700-799 800-899 900-999
Perc
ent
Dec
eden
ts
Overdose Risk Score
Predictive Power of Overdose Risk Score
Overdose risk doubles every 100
points (approx)
Individuals in the last bin are 329 times more likely to die of overdose than
individuals who score 0-199
Comparison of Odds Ratio for Drug Overdose Death
329 times more likely to die due to
drug overdose than people with score < 200
OD RiskPERCENT OF DECEDENTS
(1:100 Ratio Overall)ODDS RATIO
0-199 0.1% 1
200-299 0.7% 10
300-399 0.9% 12
400-499 1.9% 25
500-599 3.3% 44
600-699 6.3% 85
700-799 10.0% 141
800-999 13.2% 194
900-999 20.5% 329
0
50
100
150
200
250
300
350
Od
ds
Rat
io
Score
Overdose Risk Model Score strongly associated with risk
of overdose death
Machine Learning ApproachValidation of Overdose Risk Model Scores
Testing on years not used in the modeling process
shows stability of the model
NarxCare Example
Future Directions
1. Include More Data Types• Demographics
• Non-fatal overdose
• Criminal justice (CJ) data where appropriate
• Claims data
• CCD
• Mandatory reporting (NAS)
2. Include More Data• Invite more states
Future Directions
3. Study more outcomes• Non-fatal overdose
• Misuse and substance use disorders
• Arrest, Injury, etc.
Goal is to eventually detect risk at the earliest possible point to provide an optimal intervention point.
Questions?
Chad Garner, MS• Email: [email protected]
Jim Huizenga, MD• Email: [email protected]
David Speights, PhD• Email: [email protected]
