hec 2016 panel: putting user-generated data in action: improving interpretability for consumer...
TRANSCRIPT
AN INTERNATIONAL HELLO
Brazil - Opa
Chinese – nin hao Dutch – Hallo, Goededag
French – Bonjour German - Guten Tag Hawaiian - Aloha Indonesian -Selamat
Japan – konnichiwa Korean – annyeonghaseyo
Norwegian - Goddag Portugese –’Ola� Spanish - ¡Hola!
Swedish - Hej / Hallå�
Thailand - sà-wàt-dee Russian - Allo Turkey - Alo, Efendim
Italian – Ciao Israel-Shalom
Africa – Hallo
Polish – HALO/SLUCHAM
Arabic – As salam ‘alakum
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
Pa#ent generated data – The transi#on from “more” to “be6er”
HEC 2016 Workshop
WS 884 Pu(ng User-‐Generated Data in Ac8on: Improving Interpretability for Clinical and Consumer Informa8cs
Aug 30 16:30 -‐ 18:00
Panelists: Thomas WETTER, Ying-‐Kuen CHEUNG, Sanjoy DEY , XinXin ZHU, Bian YANG
Moderator: Pei-‐Yun Sabrina Hsueh
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics HEC/MIE 2016 Workshop: PuJng User-‐Generated Data in Ac#on: Improving Interpretability for Clinical and Consumer Informa#cs Katie Zhu, PhD MD
(IBM TJ Watson Research, USA)
Sanjoy Dey, PhD
(IBM TJ Watson Research, USA)
Ken Cheung, PhD
(Columbia University, USA)
Bian Yang
(Norwegian University of Science of Technology, Norway)
Thomas Wetter, PhD (Panel Discussant)
(University of Heidelberg, Germany
University of Washington, USA)
Pei-Yun Sabrina Hsueh, PhD (Moderator)
(IBM T.J. Watson Research, USA)
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
Agenda • 16:30-‐16:40 Opening Remark by Dr. Sabrina Hsueh
• EMERGING HEALTHCARE LANDSCAPE SHIFT WITH PATIENT-‐GENERATED DATA
• 16:40-‐17:20 Presenta#ons – Dr. Xinxin Zhu: So we got sensor data, now what? – Dr. Sanjoy Dey: Enhancing interpretability of computa#onal model
– Dr. Ken Cheung: SMART-‐AR to evaluate health apps for outcome op#miza#on
– Dr. Bian Yang: The need for addressing privacy issues with be6er interpretable rules • 17:20-‐18:00 Discussant summary presenta#on & Panel
discussion/audience Q&A – Dr. Thomas We6er: Pa#ent generated data – The transi#on from “more” to “be6er”
– Panel discussion (moderated by Dr. Sabrina Hsueh)
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
Summary on Workshop Theme (1)
• (1) Iden#fy immediate ac#on items to start ini#a#ng proposal for enabling evidence-‐based conversa#on with pa#ents/physicians/providers in the loop
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
Summary on Workshop Theme (1)
• 2. Implica#ons and lessons learned from the case studies -‐-‐ especially the gaps you perceived as barriers of entry
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
Summary on Workshop Theme (2)
• 3. Requirements for successful redesign of healthcare systems to accommodate pa#ent-‐generated informa#on (with a sub-‐goal of iden#fying the areas where such informa#on can make most impacts).
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
Ques#ons • 1. What is the state-‐of-‐the-‐art? • 2. What are the benefits of improving interpretability in PGHD
in ac#on? • 3. What the key dimension of interpretability of PGHD? What
are the barriers? Technical/social? • 4. What is our defini#on of interpretability? What are the
likely measures? • 5. What is the opportunity area going forward? • 6. What are the likely ac#on items to be suggested to the
community to further the discussion about improving interpretability for PGHD? – In the field of consumer health informa#cs or beyond?
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
INTRODUCTION
EMERGING HEALTHCARE LANDSCAPE SHIFT WITH PATIENT-‐GENERATED DATA
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
Pei-Yun (Sabrina) Hsueh, PhD
Wellness Analy8cs Lead Global Technology Outlook Healthcare Topic co-‐Lead Healthcare Informa8cs PIC co-‐Chair Computa8onal Behavioral and Decision Science Group Health Informa8cs Research Dept. IBM T. J. Watson Research Center • Research focus: Pa8ent-‐genera8on info from wearables and biosensor
devices/implants, Personaliza8on analy8cs, Pa8ent engagement & Adherence risk mi8ga8on, Interpretable machine learning
Opening Remark
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
Source: Based on McGinnis et al, The Case for More Active Policy Attention to Health Promotion, Health Affairs, 2002.
Health Determinants Mismatches Today’s Spending “We need to invest in addressing all determinants of health…”
BIG DATA Clinical + behavior
driven Wellness Management
It’s Big Data! It is also not just Big Data!
SOURCE: Barbara J. Sowada, A Call to Be Whole: The Fundamentals of Health Care Reform, July 30, 2003, Praeger.
IBM Watson // ©2015 IBM Corporation
NOISY, LARGE VOLUME, UNCONTROLLED
Need minimum description & quality/validity study
Solutions Population Health
Management
Condition Specific Care
Health and Wellness
Social Programs
Discovery Solutions
Real World Evidence
Ecosystem Population Health
Management
Condition Specific Care
Health and Wellness
Social Programs
Discovery Solutions
Real World Evidence
Individual
Social Programs
Education
Governments
Home Health Agencies
Practitioners
Hospitals
Therapists
Health Plans
Family
Public Health
Medical Devicesand Diagnostics
Bio-Pharma
Employers
Payers
DataInsight
IBM Watson // ©2015 IBM Corporation
To tap into the potential of DTR in open deployment, accessing a vast amount of untapped data could have a great impact
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
14
PGHD: Beyond Capturing Social/Behavioral Determinants from EHR
Institute of Medicine report (2016)
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
15
• R.W. White, R. Harpaz, N.H. Shah, W. DuMouchel, and E. Horvitz. Toward Enhanced Pharmacovigilance using Patient-Generated Data on the Internet, Nature CPT, April 2014.
Success Story: PGHD for Pharmacovigilance
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
Success Story: PGHD for Personalized Communication
Palmquist, A.E.L., Koehly, L.M., Peterson, S.K. et al. J Genet Counsel (2010) 19: 473. doi:10.1007/s10897-010-9299-8
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
Success Story: PGHD for Diagnosis
Identify the onset and progression of disease states e.g., depression, Parkinson’s, PTSD
Assist with decision making in ER (e.g., FITBIT CHARGE HR)
Source: 1. http://www.androidauthority.com/fibit-charge-hr-save-patient-685205/ 2. M. Sung, C. Marci, and A.S. Pentland, Objective Physiological and Behavioral Measures for Identifying and Tracking Depression State in Clinically Depressed Patients, MIT Technical Report, 595 (2005): 1-20. 3. S. Arora, V. Venkataraman, S. Donohue, K.M. Biglan, E.R. Dorsey, M.A. Little, High accuracy discrimination of Parkinson’s disease participants from healthy controls using smartphones, IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (2014), 3641–3644.
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
Success Story: PGHD for Care Coordination
IBM Taiwan Collaboratory
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
19
Promoting patient activation for behavioral change (Dietary intake: Burke et al., 05; Physical activity: Prestwich et al., 09; Michie et al., 09) Preventing lifestyle-related chronic diseases, e.g., Type II Diabetes
Helmrich et al, 1991;Bailey, 2001; Scottish Intercollegiate Guidelines Network, 2001; Finland National Type II Diabetes Prevention Programme, 2007; American Diabetes Prevention Program, 2008).
Increase awareness to self-monitoring (Prestwich et al., 09; Burke et al., 05)
Triggering reminders to care plans (Consolvo et al. 09; Hurling et al., 07) Personalizing communication messages and education materials (Thaler and Sustein, ‘08)
Making Sense of PGHD for Individuals
Nudge: Improving Decisions About Health
PERSONAL INFORMATICS TOOLS (auto PGHD capturing + manual input)
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
20
The Failure of Scripps Trial
Patients who monitored their health were less likely to attribute health outcomes to chance than those who didn’t monitor their health
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
Where do we meet in the middle?
???
Unsustainable, ill-supported health consumers
Healthcare Triple aim
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
22
Reference Story: Kaiser Permanente – Improved Outcome and Reduced Co Individualized Guideline Improved Clinical Outcomes § Reduce 5-year CVD risk 2.4 times more than EHR+panel support tool alone (≈ 13% absolute risk reduction)
§ ≈ 6,000 myocardial infarctions (MIs) and strokes prevented annually if applied throughout KP (≈43% increase over JNC7 guideline for the same cost)
Individualized Guideline Reduced Operational Costs § ≈ $7,000 cost savings per MI and stroke § ≈ $420M annual net savings if applied throughout KP
Source: Eddy, et al. (2011). Individualized Guidelines: The Potential for Increasing Quality and Reducing Costs. Annals of Internal Medicine, vol. 154, no. 9, p.627-634. http://www.annals.org/content/154/9/627.abstract
22
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
23
Kaiser Permanente – Improved Patient Motivation and Adherence, Increased Clinician Confidence
(Respondents were)“…more likely to report that they have been asked to change their medication, diet and exercise habits. ”
—Patient Survey
“…helped the doctor to motivate them and helped them participate in their treatment choices, i.e., making lifestyle changes and understanding the rationale for their suggested interventions.”
— Patient Focus Group
“All doctors agreed that it helps them to make the best clinical decisions for their patients.”
— Clinician Survey
23
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
Adding High Touch by Lay Care Guides
• Parallel-group randomized trial (2010-2012). – 6 primary care clinics in Minnesota. – Adults with hypertension, diabetes, or heart failure. – Assigned in a 2:1 ratio to a care guide or usual care.
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics HEC/MIE 2016 Workshop: PuJng User-‐Generated Data in Ac#on: Improving Interpretability for Clinical and Consumer Informa#cs Katie Zhu, PhD MD
(IBM TJ Watson Research, USA)
Sanjoy Dey, PhD
(IBM TJ Watson Research, USA)
Ken Cheung, PhD
(Columbia University, USA)
Bian Yang
(Norwegian University of Science of Technology, Norway)
Thomas Wetter, PhD (Panel Discussant)
(University of Heidelberg, Germany
University of Washington, USA)
Pei-Yun Sabrina Hsueh, PhD (Moderator)
(IBM T.J. Watson Research, USA)
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
Agenda • 16:30-‐16:40 Opening Remark by Dr. Sabrina Hsueh
• EMERGING HEALTHCARE LANDSCAPE SHIFT WITH PATIENT-‐GENERATED DATA
• 16:40-‐17:20 Presenta#ons – Dr. Xinxin Zhu: So we got sensor data, now what? – Dr. Sanjoy Dey: Enhancing interpretability of computa#onal model
– Dr. Ken Cheung: SMART-‐AR to evaluate health apps for outcome op#miza#on
– Dr. Bian Yang: The need for addressing privacy issues with be6er interpretable rules • 17:20-‐18:00 Discussant summary presenta#on & Panel
discussion/audience Q&A – Dr. Thomas We6er: Pa#ent generated data – The transi#on from “more” to “be6er”
– Panel discussion (moderated by Dr. Sabrina Hsueh)
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
SO WE GOT SENSOR DATA, NOW WHAT?
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
• MD (Anesthesiologist) from China Medical University
• PhD in Biomedical Informa#cs from Columbia University
• Past Experience – Chief Medical Informa#on Officer at Kforce Government Solu#ons, U.S.A.
– Associate Medical Director, Pfizer Health Solu#ons, U.S.A.
– Senior Manager, Pfizer Health Solu#ons, U.S.A.
– Clinical Program Manager, Philips North America Research Center, U.S.A.
– Healthcare Informa#cs Subject Ma6er Expert, Veterans Affairs Medical Center, U.S.A.
Xinxin (Katie) Zhu • Telehealth lead at IBM
Watson • External Advisory Board
member to Columbia Univ. Center of Advanced Technology
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
So we got sensor data, now what?
• What sensor data could help with care? • How to determine when to use what? • Are the sensor data reliable? • What is the context when data were collected? • How to interpret data in context? • Clinicians’ concerns
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
What sensor data could help with care? Use case: stress management
Subjec#ve Stressors
Psychological Response
Physiological Response
Stress Hormones
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
Many sensors are out there… Tinke
31
Approach • Plug into a smartphone • Scan finger • Provide stress/relax index
Data Tracked • Heart rate variability • Respiration rate • Blood oxygen level
Tinke Website
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
Spire
32
Data Tracked • Breathing pattern • Steps
Approach • Consistent breaths à Calmness • Uneven breaths à Tension • Fast and consistent breaths à Focus • Guided meditation
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
Pip
33
Data Tracked • Skin conductance (EDA)
Approach • Hold device between the
thumb and index fingers • Stress level via audio/
visual feedback
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
Many sensors are out there
Brain Wave (EEG sensor)
Skin Conductance (EDA sensor)
Blood Volume Pulse (PPG sensor)
Skin Temperature (Infrared Thermophile)
Heart Rate (PPG sensor)
Heart Rate Variability (ECG sensor)
Respiration Rate/Volume (RIP sensor)
RR Interval Distribution (ECG sensor)
Image Source: Neurosky, Empatica, Hexoskin
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
Hexoskin V.S. BioSens Holter ECG Valida#on
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
Brain Wave
36
Relaxed Reading a paper with a time limit
Delta - Adult slow wave sleep Theta - Drowsiness, idling, inhibition Alpha - Relaxed, reflecting Beta - Alert, busy, anxious, thinking Gamma - Short term memory usage Mu - Rest state motor neuron activity
- Produced by electrical pulses from neuron communication
- Frequency bands associated with different behaviors and emotions
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
How can people make sense of these?
37
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
Clinicians’ concerns
Information overload Unreliable data à false alarms
Clinical workflow Context, context, context!
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
Summary on Workshop Theme (1)
• (1) Iden#fy immediate ac#on items to start ini#a#ng proposal for enabling evidence-‐based conversa#on with pa#ents/physicians/providers in the loop
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
Summary on Workshop Theme (2)
• 2. Implica#ons and lessons learned from the case studies -‐-‐ especially the gaps you perceived as barriers of entry
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
Summary on Workshop Theme (3)
• 3. Requirements for successful redesign of healthcare systems to accommodate pa#ent-‐generated informa#on (with a sub-‐goal of iden#fying the areas where such informa#on can make most impacts).
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
Ques#ons (preliminary) • 1. What is the state-‐of-‐the-‐art? • 2. What are the benefits of improving interpretability in PGHD
in ac#on? • 3. What the key dimension of interpretability of PGHD? What
are the barriers? Technical/social? • 4. What is our defini#on of interpretability? What are the
likely measures? • 5. What is the opportunity area going forward? • 6. What are the likely ac#on items to be suggested to the
community to further the discussion about improving interpretability for PGHD? – In the field of consumer health informa#cs or beyond?
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
ENHANCE INTERPRETABILITY WITH PRIOR KNOWLEDGE
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
Sanjoy Dey PhD.
Postdoctoral Research Scientist, Center of Computational Health, IBM T. J. Watson Research Center, Yorktown Heights, NY 10598
Sanjoy Dey’s research interests lie in the areas of health care informa#cs, data mining
and machine learning, especially in building interpretable models by integra#ng mul#ple healthcare datasets. . In par#cular, Sanjoy is interested in building models
which aim to incorporate domain knowledge at mul#ple stages of model development (e.g., feature selec#on, cohort selec#on and study design) so that these models can infer knowledge that are complementary to the already known clinical prac#ces and
guidelines. Prior to this posi#on, he earned his Ph. D. from the department of computer science at university of Minnesota.
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
Improving Interpretability of Patients Generated Data
45
Dis
ease
H
ealth
y
Dataset 1 Dataset 2
Class label
Relation across the datasets
Analysing the obtained results from Complex Models • Interpret the model parameters so that they
can be used to infer meaningful knowledge • Visualize the obtained informa#on from a
model in a meaningful way
Taking prior knowledge into account
• Many #mes, medical knowledge are available containing useful rela#onships among clinical events
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
Dataset
Interpre#ng Complex Computa#onal Models
• Complex model parameters can be converted to metrics that are easily understandable by domain experts
– Logis#c Regression – LASSO with regulariza#on to perform
simultaneous variable selec#on
• Logis#c loss func#on can be used as Log Odds, which can be converted to Odds Ra#o -‐ where β0 the log odds for smoking for men
• Probabili#es of an event can be viewed as clinical uncertainty
Class Level
Liu et al. 2011, Jeiping Ye et al., 2012
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
Visualiza#on of the obtained model
Decision Boundaries of Logis#c Regression
Rule based representa#on of Decision Tree and Cart based Models
Graphical Models for Disease Models
Work Environment Gene
Disease
Symptoms
Westra et al. 2011, Manzi et al. 2013
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
Integra#ng Prior Knowledge with Mul#-‐source EHR data for Enhancing Interpretability
Diagnosis Codes (ICD-‐9)
Admission Assessment
Survey
Discharge Assessment
Survey
Home Healthcare
Dey et al. AMIA 13, Dey et. al., SDM 14, Westra et al. 11
Demographic, behavioral, pathological, psycho-social factors, outcome variables.
Problem Formulation:
48
260,000 patients
Data source: CMS OASIS dataset
Outcome prediction
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
Enhancing Interpretability of Patterns
49
Predictive Power
Inte
rpre
tabi
lity
• Interpretability (Relevance) and predic#on power are different goals
• Prior rela#onships present in the data can be incorporated into model
ICD-‐9 Group 1 ICD-‐9 Group 2 250.6: Neurological manifesta#on
401.1: Benign hypertension
290: Demen#a 838: Disloca#on of foot 331: Alzheimer’s disease 692.71: Sunburn 331.9: Cerebral degenera#ons
V58.42: Hip joint replacement
Neural disorders No common underlying disease
Interpretable Predic8ve
• Which group of pa#ents are likely to improve ambula#on?
• Are those factors clinically interpretable and make a homogeneous group?
G1
G2
Ideal
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
Proposed Approach Key Steps: • Integrate both survey data from EHR and
ICD-‐9 diagnoses codes to predict the improvement of urinary incon#nence
• Use clinical prior knowledge such as Clinical Classifica#on Sotware (CCS) into account to increase the interpretability
• Develop a sta#s#cal technique called Sparse Hierarchical Canonical Correla#on Analysis (SHCCA) to address these challenges
50
X Y
Algorithm: • Take the hierarchy of the CCS tree into account to
define a similarity matrix called H among the ICD-‐9 codes
• Trade-‐off between the data-‐driven and prior knowledge driven similarity of ICD-‐9 codes using λh
• Converted into convex formula#ons
• Solve the final equa#on based on gradient descent formula#on
Prior Knowledge
λh trades off between domain-driven and data-driven knowledge
Dey et. al., SDM 14
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
Sparse Hierarchical CCA (SHCCA) Parameter Selec8on: • Op#mize the parameters using cross-‐valida#on such that it
op#mizes the correla#on on valida#on data Evalua8on: Predic8on power: how well the selected group of ICD-‐9 codes can predict the improvement of outcome Interpretability: – I-‐score based on the co-‐occurrences of the ICD-‐9 terms
belonging to a group C in PubMed ar#cles – Domain knowledge by physicians and nurses
51
ti is the set of articles found with ICD-9 code i
I-‐score(C)=∑𝑖ϵ𝐶↑▒∑𝑗ϵ𝐶↑▒| 𝑡↓𝑖 ⋂ 𝑡↓𝑗 |/| 𝑡↓𝑖 ∪ 𝑡↓𝑗 |
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
Results • SHCCA has similar performances as
the baseline methods, but with fewer components
• It enhances the interpretability significantly
Predictive power of SHCCA
52
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
Components from SHCCA
Survey data 1 ICD-‐9 codes 1 Survey data 2 ICD-‐9 codes 2
Age, Prior Memory Loss,
Poor Speech, Poor Cogni8ve
Func8on, High Confusion,
Memory Deficiency, Frequent Behavioral
Problem
Demen8as, Persistent
mental disorders, Alzheimer's disease, Cerebral
degenera8ons
Surgical Wound, Fully
granulated Surgical Wound
Acercare for healing fracture of hip,
Knee joint replacement, Hip joint replacement,
Acercare following surgery of the musculoskeletal system, Acercare following joint
replacement, Acercare following surgery for
neoplasm, Acercare following surgery of
the circulatory system 53
Component relevant to Mental health
Component relevant to surgical treatment
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
Summary & Limita#ons • Summary
– Predic#on power and interpretability are two different goals, which are oten hard to achieve by computa#on models simultaneously
– Predic#ve models can be post-‐processed and visualized to make them more
interpretable
– Leveraging clinical prior knowledge such as Clinical Classifica#on Sotware (CCS) into account can increase the interpretability substan#ally
• Limita#ons – The defini#on of interpretability is oten subjec#ve and oten requires domain
exper#se
– Prior knowledge about a par#cular problem is not oten readily available in many clinical applica#ons
– Use of prior knowledge into the model op#miza#on is oten not straighvorward
54
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
SMART-‐AR to evaluate health apps for outcome op#miza#on
Ken Cheung Columbia University
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
Ying Kuen (Ken) Cheung
• PhD in Sta#s#cs (U Wisconsin, Madison WI, USA)
• Professor of Biosta#s#cs, Columbia University, New York NY, USA
• General interest: Transla#onal research in all phases • Specific areas
• Dose and treatment selec#on in adap#ve clinical trials • Op#mal behavioral interven#on for secondary stroke preven#on • Analysis of high-‐dimensional physical ac#vity data • N-‐of-‐1 trial designs • Evalua#on and dissemina#on of mobile technologies for mental
health
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
Objec#ve & Reinforcement Learning
• Data sequence: (X, A1, U1, A2, U2, …, AK, Y) – X = Individual characteris#cs – At = Apps downloaded (Ac#on) at #me t – Ut = response and use pa6ern between At and At+1
– Y = Final outcome (depression reduc#on)
• Objec#ve: Iden#fy the sequence At based on X and Ut so as to maximize Y (on average)
• Reinforcement learning: Q-‐learning, OWL, etc.
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
SMART Design
• SMART (Sequen#al Mul#ple Assignment Randomized Trial)
App 1
App 2
Ac#ve use
Ac#ve use
Non-‐use
Non-‐use
App 1 + App 3
App 2
App 2 + App 3
App 2 + reminder
App 1
App 1 + reminder
App 2
App 3
Depression reduc8on at 6 months, Y
Enrichment based on intermediate use paeern, U
P = 2/3
P = 1/3
P = 0.3
P = 0.7
P = 0.6
P = 0.4
P = 0.6
P = 0.4
P = 0.3
P = 0.7
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
SMART-‐AR Design (Cheung et al, 2015 Biometrics)
• SMART Design ü Allows learning ✗ No feedback to system ✗ Curse of dimensionality: many apps
in prac#ce
• SMART-‐AR • AR = Adap#ve randomiza#on • Assign more users to more
promising branches • Curse of dimensionality: Sot
elimina#on of poor performing apps à Improve signal-‐noise ra8o, hence interpretability of the recommender
0 20 40 60 80 100
910
1112
1314
15
Enrollment number
BD
I red
uctio
n
o o o o o o o o o o o o
+ + + + + + + + + + + +
p p p p p p p p p p p p
m m m m m m m m m m m m
Scenario 1
CODIACSBalanced
Application in conventional depression program
Cheung et al, 2015 Biometrics
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
Some Simula#on Illustra#on Non-‐adap8ve SMART SMART-‐AR
Balanced randomiza#on
CODIACS randomiza#on
Scenario 1
Probability of iden#fying the op#mal sequence
0.91 0.94 0.95
Expected adjusted value* 0.98 0.99 0.99
Variance of adjusted value 3.1 2.3 1.3
Scenario 3
Probability of iden#fying the op#mal sequence
0.53 0.51 0.51
Expected adjusted value* 0.95 0.95 0.96
Variance of adjusted value 8.5 11.0 7.4
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
Health informa#cs support issues
• SMART-‐AR requires real #me transmission between data site, apps cura#on site, compu#ng site – Large volume: Use data pre-‐processing
– Privacy & security • Health outcomes
– Valida#on of outcomes
0 10 20 30 40
Days since download first app
Ap
p ID
12
34
56
78
910
11
12
13
14
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
ADDRESSING PRIVACY/SECURITY CONCERN WITH INTERPRETABILITY
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
Bian’ intro goes here….
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
- February 4, 2015 - Hacker broke into the medical
insurance database - 80 million records stolen in
plaintext - Insurance company’s database
are not required to be encrypted by HIPAA
- administrator's credentials were compromised
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
Now … the hot term for 2016 – «ransomware»
- More than half of hospitals (in US) hit with ransomware in last 12 months
(HealthcareITNews, April 07, 2016)
- Good business model for the hackers - Low risk - Good cost-benefit efficiency - Easy to build "reputation" for
the service –
(https://www.theguardian.com/technology/2016/feb/17/los-angeles-hospital-hacked-ransom-bitcoin-hollywood-presbyterian-medical-center)
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
More A6ack vectors to Pa#ent Data Security
More attacking vectors opened due to … the shifts of healthcare patterns - now and future
• hospital -> home / cyber space (telemedicine, IoT, mobile technologies, care research)
• in-hospital treatment -> prevention (big data, health analytics, health electronics, e-drugs)
• doctor-centered -> patient-centered (telemedicine, big data, machine intelligence, cloud storage and computing)
• health care organizations -> associated business partners in liability (law and regulations, e.g., HIPAA -> HITECH (2009))
• Local service -> global service (service across the borders) • the “SafeHarbor” agreement • Facebook fined by the Belgian court
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
What does security mean for eHeath/mHealth’s future
- new breaches and "business models for hacking" would continue to come… (but take it easy)
- More liability to IT tech enablers and business associates (e.g, HIPAA ->HITECH)
- cloud / SDN makes "security as a service" that can be outsourced
- IT Tycoons (Microsoft, IBM, Google, etc.) could finally take it over (capable to take risk, more resources, global threat intelligence, etc.)
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
Future Solu#ons
- Data ownership re-definition - Generating incentives for industry to migrate from
data silos to data sharing - Patients’ awareness of their interests in their own
data - Patients’ convenience in accessing their own data - Legal support - Technology: security / privacy by design
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
DISCUSSANT SUMMARY PRESENTATION
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
Prof. Dr. Thomas Weeer • MSc / PhD in mathema8cs from Aachen Technical U,
Germany • PostDoc with IBM Scien#fic Center Heidelberg, Germany • Since 1997 Prof. of Medical Informa8cs, Heidelberg U
– Interna8onal assignments to Boca Raton (FL), Aus#n (TX), Salt Lake City (UT), Sea6le (WA)
– Affil. Faculty with Dept. BIME, U of Washington, Sea6le – Author of textbook
Consumer Health Informa#cs: New Services, Roles and Responsibili#es; Heidelberg (Springer) 2015 (eBook) resp. 2016 (Hardcover)
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
Pa#ent generated data – The transi#on from “more” to “be6er”
Thomas We6er
May be obsolete here with the title slide already
using this paraphrase of the workshop title
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
Growth is everywhere • More modali#es to collect and store data are offered • More communica#on media carry health info • More condi#ons suggest to be monitored • More ins#tu#ons consider usage • More consumers buy in
• Does this make sense? • How can we move towards meaningful ac#on? • How can we protect against unethical exploita#on?
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
Growth is everywhere • More modali#es to collect and store data are offered • More communica#on media carry health info • More condi#ons suggest to be monitored • More ins#tu#ons consider usage • More consumers buy in
• Does this make sense? • How can we move towards meaningful ac#on? • How can we protect against unethical exploita#on?
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
More data is more op#ons
.
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
More data is more op#ons
Period
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
But beware
Date are not the world; data map the world – truthfully? What is the ci#zen‘s contribu#on to the mapping? • Carrier of implanted sensors • Operator of a6ached and mobile sensors • Witness of health signs • Interpreter of health signs • Self therapist • Health plan contractor
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
More data is more tempta#ons
Ci#zens • Trust more than warranted • Shit focus from senses to data Clinicians • Shit focus from senses to data Researchers, public health • Urge to find something Big business • More big business
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
More data is more tempta#ons: Ci#zens
Percep#on of the presumably unfailable objec#ve givens as proxy for truth • Mental fixa#on on data • Unwarranted trust as decision aid • Adverse reac#on to contradictary data • Overreac#on upon alarming data
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
More data is more tempta#ons: Researchers, Public health Percep#on that regarding the massive volume of data there cannot be no effects • Do the big data mechanics • Spot peculiari#es • Publish results Knowing that 5% of significant studies are not substan#ated through an effect
Curb: Complexity reduction –
Sanjoy Rey, Ken CheungRisk: Blindfolded actionism
Curb: Plausibility, context – Katie ZhuRisk: Funding agency expectations Curb: Research ethics
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
More data is more expecta#ons
If scien#st dispose of more data their methods are challenged: • Profound interpreta#on and predic#on – Sanjoy Dey • Parsimony, wise selec#on – Sanjoy Dey • Secure storage/communica#on – Bian Yang • Insight – Ken Cheung, Sabrina Hsueh
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
More is more expecta#ons
If ci#zens volunteer their data, they expect services: • Serious PGHD into PHR into EHR integra#on • No data leakage • Explana#ons of the unexplainable • Emergency rescue in response to alarming data
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
More data can be the hays#ck
• where we don‘t find the needle • while being distracted by – hay • but someone needs the needle – now
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
Claude Shannon 1948 1)
„Informa#on is that which reduces uncertainty“ Which the needle in the hays#ck does not do
1) A mathematical theory of communication Bell Systems Technical Journal
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
Bring forth the signal from the noise
• Concentrate trials on treatments with emerging posi#v prognosis (Ken Cheung)
• Select data with high interpreta#ve or predic#ve power (Sanjoy Dey)
• Regard context to detect noise (Ka#e Zhu)
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
Are we achieving quality that sa#sfies doctors?
• Not a ma6er of taste • Doctors‘ code of conduct regulates that when trea#ng diagnosed pa#ents he – assumes responsibility for correct recordings of devices he hands to the pa#ents
– has to waive liability for data generated through other pa#ent solicited devices
while, when coaching for healthy lifestyle – anything goes
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
Are we achieving quality that sa#sfies doctors?
• Under a treatment contract a doctor is held responsible for medical errors.
• Morally, he cannot be held responsible for decisions based on false/faked data from outside his control
• Pa#ents want their data used • They cannot guarantee correct data
• A classical gridlock 1)
1) In NY/NY en.wikipedia.org/wiki/Gridlock
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
Who can do what
to solve the gridlock?
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
Are we achieving full transparency? Do we want it?
Imagine that a certain set of sensor data is so characteris#c of you that you need not register, just
deliver a sample and they know who you are.
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
Are we suppor#ng personalized medicine?
• If the wealth of our data is so large that we can iden#fy data-‐twins – A treatment for the second twin should work if it did for the first
– The end of clinical trials
Putting User Generated Data in Action: Improving Interpretability for Clinical and Consumer Informatics
Thank You
Merci Grazie
Gracias
Obrigado Danke
Japanese
English
French
Russian
German
Italian
Spanish
Brazilian Portuguese Arabic
Traditional Chinese
Simplified Chinese
Hindi Tamil
Thai Korean
Hebrew