february 28, 2006: i. sim decision support systems medical informatics – epi 206 ida sim, md, phd...

February 28, 2006: I. Sim Decision Support SystemsMedical Informatics – Epi 206

Ida Sim, MD, PhD

February 28, 2006

Division of General Internal Medicine, and Program in Biological and Medical Informatics

UCSF

Electronic Health Records for Clinical Research (cont.)

Copyright Ida Sim, 2006. All federal and state rights reserved for all original material presented in this course through any medium, including lecture or print.


EHR for Research Summary

• An EHR is not automatically going to help clinical research– if all unstructured free text, won’t help much at all

• the more structured it is (ie more defined fields), the better

– if just coded sporadically in ICD-9• problem with gamed codes• poor coverage of many clinical concepts

– if coded in SNOMED• some clinical concepts still not well covered

• EHR better than chart review; can we do even better?


Capturing Clinical Data: Now

Recode Patient Data

Financial Database

Collect Patient Data

Clinical Repository


Research Database

ICD CDEFree text


Capturing Clinical Data: Wish

Recode Patient Data

Financial Database


Clinical Repository


Research Database

SNOMED-CT


MICU

FinanceResearch

QA

Clinical Repository

Internet

ADT Chem EHR XRay PMB Claims

• Integrated historical data common to entire enterprise

What is a Clinical Data Repository?


Types of Queries

• Clinical care• What was Mr. Smith’s last

potassium?• Does he have an old CXR

for comparison?• What antihypertensives

has he been on before?• What did the neurology

consult say about his epilepsy?

• Research• What % of diabetics with

AMI admissions were discharged on -blockers?

• What was the average Medicine length of stay in 2004 compared to 2000?

• What is the trend in use of head CTs in patients with migraine?

• Is admission creatinine independent predictor of bacteremia outcomes?


Types of Data Repositories

• A data repository (aka data warehouse) is just a collection of data from other databases– is itself just a database

• Two somewhat distinct types– clinical data repository

• collects data from day-to-day clinical care, admin data, etc.• for quality improvement, outcomes research, business decision

making…– research data repository

• collects data from multiple research projects• may also collect data from day-to-day clinical care, admin data, etc.


Data Repositories: Hope• Touted for

– business decision making– health care quality improvement– outcomes research– geno-pheno correlations for translational research

• UCSF planned “Research Data Infrastructure and Services” – goal: a single clinical and research data repository

• care data from STOR, radiology, UCare etc. • research data from all UCSF research projects

– to enable correlation of clinical, genomic, imaging, etc data across data sets

– one query across all systems -- great!


• Garbage in = garbage out• Incompatible data in = garbage out

– query in one system would return meaningful results from another (interoperable)

– requires• interoperable data schemas

– type (e.g., relational)– data modeling (i.e., column names)

• common naming of data items– eg., “PNA” vs. “pneumonia”

Or Hype?


UCSF “RDIS” Example

• Standard clinical vocabulary? data representation model?• Are queries mostly within or across projects? ongoing or completed

projects or both?• Need administrative data (e.g., insurance)?

(in SNOMED-CT) xrays/CT/MRImicroarray data(in MAGE-OM) (in DICOM)

•Breast CA (not DCIS)•Menopause

•Osteoporosis (Heel US)•Menopause

Project 1

DB 1

Project 2

DB 2

Project 3

DB 3

Project 4

DB 4

•Osteoporosis (DXA)•Menopause

•Breast CA (DCIS ok)•Alzheimers (path)

RDIS

Data mining/Display ToolsRadiologySTOR


Repurposing for QI and Research?

Financial Database


Clinical Repository

Research Database

Clinical Quality Improvement

?


Data Repository Summary

• Enterprise viewpoint more appropriate for research than patient viewpoint of EHR

• Integrates data from multiple sources– need standardization of codes, definitions, and data

schemas• Querying and processing occurs “offline”

– little impact on real-time clinical care• Repository must be designed for anticipated uses

– can single repository serve clinical and research needs?

Viewpoint Time Queries

EHR Patient Real-Time ClinicalData Repository Enterprise Historical Ad Hoc


• EHR does not always = easier clinical research• Structure and coding is critical

– structure: e.g., relational schema, designed to support intended queries

– standard formats needed for genomic, imaging, etc. data– coding: standardized, coded data trumps free text

• especially important for research• but most controlled vocabularies have insufficient clinical coverage

– controlled vocabularies (e.g., SNOMED-CT) are hard to design and hard to use

• Clinical/Research data repositories must be designed “correctly” with high-quality, cross-compatible data

Take-Home Points


Decision Support Systems

Ida Sim, MD, PhD

February 28, 2006

Division of General Internal Medicine, and the Program in Biological and Medical Informatics

UCSF

Copyright Ida Sim, 2006. All federal and state rights reserved for all original material presented in this course through any medium, including lecture or print.


Outline

• Decision support systems– background, definition– clinical versus research decision support

• How decision support systems “think”• Effectiveness

– improving quality– reducing errors

• Implications


Decision Support for Quality

• Endemic quality problems– “Health care today harms too frequently and routinely

fails to deliver its potential benefits… Between the health care we have and the care we could have lies not just a gap, but a chasm.” (IOM, 2001)

• Evidence-based practice is means to quality– practice based on currently best available evidence from

clinical research– hardest way to practice, logistically impossible

• > 1,000 guidelines in National Guideline Clearinghouse• > 4,600 journals indexed in Medline

– over 10,000 RCTs per year– over 2700 systematic reviews per year


Informatics to the Rescue

• Point-of-care decision support– Medline on every desktop

• and “beyond Medline”…

– reminders (e.g., preventive care)– guideline-based recommendations

• e.g., when to prescribe antibiotics for sore throat

• Diagnostic support– how not to miss anthrax


Clinical Decision Support

• Clinical decision support system (CDSS)– software that is designed to be a direct aid to clinical

decision-making – in which the characteristics of an individual patient are

matched to a computerized clinical knowledge base– and patient-specific assessments or recommendations

are then presented to the clinician and/or the patient for a decision (Sim et al, JAMIA, 2001)

• Examples of clinical decisions to be supported?


Major Target Tasks of CDSSs• Diagnostic support

– DxPlain, QMR• Drug dosing

– aminoglycoside, theophylline, warfarin• Preventive care

– reminders for vaccinations, mammograms• Disease management

– diabetes, hypertension, AIDS, asthma• Test ordering, drug prescription

– reducing daily CBCs in hospital, drug allergy checking• Utilization

– referral management, clinic followup


What Isn’t a CDSS

• Medline• UpToDate• Static guideline repositories

– www.guideline.gov (National Guideline Clearinghouse)

• Online laboratory data, test results, chart notes• Retrospective quality improvement reports

– how your vaccination rates compare to your colleagues’


Research Decision Support

• Research decision support system– software that is designed to be a direct aid to ???

decision-making – in which the characteristics of an individual ??? are

matched to a computerized ??? l knowledge base– and ??? -specific assessments or recommendations are

then presented to the ??? for a decision

• Examples of research decisions to be supported?– determining eligibility– what to do when (if WBC<2 then hold Drug)– reporting adverse events, etc.


Similarities/Differences

• customized to patient• identify applicable

guidelines, evidence• variable presentations and

contexts• wide clinical coverage• may include diagnostic

support• involves many team

members• one locale

• uniform treatment • identify applicable patients• narrower range of

presentations/contexts• narrower clinical coverage• more procedural, less

diagnostic support• smaller defined, more

uniform target staff• could be in multiple sites• more controlled

circumstances, regulatory overlay

Research Decision SupportClinical Decision Support


Outline




• Implications


Basic Decision Support Task

• Decision– action that consumes resources in the real world

• Decision support– given starting conditions and a defined set of action

choices, recommend or rank action choices for user

• Requires some “thinking” to recommend or rank– strictly deterministic thinking– thinking with fuzziness and probabilistic features

• in starting data or reasoning procedure• outcomes (e.g. prob. of adverse reaction)


Decision Support “Thinking”• Strictly deterministic

– first-order logic rule-based systems– adhoc rule-based systems (non-mathemetical reasoning about

probability)• e.g., if high WBC AND cough AND fever AND abn. CXR then

likelihood of pneumonia is 4 out of 5

• Probabilistic/fuzzy– bayesian networks

• formal probabilistic reasoning, extension of decision analysis

– neural networks– fuzzy logic, genetic algorithms, case-based reasoning, etc., or

hybrids of these


Rule-Based Decision Support (1)

• Forward chaining/reasoning (data-driven)– start with data, execute applicable rules, see if new

conclusions trigger other rules, and so on– example

• if HIGH-WBC and COUGH and FEVER and ABN-CXR => PNEUMONIA

• if PNEUMONIA => GIVE-ANTIBIOTICS• if GIVE-ANTIBIOTICS => CHECK-ALLERGIES• if PNEUMONIA and GIVE-ANTIBIOTICS and NOT

(ALLERGIC-DOXYCYCLINE) => GIVE-DOXYCYCLINE

– use if sparse data


Rule-Based Decision Support (2)

• Backward chaining/reasoning (goal-driven) – start with “goal rule,” determine whether goal rule is

true by evaluating the truth of each necessary premise– example

• patient with lots of findings and symptoms• is this SLE? => are 4 or more ACR criteria satisfied?

– malar rash?– discoid rash?– skin photosensitivity? etc

• if SLE => ...– use if lots of data


Problems with Rule-Based DSSs• Combinatorial explosion of rules

– need rule for each contingency• if MOD-WBC and COUGH and FEVER and ABN-CXR =>

PNEUMONIA

• Rules may be contradictory– if COUGH and ABN-CXR => INTERSTITIAL-LUNG-DZ

• Rules may be circular• Need knowledge engineering and clinical

expertise to build and maintain the knowledge base over time– need to keep rules up-to-date with latest evidence

• Could rules be centrally defined and then shared?


Medical Logic Modules (MLMs)

• help_amp_for_pneumonia - Ampicillin for Pneumonia

• maintenance:– title: Ampicillin for

Pneumonia;;– filename:

help_amp_for_pneumonia;; – version: 1.00;; – institution: LDS Hospital;; – author: Peter Haug, M.D.;

George Hripcsak, M.D.;; – specialist: ;; – date: 1991-05-28;;

• validation: testing;; • library:

– purpose: Recommend the use of ampicillin for pneumonia.;;

– explanation: If the patient has pneumonia, then suggest treatment with ampicillin unless there is a penicillin allergy.;;

• keywords: pneumonia; penicillin; ampicillin;;

• citations: 1. HELP Frame Manual, version 1.6. LDS

Hospital, August 1989, p.81.;;

• For sharing forward chaining rules • Expressed in Arden Syntax (an international ASTM standard)


Sharing of MLMs: No Success• Work of reuse often greater than building from

scratch– rules are often outdated: need to check evidence base– context is under-specified

• is pneumonia rule inpatient or outpatient? in HIV patients?

– can be wrong for local context• resistance patterns vary in different locales

– definitional problems• your “pneumonia” is not my “pneumonia”

– curly braces problem• if {K+} > 5.5 => alert MD• how to access the value of K+ automatically? requires interfacing

to lab system which differs from place to place


Rule-based DSSs for Research

• More promising than for clinical decision support– usually narrow context (e.g., single study)

• small, explicitly defined, stable, uncontroversial rule base

– definitional problems resolved as part of study design– clinical context standardized or taken into account

• studies on inpatient vs. outpatient pneumonia

– trained, dedicated staff sharing same objective• e.g., is clinical guideline for saving $ or saving lives?


Decision Support “Thinking”• Strictly deterministic

– rule-based systems– adhoc (non-mathemetical reasoning about probability)

• e.g., if high WBC AND cough AND fever AND abn. CXR then likelihood of pneumonia is 4 out of 5

• Probabilistic/fuzzy– bayesian networks

• formal probabilistic reasoning, extension of decision analysis

– neural networks– fuzzy logic, genetic algorithms, case-based

reasoning, etc., or hybrids of these


Neural Networks• Finds a non-linear relationship between input parameters and

output state• Structure of network

– usually input, output, and 1-2 hidden fully connected layers

– each connection has a “weight”


Neural Network for MI Diagnosis

• Inputs (e.g., all patient characteristics in the EHR) • EKG findings (ST elevation, old Q’s)• rales (Yes, No)• JVD (in cm)

• Outputs are the set of possible outcomes/diagnoses

EKG findings

Rales

JVD

Response to TNG

Acute MI

No Acute MI


Training the Neural Network• Network gets “trained”

– feed network many examples of known patients and their diagnoses

• can handle missing data

– system iteratively adjusts the weights of the connections to find the network “pattern” that associates sets of input variables (patients) with the right output state (MI or not)

• Test network’s accuracy on another set of patients• In Baxt’s MI neural network

– training set: 130 pts with MI, 120 without– test set: 1070 UCSD ER patients with anterior chest pain


Baxt’s Acute MI Neural Net• Evaluation results: prevalence of MI 7% (Lancet, 1996)

• Results were driven by non-standard predictors– rales, jugular venous distention

• Why isn’t this neural network used more widely?– “black box” nature limits explanatory ability and lessens

acceptance– users have to input the variables manually

• interfacing to EHRs would increase adoption of DSSs

Sensitivity Specificity

Physicians 73.3% (63.3-83.3) 81.1% (78.7 – 83.5)Neural Net 96.0% (91.2 – 100) 96.0% (94.8 – 97.2)


Probabilistic Decision Support for Research

• Mostly used for eligibility determination– more adaptable than rule-based systems

• variable clinical presentations require many rules to handle contingencies

• what if data not available? fire a rule or not?

• Bayesian networks, neural networks, etc.– can handle missing data– adaptable to unanticipated presentations

• Several prototype Bayesian eligibility systems in the works…


Decision Support Methods Summary

• General limitations to clinical decision support– lack of formal, reproducible methods for making

clinical decisions – how to represent qualitative data (e.g., “looks sick”)

• Vast majority of clinically-used DSSs are rule-based systems, limited by– combinatorial explosion of rules

• Probabilistic approaches more “forgiving”, more “realistic”– can be computationally intractable


Outline




• Implications


Systematic Review of CDSS RCTs

• Only occasional modest benefit found (Hunt, JAMA 1998; updated RB Haynes 2000)– diagnosis: 1/5 studies beneficial– drug dosing: 9/15– preventive care reminders: 19/26

few studies looked at patient outcomes• 6 of 14 showed benefit

• Counted the number of systems in each category (e.g., drug dosing) that were “effective” (p>0.05)– but CDSS not all the same (apples and oranges)


Heterogeneity of CDSSs

• Preventive care reminder CDSS– A clerk routinely abstracts preventive care interventions

from paper chart into a database. Before each clinic session, nurse runs the CDSS for patients coming in that day. Guideline-based recommendations are printed out on paper and attached to front of chart. Doctor orders preventive care in usual way using paper-based methods.

• Hypertension treatment CDSS– Clinic has an EMR. During patient visit, CDSS notes that BP

and trend is too high. It checks patient’s Cr, diabetes status, cardiac status, current meds and allergies and recommends drug therapy change according to JNC VI guidelines. Presents e-prescription for MD to verify. If verified, order is sent directly to pharmacy and medication list updated.


CDSS Characteristics: Highlights• Reviewed and classified 42 RCT-evaluated systems• Tremendous variation in decision-maker/context,

how recommendation delivered, staff needed to make system run, complexity of recommended actions– 45% targeted to clinician, 55% patient, 5% both– 62% based on national guidelines or literature– 69% “pushed” recommendation to decision maker– 43% collected data directly from the EHR

• 45% required data input intermediary (11% MD), 58% of time requiring clinical knowledge

– 26% required an output intermediary


CDSS for Quality Summary• Current data suggests CDSSs can improve the

process of care and perhaps clinical outcomes– most effective at preventive care reminders– modest at best for drug dosing and active care– generally not helpful for improving diagnosis except

with trainees• Findings limited by

– methodological problems and design type of studies– insufficient appreciation of workflow component of

CDSSs– insufficient appreciation of heterogeneity of systems

• Bottom line: equivocal evidence, limited use


Outline




• Implications


Fundamental Barrier

• Better quality care <-- better decision support• Better decision support <-- “smarter” systems

– “know” more about the patient, evidence, context

• “Smarter” systems <-- more richly coded data– for EHR: SNOMED, standard EMR structure– for knowledge: coding, structures for guidelines, RCTs…

• Coded data <-- Coding of data entry• Coding of data entry <-- Greater physician time• Greater physician time --> no play --> no gain


Implications for Quality Improvement

• Clear trade-off between physician coding effort and “smarter” decision support

• Don’t expect more decision support than coding supports– generally successful decision support

• preventive care: age, last mammogram, etc.• allergies: Yes/No on specific drugs• drug dosing: weight, height, creatinine, age

– generally unsuccessful decision support• diagnostic assistance• complicated therapies (e.g., management of hypertension)

• Unrealistic to think of CDSSs as improving evidence-based practice in general


Implications for Research

• DSS possible with paper-based “charts”– narrow focus, well-defined rules, could help

standardize treatment by protocols– BUT requires double-data entry, workflow hassles

• DSS with EHR– ideal to use routinely collected EHR data and/or a

module “plugged in” to EHR– BUT requires interoperability of DSS with the EHR

• e.g., trial randomizing pts. to metformin or pioglitazone• exclusion rule #3: history of congestive heart failure


DSS Interoperability with EHR

itsa

coordcenter with DSS

ucsf.edu

LAN

Site 1/GE EHR Site 2/Epic EHR

1. Is C. Jones eligible for this trial?

2. …Exclusion Rule #3: Does C. Jones have a history of congestive heart failure?

3. Return Yes if “congestive heart failure” is in Past Medical History

HL-7 communications protocol

4. If Yes to history of CHF, C. Jones is not eligible


Summary on Decision Support

• Clinical care and research “decisions” similar• Different methods useful for different types of

decisions– using deterministic versus “fuzzy/probabilistic” reasoning

• Equivocal evidence for improving quality– limited by methodological and other shortcomings– workflow and organizational inputs generally

underappreciated• Fundamental trade-off between

– effort of coded data and quality (e.g., nuance) of decision support, and

– acceptability, effectiveness


Barriers• Near-term for clinical care

– knowing which decisions to support (e.g., preventive care reminders)

– data exchange among legacy systems– integrating decision support smoothly into EHR, incentives for

quality, etc. (3/15 class)• Near-term for research

– paper-based systems, requiring double-data entry• Longer term

– standard clinical vocabulary with adequate semantic coverage– much wider use of EHRs– efficient coding: of what, by whom, when, why– interoperation of EHRs with decision support systems– more explicit decision-making strategies for clinical care


Teaching Points• CDSSs have great promise for improving

quality/reducing error– but promise essentially not yet realized

• Much can be done today but only in limited settings

• “HAL”-like artificial intelligence not the main barrier

• Greater decision support for care and research requires– wider EHR use– richer, standard clinical vocabulary – better interoperability

february 28, 2006: i. sim decision support systems medical informatics – epi 206 ida sim, md, phd...

Documents

admin data

clinical data repository

collection of data

administrative data

research data infrastructure

garbage outincompatible

data setsone query

distinct typesclinical