1

Reporting and Learning from Health IT-Related Events Toward Safer Healthcare

Session 166, Wednesday, February 13, 2019

Yang Gong, MD, PhD, UTHealth

2

Yang Gong, MD, PhD

Has no real or apparent conflicts of interest to report.

Conflict of Interest

3

• Introduction (~5 minutes)

– Moderator

– Speaker

• Presentation (~45 minutes)

– Scan QD code for details

• Q&A (~10 minutes)

• Extended discussion (Posterior to the session)

– Twitter @gngyng

– LinkedIn GONGYANG@gmail.com

Agenda

4

• State the benefits and risks of HIT for patient

safety/healthcare quality improvement

• Identify the barriers of reporting and analyzing HIT events

and challenges for turning HIT event reports into actionable

knowledge

• Discuss how data representation and knowledge

management in FDA MAUDE incident reports can facilitate

quality improvement towards a better and safer healthcare

system

Learning Objectives

5

Patient Safety - Pressures and Incentives

Medical error. (Makary & Daniel, 2016)

Leveraging patient safety research: Fifteen-year

efforts since “To Err Is Human” (Liang, Miao, Kang… &

Gong, submitted manuscript, 2018)

6

IndividualReportsIndividualReports

• Clinicians

• Patients

Aggregated DataAggregated Data

• Hospitals

• PSO

AnalysisAnalysis

• Hospitals

• PSO

ActionableKnowledgeActionableKnowledge

• Clinicians

• Hospitals

Resolving ProblemsResolving Problems

• Clinicians

• Hospitals

Improving Safety through Reporting

• Patient Safety Event (PSE)Reporting Initiatives

• To Err is Human (2000)

• Patient Safety and QualityImprovement Act (2005)

• Patient Safety Organizations (PSO)

• Why Reporting

• Learn from lessons

• Reporting Mechanism States working with PSO by 2017

7

Perceived Barriers

• Lack of instructions and training

• Unsatisfactory usability of

classifications/taxonomies

– Lengthy reporting forms

• Time-consuming

– Competing with other priorities

• Lack of motives

– No feedback

– Observed event seemed “trivial”

» A trivial tip --> a large ‘iceberg’ under water

8

Challenges in Event Reporting

• Structured data vs. narratives

• Structured data: standardized but limited representations

• Narratives: flexible, content-rich, causal and temporal information

• Complexity of medication error reports

8

Multi-Stage

Multi-Personnel

Multi-Factor

Reporting

Analysis

Actions

Learning

Barriers

9

Problem in Reporting– Low Reporting Quality

Data Type Ideality Reality

Structured data Easy to interpret Rarely answered

Unstructured data Supplementary Info. Essential Info.

10

Problem in Reporting Errors- Lacking Analysis Tools

• A New Report: “RN removed 100mcg fentanyl from the

omnicell in endo room 2 documented on the anesthesia

sheet that he gave 25mcg, no waste recorded. 75mcg

fentanly unaccounted for.”

Historical Reports & SolutionsReported, so what?

11

• Institute of Medicine (IOM)– HIT plays an imperative role in saving healthcare cost, improving patient outcomes,

decreasing occurrence of medication errors, and refining healthcare process measures across diverse settings

• Agency for Healthcare Research and Quality (AHRQ) – use of information and communication technology in healthcare to support the delivery of

patient or population care or to support patient self-management

– Synonym

• electronic health records (EHR) and

• EHR components

– computerized provider order entry (CPOE) or

– clinical decision support system (CDSS)

– Generalized Health IT also includes

• administrative or practice management systems

• automated dispensing systems

• laboratory information systems, and

• diagnostic imaging systems

• Nowadays, Health IT has become an integral part of healthcare and has been widely applied to collect, transmit, process, display, and store patient data

What is Health IT

12

• Listed in the top 10 technology-related hazards– new uncertainties and risks for patient safety

– disrupting established work patterns

– creating new risks in practice, and

– encouraging workarounds

• The adoption of EHR has revealed potential safety implications related to

– EHR design, implementation, and use related to

• technological features of EHR

• users and workflows

• organizations, rules and regulations

Challenges of Health IT

13

• Understand and manage the risks– Sociotechnical context

– Sittig and Singh 8-dimensional sociotechnical model

• Tool in patient safety studies

• Complexities of technology

• Users in workflow and external or organizational policies

• Health IT in event reporting

– Citied as one of contributing factors in reporting systems

– No health IT exclusive sources to patient safety studies

– AHRQ Common Formats (CFs): Common definitions and reporting formats (QD code linked)

• Hospitals, Community pharmacies, Nursing homes

Meeting the Challenges

14

Goal

• Develop a user-centered, knowledge-based reporting and learning system

– Help healthcare practitioners better report events

– Connect with relevant reports

– Learn how to address causes of errors

– Improve the behavior at work

15

Method –Filter on Structured Data

• 4,947,220 reports from MAUDE 2008-2016

• Keyword searching on Generic Name and Manufacturer Name

• Identify HIT-related events from sample reports through domain expert review

• 6 inclusion criteria

• 4 exclusion criteria

• Assess reviewer consistency by Cohen’s kappa

Workflow of reviewing a report from FDA MAUDE database

16

Method– Classifier on Unstructured Data

• Term frequency (TF) – inverse document frequency (IDF)

• Biterm topic model

• Extract the semantic themes (topics) from a corpus of short documents

• Classifiers

• Random forest, Logistic regression, Naïve Bayes, SVM, J48, JRip

• Gold standard:

• HIT-related and non-HIT event reports identified by domain experts

17

Result– Keyword-based HIT Event Filter

Generic Names Manufacturer Names Total

Inclusion Keywords 94 38 132

Exclusion Keywords 21 0 21

Year

Raw reports Filtered reports

ReportsHCFA/Manufactur

er/DistributorReports

HCFA/Manufactur

er/Distributor

2008 145,598 9,148 1,817 146

2009 201,996 9,906 2,640 214

2010 327,961 10,792 3,434 316

2011 414,083 12,597 2,371 307

2012 520,043 12,952 4,825 308

2013 636,145 12,516 3,551 313

2014 867,451 12,927 4,338 380

2015 965,240 15,762 17,963 384

2016 868,703 15,023 4,685 408

Sum 4,947,220 45,624

• Inclusion & exclusion keywords

• Raw and filtered reports of 2008-2016 MAUDE database

18

Result – Contributing Factor Distribution Analysis

19

Result - FDA Medical Device Event Reports

• FDA Manufacturer and User Facility Device Experience (MAUDE) database

• ~ 6,000,000 events involving medical devices

• structured & unstructured data fields

• 0.46% ~0.69% are HIT-related events

– Up to 50,000 HIT-related events (QD code linked) Trend of MAUDE reports (bars) and MAUDE

related publications (line) since 2000

Filter on Structured

data

Classifier on

Unstructured dataRaw Data HIT

20

Result – An Integrated Model of HIT event Identification

0.4~0.9% 50% 97%Proportion of

HIT events

Filter on

Structured

data

Classifier on

Unstructured

dataRaw Data HIT

Trade off between precision and recall Grow the HIT database

21

Underway– Prototype An Integrated Reporting and Shared

Learning System in Healthcare Community

22

Predictive Text Entry

• To support reporting

– Cueing list, auto-suggestion

• By two-group randomized test

– Improved text generation

– Improved data consistency and quality

Entered and

tagged-in text

Initial letters

of input

Auto-suggestion:

matched text

entry hits

(# of hits <=10)

Narrative data entry field equipped with text prediction functions

EF

GC

B

Main component lists multiple-choice questions in slide-in mode

Cueing list that reminds

of the content or content

categories of reportable data

A

D

C

Structured Data Entry – 13 MCQs and four of them have narrative fields as illustrated as the part B

Unstructured Data Entry – One narrative comment field

C: Cueing List

aids in data entry of

specified single-text

field

(B) in the structured

question, or

comment field

G: Auto-suggestion

Suggesting the words, phrases

and sentence in the context to

describe the event details

23

Managing PSE Knowledge

• Ontology

– Interoperability among

• home-grown systems

• patient safety organization (PSO) systems

– Data integration

• Organizing prevailing classifications

– Decision making

24

Data Source & Annotation• One year data from a PSO

institute (2016)

• 2,576 medication error reports (including adverse drug reaction, ADR)

• Incidents & near misses

• Unstructured data

• Manual annotating

• Two patient safety domain experts with pharmacy or clinical backgrounds

• Followed the NCC MERP taxonomy for medication errors, plus ADR

Error

Stages

(6)

Error

Stages

(6)

Error

Types

(8)

Error

Types

(8)

Error

Causes

(5)

Error

Causes

(5)

Administering

Dispensing

Med. Rec.

Monitoring

Ordering

Transcribing

ADR

Billing Issue

Missing Dose

Wrong Admin.

Wrong Document.

Wrong Dose

Wrong Drug

Wrong Time

Devices (HIT)

Information Deficit

Pathophysiological Factor

Performance Deficit

Others

PSO Reports

25

Pipeline– Identifying Three Key Factors

Remove punctuation

Remove number

Rainbow stop word

N-grams tokenizer

TF-IDF

26

Pipeline– Similarity Measurement

Raw Reports

Similarity

Calculation

Grouped ReportsLabels:

• Originating stage

• Type

• Cause

27

Data Elements in Medication Error

28

Leveraging Patient Safety

• Information models act as the core

• Knowledge base plays an central role in the knowledge support

• Transforming reports into actionable knowledge

29

Knowledge Support

• Identify similar cases based on query– Web M&M (PSNet)

– Patient Safety Organization (PSO) data

– Data from home-grown system

• Provide solutions and suggestions

30

Prototype

31

Innovative Design

Current Frames• Reports are stored entry by entry

• Reporters learn nothing

• No feedback for systems

Proposed Frames• Reports are annotated on the same feature tree

• Provide solutions for reporters

• The system can learn from user feedback and preferences

v

Feedback / Preferences

Knowledge Support (e.g., Solutions)

32

Identifying Relevant Cases

33

Exploring Event Connections

PSE Space

Topic Space

?

Topic Model

34

Developing a PSE Knowledge Base

Topics

Reports

Solutions

A PSE Knowledge Base

35

Acknowledgement

• Funding

– Agency for Healthcare Research & Quality, 1R01HS022895

– UTHealth Innovation in Cancer Prevention Research Training Program (Cancer Prevention and Research Institute of Texas grant RP#160015)

– University of Texas System Grants Program #156374

• Current lab members

– Hong Kang, PhD

– Pei-Yin Yang, MS

AlumniChen Liang, PhD

Ju Wang, PhD

Sicheng Zhou, MS

Bin Yao, MD, MS

Qi Miao, BS

Hsing-yi Song, MD, MS

Xinshuo Wu, MD, MS

Swananda Pandit, MS

Lei Hua, PhD

James Richardson, MS

Zhijian Luan, MS

Yanyan Shen, MHA

Rajitha Gopidi, MHA

Dan Wang, PhD

Mathew Koelling, MHA

CPRIT Summer intern

Cindy Songting Wu

Frank Wang

Elisa Ali

Melanie Klock

Ethan Wang

CollaboratorsJing Wang, PhD, RN

Nnaemeka Okafor, MD, MS

Hua Xu, PhD

Tina Hilmas RN, BSN

Becky Miller MHA

Amy Vogelsmeier, PhD, RN

36

• THANK YOU!

• LinkedIn: gongyang@gmail.com

• Twitter: gngyng

• Please complete online session evaluation

Questions