VA HSR&D Salt LakeConsortium forHealthcareInformaticsResearch

CHIR MRSA

Brad Doebbeling, MD, MSc; Jennifer Garvin, PhD; Mahesh Merchant, PhD; Mike Rubin, MD; Rick Martinello, MD; Mary Goldstein, MD; Phil Foulis, MD; Steve Luther, PhD; Pradeep Mutalik, PhD; Matt Scotch, PhD, Samson Tu, MS; Susana Martins, MD; Jeff Friedlin, DO; Katalina Gullans, MA

Indianapolis, IN; Salt Lake City, UT; Palo Alto, CA; West Haven, CT; Tampa, FL.

Project Origin

Healthcare-associated infections (HAIs) leading cause of preventable death.

VA Pittsburgh Reduced MRSA Significantly!

VHA National MRSA Reduction Initiative

HA Surveillance Time Intensive

Staff need to Focus on Problems, Intervene Early

Need for Automated Classification of MRSA Status

Interdisciplinary, multisite collaborative extracting structured and unstructured data from electronic health records (EHR)

(VHA, METHICILLIN-RESISTANT STAPHYLOCOCCUS AUREUS (MRSA) INITIATIVE, http://www1.va.gov/vhapublications/ViewPublication.asp?pub_ID=1525. 2007.)

CHIR MRSA Project Specific Aims

Aim 1: Develop, review and refine an ontology for clinically- and epidemiologically-relevant concepts to enable detection of MRSA for reporting purposes and gather requirements for developing a MRSA surveillance and reporting tool

Aim 2: Index MRSA-related Concepts in Clinical Narrative

Aim 3: Clinical Inference and Analysis of MRSA-Related Information Contained in the Medical Record

Aim 4: Develop and evaluate a prototype surveillance application that uses automatically processed VA electronic health record data

Translational Aim: Evaluate algorithms for making multi-type predictions based on heterogeneous data, using MRSA as a clinical domain

Research Questions/Hypotheses

Developing an ontology for clinically- & epidemiologically-relevant concepts enables detection of MRSA for reporting purposes

Concepts from practice guidelines, interviews, clinical narrative, and machine learning help operationalize use cases

Developing an automated MRSA surveillance system is as reliable and valid as human reviewers

Methods

1. Identify relevant documents; 2. De-identify documents; 3. Determine document quality; 4. Build a clinical and population-based ontology;5. Annotate documents; 6. Extract and define MRSA relevant concepts; and 7. Determine clinical relevance and perform inference

modeling to identify linguistic elements assoc. with MRSA.

Microbiology completes a microbiology test for all patients that tested NEGATIVE

at admit.

Microbiology completes a microbiology test for all patients that tested NEGATIVE

at admit.

ALL positive MRSA are reported to Diana and sent to BUGS (printer)

ALL positive MRSA are reported to Diana and sent to BUGS (printer)

ADMITS ADMITS

DISCHARGEDISCHARGE

Ordered for ALL patients admitted to the hospital.

Ordered for ALL patients admitted to the hospital.

Nasal Nares DNA Swab (2 swabs taken for each patient)

Nasal Nares DNA Swab (2 swabs taken for each patient)

Swabs are run by molecular tech once daily. Swabs are processed at 6am – and results are available to unit by 2:30pm.

Swabs are run by molecular tech once daily. Swabs are processed at 6am – and results are available to unit by 2:30pm.

Patients that are swabbed after 8am on a given-day, their tests are not run until the following

day.

Patients that are swabbed after 8am on a given-day, their tests are not run until the following

day.

Capacity to run 96 swabs.Indy averages 40-50 day

Capacity to run 96 swabs.Indy averages 40-50 day

Positive MRSA phone call to MRSA coordinator and to unit. Results inputted in to DHCP. (if positive, 2nd swab is placed in salt broth and saved for growth. Saved

for files)

Positive MRSA phone call to MRSA coordinator and to unit. Results inputted in to DHCP. (if positive, 2nd swab is placed in salt broth and saved for growth. Saved

for files)

Tests only detect “colonization.” There are no tests run for “infection” or

determinations made regarding colonization v infection in molecular

pathology.

Tests only detect “colonization.” There are no tests run for “infection” or

determinations made regarding colonization v infection in molecular

pathology.

MRSA swabs at discharge are to capture “conversions” only (what patients

contracted MRSA during the hospital stay).

MRSA swabs at discharge are to capture “conversions” only (what patients

contracted MRSA during the hospital stay).

Discharge swab is tied to physician

orders. Physicians could use a tool that is “physician

timed discharge” for

ordering swab – this

not used frequently.

Discharge swab is tied to physician

orders. Physicians could use a tool that is “physician

timed discharge” for

ordering swab – this

not used frequently.

TRANSFERSTRANSFERS

Only patients that tested NEGATIVE at admit are swabbed at transfer

Only patients that tested NEGATIVE at admit are swabbed at transfer

Swabs are ordered by the physician in the transferring unit. Receiving unit ‘should’

also order a swab. Indy working to automate this process.

Swabs are ordered by the physician in the transferring unit. Receiving unit ‘should’

also order a swab. Indy working to automate this process.

Molecular swab testing process completed in same fashion as admit process.

Molecular swab testing process completed in same fashion as admit process.

If a swab is ordered at transfer for a patient that tested

POSITIVE at admit, the molecular tech will cancel the order.

If a swab is ordered at transfer for a patient that tested

POSITIVE at admit, the molecular tech will cancel the order.

“swabbing technique” molecular tech monitors the occurrence of PCR inconclusive

results. Inconclusive results are run again the next day. If still inconclusive, a resubmit order

is issued.

“swabbing technique” molecular tech monitors the occurrence of PCR inconclusive

results. Inconclusive results are run again the next day. If still inconclusive, a resubmit order

is issued.

Units that have a high rate of PCR inconclusive results are given training on swabbing

techniques.

Units that have a high rate of PCR inconclusive results are given training on swabbing

techniques.

ER and Psych do NOT do nasal swabs.

ER and Psych do NOT do nasal swabs.

Patients go to their unit/ward while awaiting swab results.

Patients go to their unit/ward while awaiting swab results.

Patients with a MRSA warning/flag within the last YEAR are placed in isolation pending swab

results.

Patients with a MRSA warning/flag within the last YEAR are placed in isolation pending swab

results.

Positive MRSA patient kept in isolation until cleared (negative swab).

Positive MRSA patient kept in isolation until cleared (negative swab).

Molecular biology might

have two swabs for

same patient at transfer.

Molecular biology might

have two swabs for

same patient at transfer.

Conversion count is

attributed to the last unit

that swabbed.

Conversion count is

attributed to the last unit

that swabbed.

Determination of

Colonization v.

Infection??

Machine Learning

MRSA Pipeline Diagram

Use Cases

Clinical Relevance &

Inference Modeling

Information Extraction

Quality of Documents

De-Identification

Building Ontology

Identify Relevant

DocumentsData

Extraction

*Workflow Data

Collected(Observation

&Interview)

Annotation of Documents

Relevant Publications

Ontology

Natural Language Processing

Feedback provided at each step to previous and all other steps affectedby changes

Documents Ready for NLP

*Selected Workflow Data Used

MRSA Ontology Annotation-inform schema and link free text

to ontology concepts A source of terms (including synonyms) for

natural language processing (NLP)

…Patient 123456789… MRSR…

mouth..

MRSR ontologyin Protégé

Protégé API

MRSA terms &

relationships for NLP

Patient 123456789MRSA has anatomical locationOral cavity structure

NLP engine

Goldstein, Tu, Martin @ Palo Alto

Uses of MRSA Ontology

A vocabulary for queries and inference

…Patient 123456789… MRSR…

mouth..

NLP engine

MRSA-related

dataVISTA data

Inference engines

Rules/Influence

Diagrams…

MRSA surveillance application

Queries

Ontology

Results MRSA Pipeline Model Ontology Development Sampling Plan & Approvals Use Cases Draft Annotation Schema Development of system-Name: Automatic Classification

of MRSA (ACOM): This NLP system will classify patients who have a positive

MRSA culture into those who have an infection and those who are colonized using data from the VHA EMR system.

This will provide assistance to the Infection Preventionist (actor).

The trigger for this system will be the positive culture or PCR for MRSA (note that this will trigger isolation procedures).

Interim Conclusions Communication & FTF meetings key to effective

progress & collaboration. First MRSA ontology developed with a clinical,

healthcare or population emphasis. Ontology will be used for clinical decision support

combining concepts identified in NLP process with other structured data to enhance MRSA surveillance.

Will help differentiate between patients colonized from those infected and provide timely information for multiple clinical and decision support purposes.

Plan create a state-of-the-art surveillance tool that will help reduce MRSA infections in VHA.

Leadership Plans for FY2011 Foster stronger communication between CHIR MRSA

subgroups and CHIR projects VA CHIR-MRSA Use Case Interview conducted at six VA

facilities Development of annotation timeline Publication and manuscript development Work with document quality on CHIR MRSA research aims Connecting the CHIR MRSA pipeline – how the subgroups

inter-relate Provide leadership and direction for subgroups to attain

‘next steps’ and receive deliverables from other teams to accomplish goals and research aims

Test annotation schema with data uploaded to VINCI Analysis of use case interview data – that will inform the

use case

Goals/Expectations for Impact

Milestones Expected Date

Ontology of clinically, epidemiologically relevant concepts that enhance MRSA reporting and surveillance

9/30/10

Development of use cases, incorporation of machine learning

12/30/10

MRSA documents for annotation, sharing, modeling on VINCI

6/30/11

NLP System to extract MRSA related concepts from VA clinical documents

6/30/12

Prototype MRSA surveillance application that uses structured and unstructured data sources

9/30/2013

Backup Slides

Key Workflows Details of Accomplishments Detailed Plans for Subgroups for

FY2011

Ontology Plans for FY2011

Work with annotation team to apply ontology to annotation schema (iterative process).

Incorporation of Drug Terms into the ontology. Mapping of terms to NDF-RT.

Ontology Team works with NLP group to facilitate integration. (iterative process).

Through domain expert meetings, explore possibilities of collaboration with the Clinical Inference and Modeling project.

Preparation of manuscript describing the MRSA ontology for a clinical journal.

Machine Learning Plans for FY2011

Create shared work space on VINCI virtual environment to interact with Tampa team

Enhance versions of GATE, cTakes, and YTEX pipelines to support MRSA use cases

Explore methods to identify relevant documents in positive cases and develop methods to use them in identifying specific infections

Extract a set of age-matched control documents and use them to hone the NLP rules for classification

NLP Plans for FY2011

Create shared work space on VINCI virtual environment to interact with Tampa team

Enhance versions of GATE, cTakes, and YTEX pipelines to support MRSA use cases

Explore methods to identify relevant documents in positive cases and develop methods to use them in identifying specific infections

Extract a set of age-matched control documents and use them to hone the NLP rules for classification

Project Management Received approval for NDS data access for 51 CHIR MRSA team members Received Indianapolis IRB approval for data extraction of local patient data that

will be uploaded to VINCI. Additionally, received local IRB to cover all CHIR MRSA members to access and analyze the Indianapolis patient data that will be stored on VINCI.

Led development and refinement of the Use Case Questionnaire. Received Indianapolis IRB approval to conduct all interviews from Indianapolis. Conducted initial Use Case Questionnaire interviews with 4 key informants.

Scheduled interviews with MICU physician and SICU nurse. Planned Face-to-Face Meeting held in Salt Lake City, April 18 – 20th, 2010 Participated in the development of the MRSA Working Use Case and Workflow

Analysis AMIA Abstracts; review and edits of all abstracts submitted from CHIR MRSA

subgroups Lead weekly CHIR MRSA team calls; responsible for agenda development,

minutes, and follow-up Interface with CHIR MRSA Annotation subgroup, Ontology subgroup, Machine

Learning subgroup, and NLP subgroup Development of patient use cases utilizing de-identified Indianapolis data for

annotation schema testing Developed CHIR MRSA pipeline Presenter in the CHIR MRSA WIP

Ontology Accomplishments Scientific Literature and MRSA Surveillance Guidelines Identified. Literature and

Guidelines discussed with MRSA team and domain experts. Selected Literature posted to Sharepoint.

Identification of Formal MRSA Terms from Literature and Guidelines. Terms added to the ontology in Protégé Environment.

Ontology team identified standards to use for concepts in ontology in collaboration with standards team.

Addition of SNOMED code to ontology concepts. Terms in Sample MRSA reports from Indianapolis integrated into MRSA ontology. Synonyms added to concepts in ontology using ULMS. Meetings with MRSA clinical domain experts to review and refine MRSA Ontology

Terms. Experts are provided Protégé Access to MRSA ontology. MRSA Ontology Version 1* handed off to MRSA NLP team. Ontology team provides

guidance to NLP team to integrate Protégé output with GATE. Ontology refined to reflect the latest concepts/terms/synonyms as ontology team is

informed. Version 2* to Annotation Team for examination and annotation strategy

development. Ontology sent to domain experts, refined to include only MRSA relevant concepts

(current version has terms related to other health associated infections). Potential MRSA relevant drug list (antibiotics, etc) sent to domain experts for review.

Explore possibilities of integration with Clinical Inference and Modeling project.

Machine Learning Accomplishments

Domain Expertise: Pathology and Laboratory Scientific literature and MRSA surveillance guidelines identified for MRSA Ontology sub-group.

Discussed literature and guidelines with other domain experts and MRSA ontology subgroup team members. Emailed literature to group. Sent Tampa VA MRSA surveillance documents to both Ontology sub-group and larger MRSA project group. Documents posted to SharePoint.

Identification of formal MRSA terms from literature and guidelines. Meetings held with MRSA Ontology Subgroup to review and refine MRSA Ontology Terms. Reviewed ontology from MRSA Ontology Subgroup to refine scope of ontology. Provided ongoing consultation to MRSA Ontology sub-group on Pathology and Laboratory testing and

reporting related to MRSA. MRSA Staff Interviews Assisted with development of interview questionnaire for infection control staff. Piloted MRSA interview questionnaire. Consulted with project staff regarding feasibility and logistics of conducting face-to-face or telephone

interviews with infection control staff. Machine Learning Defined use cases for machine learning of Tampa VA MRSA surveillance data. Presented to MRSA Project

Team. Identified and selected 1,200 cases from Tampa VA MRSA surveillance database for data analysis. Identified and extracted 855,244 progress notes from selected cases for use in machine learning

analysis. Used lab test dates and administrative data to reduce number of progress notes for analysis to 98,889. Identified progress note types likely to contain MRSA-related information; Presented note types to MRSA

Project Team. Conducted post-processing of machine learning results to facilitate interpretability. Provided a preliminary report with results of machine learning analyses to MRSA Project Team and MRSA

Ontology Sub-group. Shared case selection and data extraction methodologies to assist with local data extraction efforts at

other project sites. Emailed to Indianapolis project management staff. Presented to MRSA project team.

NLP Accomplishments Successfully imported the MRSA Ontology into GATE NLP. Created a preliminary NLP pipeline for information extraction including the use of

processing resources such as logic rules and gazetteers. Published preliminary pipeline as an executable on MRSA Sharepoint Site. Developed UIMA/cTakes pipeline for NLP of MRSA documents. Created YTEX, a version of cTakes, for machine learning classification of free text

documents. Identified and selected 200 cases from West Haven VA MRSA surveillance

database for data analysis. Identified and extracted 17,000 progress notes from selected cases for use in

machine learning analysis. These notes are within a specified time window of a positive culture.

Domain Expertise: Infection Prevention Scientific literature and MRSA surveillance guidelines identified for MRSA

Ontology sub-group. Discussed literature and guidelines with other domain experts and MRSA ontology subgroup team members.

Identification of formal MRSA terms from literature and guidelines. Meetings held with MRSA Ontology Subgroup to review and refine MRSA Ontology

Terms. Reviewed ontology from MRSA Ontology Subgroup to refine scope of ontology. Provided ongoing consultation to MRSA Ontology sub-group on Infection

Prevention