bonnie: an electronic clinical quality measure testing tool · bonnie is a clinical quality measure...

Approved for Public Release; Distribution Unlimited. 17-0467 © 2017, The MITRE Corporation. All Rights Reserved.

C M S A L L I A N C E T O M O D E R N I Z E H E A L T H C A R EFebruary 2017

Bonnie:An Electronic Clinical Quality Measure Testing Tool

Lizzie CharbonneauThe MITRE Corporation


What is Bonnie?

Bonnie is a clinical quality measure testing tool that allows:

– Loading electronic Clinical Quality Measures (eCQM) from the Health Quality Measure Format (HQMF) format

– Exploring the behavior and complexity characteristics of clinical quality measures

– Rapidly building synthetic patients using data elements defined as part of the measure definition

– Testing synthetic patients against existing and updated versions of a clinical quality measure

– Creating synthetic patients for use in certification with Cypress

– Sharing synthetic patient records with other Bonnie users using the patient bank

2

Approved for Public Release; Distribution Unlimited. 17-0467 © 2017, The MITRE Corporation. All Rights Reserved. 3

Why Bonnie?

Prior to Bonnie, the only way to test electronic clinical quality measures (eCQM) was by hand.

– eCQMs are large and complex.

– Checking by hand is time consuming and error prone.

Before Bonnie was in use, 381 logic errors were reported by implementers. Each measure had at least one issue.

Bonnie was introduced to enable automated testing of eCQMsprior to publication.

After Bonnie was introduced, the number of reported errors dropped by 95%.


History

First released April 2014

Used extensively for testing 2015 and 2016 annual update measures– 2015

Over 5,000 synthetic test patient records, 80% logic coverage

– 2016 Over 10,000 synthetic test patient records, 100% logic coverage

Currently being used for testing 2017 annual update measures

– Over 16,000 synthetic test patient records– Striving for 100% coverage of the logic

National Quality Forum (NQF) recommends use of Bonnie to support NQF eCQM endorsement

4


Bonnie Usage over Time

5

5,000

10,000

15,000


Users of Bonnie

Measure Developers

– As part of the EHR Incentive Program, Quality Payment Program, and other CMS Quality Reporting Programs (e.g., IQR, CPC+)

– NQF recommends use of Bonnie for NQF endorsement

EHR Vendors

– Get head start on integrating new measures into their systems

– Perform calculation validation

6


Bonnie Screenshot

7


Bonnie Screenshot (continued)

8


Bonnie Is under Active Development

Current development work includes:

– Support for Clinical Quality Language (CQL)

Expect all Bonnie functionality including new features to be updated to support CQL

Recently completed development work includes:

– Measure History

– Patient Dashboard

– Value Set Debugging

– Excel Patient Export

– Bonnie Application Programming Interface (API)

– Quality Data Model (QDM) 4.3 Update

9


Recent Features

10

API

Value Set Debugging

Excel Patient Export

Bonnie API


New Feature: Measure History

When measures are updated between annual update cycles, it can be difficult to keep track of the nuanced changes between measure versions.

Patient calculations often change across measure versions. It can be difficult to recall how a patient previously calculated and how they’re expected to calculate with the new measure version.

New measure history feature includes:

– A view of patient calculations across measure versions

– A view of how measure logic has changed across versions

– A detailed view of patient calculations across measure versions

This feature is currently in production.

11


Measure History (continued)

12



13



14



15


New Feature: Patient Dashboard

User feedback indicated that patient test deck planning and editing is easier to do within the context of other patients.

New patient dashboard feature includes:

– Showing patient calculation results in the context of other patients

Includes aggregate results and individual logic clause results

– Enabling inline editing of patient metadata (name, expected values, birthdate and deathdate, etc.)

– Allowing users to easily access Patient Builder editor

This feature is currently in production.

16


Patient Dashboard (continued)

17



18



19


Support for the Clinical Quality Language (CQL)

The CQL is a new standard for trial use (STU) for describing eCQMs.

CQL logic is expected to eventually replace existing QDM logic.

Regression testing during the transition will be vital to ensure that measures continue to convey the same clinical intent.

Bonnie will support testing of both CQL-based and QDM-based measures during the transition.

The functionality that Bonnie implements will leverage the extensive existing open source CQL implementation.

Initial CQL support is available on the Bonnie staging server: https://bonnie.ahrqstg.org/

20

https://bonnie.ahrqstg.org/


Support for CQL (continued)

21


Support for CQL (continued)

Upcoming Feature Improvements for CQL

– More readable logic display

– Logic highlighting

– More granular coverage and complexity calculations

– Patient Dashboard support for CQL

– Excel Export support for CQL

– Measure History support for CQL

22


Bonnie User Group

Interaction with our user community is vital as we

– Seek feedback for new features under development

– Determine the future direction of the application

– Share news about updates and changes

The online forum for the Users Group is

– https://groups.google.com/forum/#!forum/bonnie

23

https://groups.google.com/forum/#!forum/bonnie


Where to Look for Info

Main Bonnie site

– https://bonnie.healthit.gov/

JIRA for tracking issues

– https://oncprojectracking.healthit.gov/support/projects/BONNIE

Contact development team

– [email protected]

Open Source on GitHub

– https://github.com/projecttacoma/bonnie

24

https://bonnie.healthit.gov/

https://oncprojectracking.healthit.gov/support/projects/BONNIE

mailto:[email protected]

https://github.com/projecttacoma/bonnie


Questions?

25


Backup Slides

26


Clinical Quality Measures

Health Care Reports

Measure provider delivery of standard of care to patients

Electronic specification: eCQM

Combine logical statements with clinical concepts

27


Patient Populations

28


History

The process of Measure Development has been grueling:

– Does the measure logic capture clinical intent?

– Lots of subtleties: timing, operator precedence, etc.

– Hundreds or thousands of lines of logic

– Every release is manually reviewed

29


Final Notes

Open source

– http://bonnie.healthit.gov/

30

http://bonnie.healthit.gov/

bonnie: an electronic clinical quality measure testing tool · bonnie is a clinical quality measure...

Documents