DEVELOPMENT OF QUALITY MEASURES AND DATA DICTIONARYTowards National Definitions and Agreed StandardsDr Peter Jones MSc EBHC (Oxon) FACEM

With thanks to Dr Alana Harper FACEM and Dr James LeFevre FACEM

MOH Forum, Wellington 5/5/2014

Why Bother? Each ED could just

decide what to measure and how to measure it

Problems Time Resources Skills Duplication (x26) Accuracy Comparisons

‘Admission’=3hrs?

To maximise the return for effort in a resource constrained environment

Aim of this presentation

How to operationalise the MOH suite of quality measures using a real world example Data definitions / development of a ‘Data Dictionary’ Data collection for process and clinical outcomes

My QI Background Auckland City Hospital / ADHB

Morbidity / Mortality 2000-05 Time to Thrombolysis KPI 2000-05 ED Ultrasound Credentialing 2000-03 Procedures Database 2000-05 AHQAS 2001 Cardiac Arrest Documentation 2000-2010

UHCW NHS Trust Audit Lead 2005-06

ACEM Quality Management Subcommittee 2011-current

SOPH PhD Student: Best measure of ED Overcrowding?

Shorter Stays in ED National Research Project ADHB/ SOPH Auckland University 2009-current

Health Policy, Effective Practice, Epidemiology, Māori Health, Health Economics, Biostatistics, Clinicians

HRC funded 10-588 MREC approved MEC 10/06/060 Multi-stream Mixed-Methods Research

What was done to implement the SSED target? What effect on markers of care? Lessons for future health/public service policy

Kaupapa Māori Research Approach

SSED NRP Stream 2 Quality of Care Quantitative analysis

13 ‘Quality’ Measures Process and Outcome

Nationwide (routine flow data) 4 Case Study Sites (clinical markers)

Richness of information Target results Q1 2010

Any difference 2006-08 vs 2010-12? Adjust for Ethnicity / Age / Deprivation

SSED NRP Quality Outcomes

Primary (Nationwide) ED LOS

Access Block (wait for admission >8hr in ED) Overcrowding

Hospital LOS Re-attendance rates within 48 hours of discharge Re-admission rates within 28 days of discharge

SSED NRP Quality Outcomes

Secondary Mortality (N)

hospital inpatients for ED attenders at 10, 30 and 90 days

Time to treatment in acute asthma (4 sites) Time to reperfusion for myocardial infarction (1 site) Time to theatre for fractured neck of femur (1 site) Time to appendectomy for acute appendicitis (1 site) Time to antibiotics for severe infections (1 site) Proportion of patients who leave without being seen

(N) ‘Gaming’ the target (N)

a spike of ED discharges at or near the target time (N) digit bias in recording time of ED discharge (N) re-designation of ED patients to ‘stop the clock’ (CS ?N)

SSED NRP Quality Clinical Markers

Clinical Quality Markers Selection Literature review / Evidence Search Reference Group Meeting December 2010

SSED NRP Quality Clinical Markers Clinical Quality Markers Selection

Critical Appraisal of Quality Indicators (QICA)

SSED NRP Quality Data Required

Two Sources, three types of Information NZHIS

Clinical diagnosis data from 4 case sites (ICD) DHBs

Routinely collected process dataED and acute direct inpatient admissions (PIMS)

Hospital Bed OccupancyCensus at night (Bed Management System)

SSED NRP QualityData Collection Plan

7yrs Presentations

Site SpecificClinical Indicators

Process Indicators

SSED NRP Quality Data Dictionary

SSED QualityData Dictionary Data elements piloted in consultation

with analysts from three DHBs Different names for same process e.g.

Episode=Event=Visit=Case PIMS different

And NZHIS No time stamped process data Highlighted inconsistent submission from DHBs

SSED NRP QualityData Dictionary Then consulted all 20 DHBs

Different PIMS collected data in different ways Not all data elements currently defined Some ED & Inpatient systems not integrated

Level of data capture differs Capture patient event separately Not all Process measures captured by all DHBs

SSED NRPData Dictionary

Dictionary Facilitated Data Cleansing Convert different DHB data format to unified set Identify outliers and validate Link NZHIS data with DHB data

NNPAC/NMDS Identify duplicates from DHB data DHB data has elements missing from NNPAC/NMDS

Jan

Feb

Mar

Apr

May Jun

Jul

Aug

Sep Oct

Nov

Dec Jan

Feb

Mar

Apr

May Jun

Jul

Aug

Sep Oct

Nov

Dec Jan

Feb

Mar

Apr

May Jun

Jul

Aug

Sep Oct

Nov

Dec Jan

Feb

Mar

Apr

May Jun

Jul

Aug

Sep Oct

Nov

Dec Jan

Feb

Mar

Apr

May Jun

Jul

Aug

Sep Oct

Nov

Dec Jan

Feb

Mar

Apr

May Jun

2006 2007 2008 2009 2010 2011

4

5

6

7

8

9

10

11

12

0

100

200

300

400

500

600

700

800

900

1000

Mean ED LOSData Quality versus true outliers

Mean ED LOS (Hours)

Count of ED LOS > 24 Hours

TIme

LOS

(Hou

rs)

Num

ber o

f Pat

ient

s Pe

r Mon

th

Jan

Feb

Mar

Apr

May Jun

Jul

Aug

Sep Oct

Nov

Dec Jan

Feb

Mar

Apr

May Jun

Jul

Aug

Sep Oct

Nov

Dec Jan

Feb

Mar

Apr

May Jun

Jul

Aug

Sep Oct

Nov

Dec Jan

Feb

Mar

Apr

May Jun

Jul

Aug

Sep Oct

Nov

Dec Jan

Feb

Mar

Apr

May Jun

Jul

Aug

Sep Oct

Nov

Dec Jan

Feb

Mar

Apr

May Jun

2006 2007 2008 2009 2010 2011

4

5

6

7

8

9

10

11

12

0

100

200

300

400

500

600

700

800

900

1000

Mean ED LOSData Quality versus true outliers

Mean ED LOS Outliers Fixed

Count of ED LOS > 24 Hours

TIme

LOS

(Hou

rs)

Num

ber o

f Pat

ient

s Pe

r Mon

th

SSED Indicator Selection and Data Dictionary Resources Required 24 months work 0.5 FTE Emergency Medicine Specialist 1 FTE Research Fellow 1FTE Data Manager Office / Dedicated desk space PCs (high spec / dual screen) Software (reference management / pdf writer / PIMS) Online Journal Access Monthly team meetings Academic and Administrative support Liaison with NZHIS and all DHB IS departments

SSED NRP Clinical QualityData Collection

Site SpecificClinical Indicators

• List of ICD codes to NZHISJ45 (0,1,8,9); J46

• Events with that ICD code in each time period for each site

Date / NHI / Demographics

• Random sample events

• List of NHIs to the sites• Manual collection of data

Trained senior clinician data collectorsmultiple site visits

• Data accuracy / cleaning

SSED NRP Clinical QualityData Collection Tools

‘Intelligent’ spreadsheets Data validation checks / protected formulas

SSED NRP Clinical Quality Data Cleansing Data Accuracy

≈15% independently checked Eligibility 283/297 = 97% Primary outcome 366/387 = 95% Comorbidities1198/1288 = 93%

Data Cleaned Errors corrected where possible No imputation for missing data

Time Stamp Data Also Required Cleaning

SSED Data Collection Resources Required

12 months work 2-3 sets of records per hour for clinical quality indicators

0.5 FTE Emergency Medicine Specialist 1 FTE Research Fellow 1FTE Data Manager Office / Dedicated desk space PCs (high spec / dual screen) Software (data collection tools) Clinical records departments

Desk / PC / Laptop

Development of Quality Measures and Data Dictionary: What’s Needed? MOH

Data definitions / dictionary IT at each site standardised and audited Standardised ‘intelligent’ Tools for data collection

Web vs local DHB

Dedicated resources for quality Space / Time / Admin / Clinical Records People (thank you NZMC)

More than lip-service!

Development of Quality Measures and Data Dictionary

Towards National Definitions and Agreed Standards

Questions / Discussion

SSED NRP Stream 2 Quality Method: Clinical Markers

Eight conditions identifiedRepresent whole systemPilot 50 sets notesElectronic DEF with logic (reduce errors)Clinically important difference aprioriSample size 90% power, alpha 0.05 (2 tailed)

SSED NRP Stream 2 Quality Method: Clinical Markers

Sample Size Calculations Not all outcomes could be measured at all sites

10000 sets of notes = not feasible Clinical records departments / our resources

Quality Indicator Critical Appraisal Tool 2 Authors independently appraised the indicators Outcome brought to whole team Different views Compromise

Desire to measure >1 outcome and reflect whole system

1 outcome all sites; 1 other outcome each site

SSED NRP Stream 2 Quality Method: Clinical Markers Asthma (30 min difference time to steroid)

Common / relevant / important All ages / Mäori / Pacific Data available / accessible / sample size manageable all sites

Acute Myocardial Infarction (15 min difference time to lysis) Evidence strong but practice changed over time:1 site

Sepsis (60 min difference time to steroid) Evidence moderate / important / relevant / all ages / Mäori / Pacific Sample size issues + difficult data collection: 1 site

Appendectomy (12hr difference time to theatre) Whole system (balance)

Fracture NOF (6hr difference time to theatre) Whole system (balance)