systematic review of patient safety indicators (psi ... · systematic review of patient safety...

IUMSP Institut universitaire de médecine sociale et préventive, Lausanne

Dr Jean-Marie Januel, PhD, MPH, RN

Systematic review of Patient Safety

Indicators (PSI) Validation Studies

&

Focus on PSI 12

OECD Health Care Quality Indicators

Patient Safety Subgroup Meeting

Paris, France, May 9, 2012

Institute of social and Preventive Medicine (IUMSP),

Lausanne University Hospital, Switzerland.

Department of Medical Information, Health Evaluation and Clinical Research,

Hospices Civils of Lyon, France.


2

SUMMARY

I. Systematic review of PSI validation studies

o Methods of the review and meta-analysis

o Pooled estimate of PPV

o Impact of the “Present on Admission” code

o Conclusion

II. Validating and comparing PSI 12

o A Pilot Validation Study in France for Assessing the

Positive Predictive Value of PSI 12

o International Project of PSI 12 Comparison Against

an External Benchmark


3

Part I.

Systematic Review of

Patient Safety Indicators

(PSI) Validation Studies


4

Research strategy

2001 to 2011 (= 11 full years)

o PubMed

o EMBASE

Algorithm used

[“Patient Safety Indicator” OR “Patient safety Indicators”]

OR

[(“PSI” OR “PSIs”) AND (“AHRQ” OR “Agency for

Healthcare Research and Quality”)]


5

Inclusions / Analysis

Studies concerning PSI in adults

Principal objective to validate PSI (outcome

measured)

o POSITIVE PREDICITVE VALUE (PPV)

o Impact of Present on Admission (PoA) code using

pooled Rate Ratio (RR) of the proportion of PSI without

PoA divided by the proportion of PSI with PoA (as the

risk to “over-indentify” PSI without PoA code)

Statistical analysis

o Random effects across studies

o Pooled estimates and Test of variance using I2


6

Descriptive results

153 articles focused on PSI

Including

o 131 Original studies

o 13 Reviews

o 9 Commentaries

17 studies that estimated PPV

5 studies that showed impact of “Present

on Admission” (PoA) code


7


8

Pooled PPV for PSI in the literature

Pooled estimate of PPV Heterogeneity

PSI Label N-studies % (95% CI) I²(%) P-value

PSI 5 - Foreign body left during procedure 1 45% (35% - 56%) na na

PSI 6 - Iatrogenic pneumothorax 2 76% (72% - 81%) 0.0% 0.369

PSI 7 - Vascular catheter related bloodstream infection 2 46% (30% - 63%) 87.4% 0.005

PSI 9 - Postoperative hemorrhage or hematoma 1 75% (66% - 83%) na na

PSI 10 - Postoperative physiologic or metabolic derangement 2 60% (53% - 68%) 3.7% 0.308

PSI 11 - Postoperative respiratory failure 3 76% (67% - 84%) 88.9% <0.001

PSI 12 - Postoperative pulmonary embolism and deep vein thrombosis

9 50% (39% - 61%) 97.1% <0.001

PSI 13 - Postoperative sepsis 2 49% (41% - 57%) 21.0% 0.261

PSI 14 postoperative wound dehiscence 2 79% (63% - 94%) 92.1% <0.001

PSI 15 - Technical difficulty / Accidental punction or laceration during procedure

3 85% (77% - 93%) 76.5% 0.014

No significant heterogeneity Significant heterogeneity


9

Pooled PPV for PSI 12

NOTE: Weights are from random effects analysis

Overall (I2 = 97.1%, P = 0.000)

STUDY

6. Henderson et al., 2009

3. White et al., 2009

7. Henderson et al., 2009

9. Zhan et al., 2007

8. Zhan et al., 2007

2. Romano et al., 2009



1. Kaafarini et al., 2011

49.97 (38.65-61.29)

PPV (95% CI)

54.46 (44.78-63.90)

63.64 (54.40-72.19)

50.00 (40.40-59.60)

62.65 (58.55-66.63)

29.22 (24.20-34.65)

21.99 (16.93-27.76)

62.65 (58.55-66.63)

62.39 (57.74-66.87)

42.86 (33.55-52.55)

100.00

Weight %

10.73

10.84

10.72

11.45

11.34

11.32

11.45

11.40

10.74

0 0 25 50 75 100


10

Summary of Meta-Analyses for assessing the impact

of the Present on Admission (PoA) code

NOTE: Weights are from random effects analysis

.

.

.

.

.

.

.

PSI 1 - Complication of anesthesia

(I2 = 82.0%, P = 0.019)

PSI 8 - Postop. hip fracture

(I2 = 0.0%, P = 0.825)

PSI 9 - Postop. hemorrhage or hematoma

(I2 = 0.0%, P = 1.000)

PSI 10 - Postop. physiologic or metabolic derangement

(I2 = 58.6%, P = 0.089)

PSI 11 - Postop. respiratory failure

(I2 = 22.7%, P = 0.274)

PSI 12 - Postop. PE/DVT

(I2 = 90.7%, P < 0.001)

PSI 13 - Postop. sepsis

(I2 = 0.0%, P = 0.727)

PSI

4.04 (0.32-51.21)

4.87 (1.25-19.01)

1.15 (1.04-1.28)

1.59 (1.10-2.29)

1.22 (1.07-1.40)

1.80 (1.43-2.25)

1.23 (1.05-1.45)

RR (95% CI)

30 / 44'520

12 / 25'748

745 / 125'201

275 / 67'617

714 / 65'836

2'586 / 209'159

327 / 38'540

PSI / Sample

without PoA

11 / 44'497

2 / 25'193

646 / 125'015

195 / 67'436

578 / 65'606

1'528 / 207'126

265 / 38'444

PSI / Sample

with PoA

1 1 2 3 4 5 10 50


11

CONCLUSIONS – Part I

PSI validation is based on PPV in most

instances

Differences between studies were

important for PPV

Significant impact of PoA code that

should be taken into account in future

works and ICD development


12

Part II.

Focus on the PSI 12


13

IIa. A Pilot Validation Study in

France for Assessing Positive

Predictive Value of the PSI 12


14

Assessing PSI 12 PPV - Pilot Study

Positive Predictive Value of the PSI 12 (Postoperative Pulmonary Embolism and Deep Vein Thrombosis)

154 inpatients at the University Hospital of

Lyon, France with identified PSI numerator

cases from DRG database in 2008

Stratified analysis by subgroups = type of

surgery

Chart Review of Medical Record

PSI + PSI -

DR

G PSI + True Positive Cases False Positive Cases

PSI - False Positive Cases True Positive Cases


15

PSI 12 PPV and False Positive Cases

Patients

with PSI+

True Positive

Cases

False Positive Cases

Total < Admission

Confusion DVT / PE

Absence of

Event

N (%) N PPV (%) N (%) N (%)* N (%)* N (%)*

Total VTE 154 (100.0) 123 (79.9) 31 (20.1) 7 (22.6) 5 (16.1) 19 (61.3)

TYPE OF SURGERY

Hip Arthroplasty 43 (27.9) 38 (88.4) 5 (11.6) 1 (20.0) 1 (20.0) 3 (60.0)

Knee Arthroplasty 58 (37.7) 56 (96.6) 2 (3.4) 0 (0.0) 1 (50.0) 1 (50.0)

# Femur 19 (12.3) 15 (79.0) 4 (21.0) 2 (50.0) 2 (50.0) 0 (0.0)

Other Orthopaedic 13 (8.4) 8 (61.5) 5 (38.5) 3 (60.0) 1 (20.0) 1 (20.0)

Digestive Surgery 13 (8.4) 6 (46.2) 7 (53.8) 0 (0.0) 0 (0.0) 7 (100)

Thoracic Surgery 8 (5.2) 0 (0.0) 8 (100) 0 (0.0) 0 (0.0) 8 (100)

* Based on False Positive Cases


16

This pilot study allowed to…

Estimate PPV using refined PSI 12

(denominator focused on HIP and KNEE

inpatients only)

Analyse data from multiple hospital

centers in France (CLARTE Project)

o 80 hospitals (representative of all types of

hospitals, nationwide geographic distribution)

o 550 inpatients with positive PSI 12


17

IIb. International Project of

PSI 12 Comparison Against an

External Benchmark in Patients

undergoing Hip and Knee

Arthroplasty


18

What is good value for comparison?

Zero Risk ?

… But studies on healthcare adverse events show

that 45% of those are avoidable…

Average mean of providers are compared ?

… but it’s a theorical value that doesn’t take into

account possibilities to reduce the risk according to

improve healthcare toward “state of art” practice…

External value according “state of art”

healthcare practice (recommendations,

guidelines)


19

The Proposal / Framework conceptual

Enlargement of our previous project of PSI 12 using

Swiss, French and Canadian data (JM Januel presentations

at IMECCHI meetings in Calgary on May 12th-13th, 2008; and

in Vancouver on November 9th-10th, 2009)

To develop an external independent benchmark that takes

into account “state of the art” healthcare practice

(Januel JM et al. JAMA 2012; 307(3): 297-303.)

To compare adjusted occurrence rate of symptomatic

postoperative VTE (PE and DVT respectively) during

patient hospital stay (i.e., before discharge) from several

countries using PSI 12 and this external benchmark


20

Population setting

Data for years 2008 and 2009

(excepted in France, 2007 and 2008, because

change in PDx coding rule on April 1st, 2009)

HIP and KNEE arthroplasty inpatients

(surgical procedure codes using country specific

classifications)

Stratified analysis for HIP and KNEE

arthroplasty independently

o Total and Partial Prosthesis, Prosthesis replacement


21

ANALYSIS

Calculate PSI 12 (= outcome measurement)

Develop hierarchical models using data from

each country independently

(Two-level logistic regression models)

Assess differences in practices used to

diagnose DVT based on the proportion of use of

lower extremity ultrasound by providers in each

country (procedure codes)


22

Januel JM, et al. JAMA 2012;307(3): 294-303


23

Pooled Occurrence Rates of VTE

(Januel et al. JAMA 2012)

TPHA TPKA

% (95% CI) I² P % (95% CI) I² P

LMWH (Observ.) 0.83 (0.19-1.48) 67.3% 0.230 1.98 (0.69-3.27) 84.0% <0.001

LMWH (RCT) 0.51 (0.26-0.76) 45.4% 0.010 1.24 (0.87-1.60) 36.7% 0.060

Direct Directe IIa/Xa Inhibitors (RCT)

0.31 (0.03-0.59) 32.8% 0.070 0.81 (0.54-1.07) 42.5% 0.047

Indirect IIa/Xa Inhibitors (RCT)

0.68 (0.26-0.97) 0.0% 0.380 0.77 (0.00-1.65) - -

TOTAL 0.53 (0.35-0.70) 49.4% <0.001 1.09 (0.85-1.33) 55.7% <0.001


24

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

1.6

1.8

2.0

Ad

juste

d o

ccu

ren

ce r

ate

s,

% w

ith

95%

CI

(Weig

thed

on

pro

vid

er

pro

ced

ure

case v

olu

me)

Procedure case volume by provider annually

Upper limit = 0.70%

Lower limit = 0.35%

<100 [100-200[ [200-400[ ≥400 All Providers

Mean = 0.53%

France Switzerland


25

Patient’s factors associated to VTE (PSI 12)

Swiss national data (2008-2009) French national data (2007-2008)

INPATIENT LEVEL FACTORS OR (95% CI) P OR (95% CI) P

Sex (F) 1.03 (0.74 - 1.45) 0.854 1.41 (1.31 - 1.51) <0.001

Age (≥65 yrs) 0.86 (0.57 - 1.29) 0.460 1.53 (1.39 - 1.68) <0.001

LOS (Log) 2.18 (1.57 - 3.02) 0.000 2.28 (2.11 - 2.46) <0.001

Hypertension 0.70 (0.44 - 1.12) 0.134 1.02 (0.94 - 1.10) 0.651

Arrhythmias 0.87 (0.49 - 1.55) 0.635 1.03 (0.93 - 1.15) 0.539

Valve disorder 0.78 (0.37 - 1.65) 0.508 0.71 (0.58 - 0.85) <0.001

Diabetes (uncomplicated) 0.61 (0.27 - 1.41) 0.249 0.77 (0.68 - 0.88) <0.001

Diabetes (complicated) 0.18 (0.02 - 1.68) 0.132 0.87 (0.66 - 1.16) 0.351

Renal failure 0.30 (0.14 - 0.65) 0.002 0.74 (0.60 - 0.91) 0.005

Rheumatic disease 0.35 (0.04 - 3.04) 0.339 0.79 (0.53 - 1.17) 0.235

Obesity 0.60 (0.27 - 1.34) 0.213 1.21 (1.05 - 1.39) 0.007

Cancer 0.88 (0.38 - 2.02) 0.757 1.14 (0.95 - 1.37) 0.154

Dementia 1.05 (0.56 - 1.99) 0.878 0.67 (0.56 - 0.79) <0.001

Death 7.37 (4.05 - 13.41) <0.001 2.62 (2.18 - 3.14) <0.001

Mean number of SDx coded 1.25 (1.18 - 1.31) <0.001 1.09 (1.08 - 1.11) <0.001


26

Provider’s factors associated to VTE (PSI 12)

Swiss national data (2008-2009) French national data (2007-2008)

HOSPITAL LEVEL FACTORS OR (95% CI) P OR (95% CI) P

Ultrasound of lower extremity

None (reference) 1.00 1.00

<20% of inpatients 1.56 (0.95 - 2.54) 0.076 1.25 (1.05 - 1.49) 0.010

[20% - 80%[ NA NA NA 3.68 (2.92 - 4.64) <0.001

≥80% of inpatients NA NA NA 4.17 (3.25 - 5.36) <0.001

Procedure case volume

<100 annually (reference) 1.00 1.00

[100 - 200[ 0.68 (0.38 - 1.22) 0.193 1.07 (0.90 - 1.26) 0.438

[200 - 400[ 0.74 (0.41 - 1.34) 0.322 1.14 (0.94 - 1.38) 0.169

≥400 0.62 (0.30 - 1.27) 0.189 1.25 (0.95 - 1.64) 0.110


27

A new method to validate and compare data

from several providers / countries

Major study to assess international comparison

against evidence based benchmark

o Symptomatic event of VTE (including PE and DVT)

o Before hospital discharge (i.e., available data)

o Against benchmark that reflect best updated practice

(“state of the art”)

Specific exposed population (HIP / KNEE

arthroplasty) using PSI 12 outcome measurement

o Patients with high risk of postoperative VTE

o Subgroups with good PPV (>85% in pilot study in

France)


28

Using complex but performing models

Models take account of important parameters

related to potential biases (across hospitals in

each country, between countries)

o Heterogeneity of measuring events due to different

diagnostic methods available to assess deep vein

thrombosis (In particular the use of more or less

systematic examination of the lower limbs using

ultrasound, according the fact that both symptomatic

and asymptomatic DVT are identified in these cases)

o Proposal to include a questionnaire to collect

information on habits, coding rules and healthcare

practice in participating countries


29

Rationality and Pertinence of the Benchmark

Need to assess healthcare quality using a

patient safety outcome measurement

o Events occuring during hospitalization stays

(between surgery and discharge)

o Directly linked to adherence and observance of the

recommended antithrombotic prophylaxis

implemented for patients

o Symptomatic events = usual healthcare practice


30

Limitations

Missing outcome registration for events that

occur after discharge

o Near 50% of VTE taken into account

o Difficulty to attribute post discharge events to

healthcare defaults or organizational problems in

hospitals

(e.g., responsibility of the patient for the observance

and the adherence of the prophylactic treatment at

his home)


31

CONCLUSIONS – Part II

To refine the PSI 12 denominator according

to specific sub-groups with high risk of VTE

and potential good PSI 12 PPV is the good

way for the near future (patients undergoing

HIP and KNEE arthroplasty)

Refinements needed to separate DVT and PE

according to the potential differences related

to diagnostic assessment of events (use of

lower extremity ultrasound)


32

Contact: [email protected]

systematic review of patient safety indicators (psi ... · systematic review of patient safety...

Documents