Quality Quality ImprovementImprovementPitfalls and Pitfalls and

OpportunitiesOpportunitiesJames Hallenbeck, MDJames Hallenbeck, MD

ACOS/Extended CareACOS/Extended Care

VA Palo Alto HCSVA Palo Alto HCS

ObjectivesObjectives

Compare top-down and grassroots Compare top-down and grassroots strategies for quality improvement in strategies for quality improvement in terms of advantages and disadvantagesterms of advantages and disadvantages

Discuss different uses of quality Discuss different uses of quality monitorsmonitors Comparisons across groupsComparisons across groups Quality improvementQuality improvement

Better respond to queries regarding Better respond to queries regarding monitors, especially performance monitors, especially performance measures, especially MDSmeasures, especially MDS

Be able to…….

How this might be useful to How this might be useful to you…you…

Self defense - protecting you and Self defense - protecting you and your program when data does not your program when data does not accurately reflect the quality of care accurately reflect the quality of care you deliveryou deliver

Promote better use of monitors Promote better use of monitors (MDS and others) for quality (MDS and others) for quality improvementimprovement

Purposes of Quality Purposes of Quality Improvement Improvement

MonitorsMonitors ComparisonsComparisons

VISNs, facilities, wards, cliniciansVISNs, facilities, wards, clinicians Stimulus for quality improvement Stimulus for quality improvement

effortsefforts Benchmarking and ongoing Benchmarking and ongoing

evaluation of quality improvement evaluation of quality improvement effortsefforts

What Makes a Good What Makes a Good Monitor?Monitor?

(Top-down or grassroots)(Top-down or grassroots) Desired outcome strongly linked to Desired outcome strongly linked to

what is measured (i.e. minimally what is measured (i.e. minimally influenced by factors outside the health influenced by factors outside the health system’s control)system’s control)

System change, informed by monitor, System change, informed by monitor, would likely result in better outcomeswould likely result in better outcomes

Accuracy Accuracy Minimal false negatives & false positivesMinimal false negatives & false positives

QI INDICATORSQI INDICATORSTOP TO BOTTOMTOP TO BOTTOM

Enables comparisons Enables comparisons across large systemsacross large systems

Establishes Establishes organizational organizational prioritiespriorities

Buy-in from top Buy-in from top managementmanagement

If successful, results If successful, results in broad, measurable in broad, measurable improvementimprovement

Blind to local issuesBlind to local issues By attending to By attending to

certain priorities, certain priorities, others may be others may be ignoredignored

Lack of buy-in from Lack of buy-in from front-line stafffront-line staff

Need large numbersNeed large numbers Disempowerment at Disempowerment at

local levelslocal levels

Advantages:

Disadvantages:

What Makes a Good Top-Down What Makes a Good Top-Down Monitor?Monitor?

Simple, discrete dataSimple, discrete data Counts of thingsCounts of things

Minimal variation in local systems and Minimal variation in local systems and populationspopulations Comparing apples to applesComparing apples to apples

Large data numbers – in numerator and Large data numbers – in numerator and denominatordenominator

Data gathering, measurement and comparison Data gathering, measurement and comparison automatic automatic

Encourages GOOD clinical practice (not just Encourages GOOD clinical practice (not just designed to identify bad practice)designed to identify bad practice)

Examples Good Top-Down Examples Good Top-Down MonitorsMonitors

Flu vaccine administrationFlu vaccine administration

Appointment waiting timeAppointment waiting time

QIQIGrassrootsGrassroots

Tailored to local Tailored to local circumstancescircumstances

Better buy-in from Better buy-in from front-line stafffront-line staff

Can improve Can improve morale through morale through empowermentempowerment

Comparing Comparing outcomes across outcomes across systems difficultsystems difficult

Potential lack of Potential lack of buy-in from top buy-in from top managementmanagement

Harder to Harder to disseminate best disseminate best practicespractices

Politically, invisiblePolitically, invisible

Advantages Disadvantages

Examples Grassroots QIExamples Grassroots QI

Use of out-of-hospital DNR formsUse of out-of-hospital DNR forms

Addressing options of autopsy/organ Addressing options of autopsy/organ donationdonation

Apples to Apples Apples to Apples Problem…Problem…

Case-mix issuesCase-mix issues Severity of illness (Problematic: Severity of illness (Problematic:

pressure ulcers, weight loss, pressure ulcers, weight loss, dehydration in Hospice)dehydration in Hospice)

Gender (Problematic: UTI rates)Gender (Problematic: UTI rates) Depression/atypical neuroleptics Depression/atypical neuroleptics

(Problematic: geropsych units, hospice)(Problematic: geropsych units, hospice)

Vertical Hierarchy Vertical Hierarchy ProblemProblem

What is “true” at one level of a What is “true” at one level of a vertical hierarchy is not necessarily vertical hierarchy is not necessarily “true” at another (higher or lower) “true” at another (higher or lower) Example: Newtonian physics versus Example: Newtonian physics versus

quantum mechanicsquantum mechanics 30,000 feet view: reveals certain 30,000 feet view: reveals certain

patterns, otherwise invisible, but patterns, otherwise invisible, but obscures otherobscures other

Vertical HierarchyVertical Hierarchy

Nation

VISN 1 VISN 3VISN 2

Facility 1

Facility 2 Facility 3 Facility 4

Division/Ward

Division/Ward

Clinician + PatientClinician + Patient Clinician + Patient

Connection Between What Connection Between What is is MeasuredMeasured and Desired and Desired

Outcome ProblemOutcome Problem Many outcomes (good and bad) Many outcomes (good and bad)

minimally dependent on what we as minimally dependent on what we as clinicians doclinicians do

Examples:Examples: UTI rates UTI rates Fall ratesFall ratesImportant to tease out those aspects of the outcome that are

dependent on the healthcare system. Ideally, these aspects should be measured and worked on.

Environment

Luck

Healthcare System

Clinician Patient

Patient

Variables

Example: FallsExample: Falls Falls related to transfers/ambulation---18Falls related to transfers/ambulation---18 Falls related to toileting—13Falls related to toileting—13 Falls related to Falls related to

reaching/picking/bending—7reaching/picking/bending—7 Claimed did not fall/eased self to floor--5Claimed did not fall/eased self to floor--5 Slipped/tripped-2Slipped/tripped-2 Falls related to medical condition---2Falls related to medical condition---2 Falls related to inappropriate footwear-2Falls related to inappropriate footwear-2

1 month data: 47 falls

Can you tell from this data how these falls might be related to the healthcare system?

Numerator/Denominator Numerator/Denominator ProblemProblem

Percentages Percentages meaninglessmeaningless in isolation in isolation (numerator/denominator not given)(numerator/denominator not given)

Small numbers – 1 – 10 always Small numbers – 1 – 10 always suspicioussuspicious

Numerator problem: rare eventsNumerator problem: rare events Denominator problem:Denominator problem:

Restricted population based on:Restricted population based on: Geography (single ward)Geography (single ward) Time interval (monthly tracking)Time interval (monthly tracking)

977

27

50

6

1510

13

8

19

12

1312108

1712

11

20

1715

1011

7410

11

103

42

33

2328

36

26

172520

16

60

5359

56

71

63

5557

40

0

10

20

30

40

50

60

70

80

Oct

ober

Novem

ber

Decem

ber

Januar

y

Februar

y

Mar

chApril

May

June

July

Line 1

Line 2

Line 3

Line 4

TOTAL

Oct ‘04 - Jun ‘05 TotalFALLS REPORT

Coin-Toss QICoin-Toss QI

10 Facilities toss a coin 10 times10 Facilities toss a coin 10 times Heads is Good, Tails is BadHeads is Good, Tails is Bad True chance (within infinite tosses) True chance (within infinite tosses)

= 50%= 50%

Let’s compare facilities to “national” Let’s compare facilities to “national” datadata

Coin TossCoin Toss

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

1 2 3 4 5 6 7 8 9 10

PercentageHeads

Of 100 tosses 56/100 Heads = “National” average

“Problem”

Facilities“Exceptional” Facilities

ImplicationsImplications

Unadjusted percentages create a Unadjusted percentages create a bias in which small bias in which small

facilities/programs with small facilities/programs with small numbers to report for a given numbers to report for a given indicator are more likely to be indicator are more likely to be

classified at extremes (failing or classified at extremes (failing or exceptional)exceptional)

Data Validity ProblemData Validity Problem

Ideal:Ideal: AutomatedAutomated Unambiguous resultsUnambiguous results

Problematic:Problematic: Multiple people/services responsible for Multiple people/services responsible for

data entrydata entry Results dependent on subjective Results dependent on subjective

judgmentsjudgments Questionable inclusion/exclusion criteriaQuestionable inclusion/exclusion criteria

MDS ExamplesMDS Examples

Dehydration monitor:Dehydration monitor:

Little or no activity monitorLittle or no activity monitor

Bedfast Status monitorBedfast Status monitor

FY 05 Bed Fast Status

0.0%10.0%20.0%30.0%40.0%50.0%

Oct

ob

er

De

cem

b

er

Fe

bru

ary

Ap

ril

Jun

e

Blue = sub acute and hospice wards, Red = national

Prevalence of Little or No Activity

0.0%2.0%4.0%6.0%8.0%

10.0%12.0%14.0%

Palo Alto

Menlo Park

Livermore

VISN 21

National Average

DehydrationDehydrationSo, What’s the Problem So, What’s the Problem

Here?Here?VA Sierra Pacific Network (VISN 21)

QI#15 Prevalence of dehydrationFiscal Year 2005

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep

CCHCS NCHCS SFVAMC PIHCS SNHCS PAHCS VISN 21 National Average Target

MDS Quality Indicators:MDS Quality Indicators:What does the literature What does the literature

show?show? MDS tends to under-report problemsMDS tends to under-report problems

Example: depression: Example: depression: Differences in rates often reflect either Differences in rates often reflect either

differences in reporting or case mix – differences in reporting or case mix – NOT qualityNOT quality Example: High/Low Pressure Ulcer NH did Example: High/Low Pressure Ulcer NH did

not differ significantly in care processes not differ significantly in care processes (Bates-Jensen)(Bates-Jensen)

Little evidence that MDS indicators per Little evidence that MDS indicators per se useful in improving processes of care se useful in improving processes of care (Rantz)(Rantz)

VISN 21 National Average Target

First Quarter 0.0% 7.9% 4.8% 7.4% 11.0%

 

 

 

Second Quarter 0.0% 12.0% 3.3% 8.8% 11.0%

 

 

 

Third Quarter 3.9% 9.1% 0.0% 7.3% 10.7%

 

 

 

Fourth Quarter 0.0% 9.2% 0.0% 7.2% 10.4%

 

 

 

Prevalence of Depression Without Antidepressant Treatment

Month VISN 21 National Average Target

First Quarter 0.0% 3.7% 4.0% 3.6% 4.7%

 

 

 

Second Quarter 0.0% 5.0% 1.3% 3.4% 4.5%

 

 

 

Third Quarter 0.0% 2.9% 0.0% 2.1% 4.4%

 

 

 

Fourth Quarter 0.0% 3.5% 0.0% 2.2% 4.3%

 

 

 

Comparison to Published Comparison to Published RatesRates

Percentage of NH depressed on Percentage of NH depressed on MDS: ~ 10-36%MDS: ~ 10-36%

Gold standard comparisons: ~ 46-Gold standard comparisons: ~ 46-55% (Schnelle, Simmons) 55% (Schnelle, Simmons) “The MDS “The MDS depression quality indicator underestimates the prevalence depression quality indicator underestimates the prevalence of depressive symptoms in all homes, but in particular, of depressive symptoms in all homes, but in particular, among those reporting low or nonexistent rates”among those reporting low or nonexistent rates”

Percentage of depressed patients Percentage of depressed patients without antidepressant treatment: without antidepressant treatment: 45% (Brown)45% (Brown) Note: not

validated against pharmacy database

Suggestions for Using MDS Suggestions for Using MDS (and similar) Monitors for (and similar) Monitors for

QIQI Consider the good intentConsider the good intent of the monitor of the monitor

It would be bad to have patients starving or It would be bad to have patients starving or thirsty, unnecessarily incontinent, subject to thirsty, unnecessarily incontinent, subject to polypharmcy, inappropriately treated with polypharmcy, inappropriately treated with antipsychotics, who never get out of bed and antipsychotics, who never get out of bed and are bored to tearsare bored to tears

Step BackStep Back from the individual and the from the individual and the percentages and ask, percentages and ask, “what part of this “what part of this outcome might we have some influence on?outcome might we have some influence on?

Brainstorm with staffBrainstorm with staff

Suggestions for Using MDS Suggestions for Using MDS (and similar) Monitors for (and similar) Monitors for

QIQI Consider feasibility of possible Consider feasibility of possible

interventionsinterventions

Benchmark interventionsBenchmark interventions

Note changes following interventionNote changes following intervention

Revise interventionRevise intervention

Example: FallsExample: Falls Transfers/ambulation:Transfers/ambulation:

Wheelchair breaks working and routinely inspected on Wheelchair breaks working and routinely inspected on patients’ wheelchairs?patients’ wheelchairs?

ToiletingToileting Nightlights available, sleepers, patient instructions to Nightlights available, sleepers, patient instructions to

call for assistancecall for assistance Slipped/trippedSlipped/tripped

Inspect rugs, walkways especially during rainy/snowy Inspect rugs, walkways especially during rainy/snowy seasonsseasons

Falls related to medical conditionFalls related to medical condition Review drug therapy for high-risk patientsReview drug therapy for high-risk patients

Inappropriate footwearInappropriate footwear Inspect patients’ footwear, none-slip socks and slippersInspect patients’ footwear, none-slip socks and slippers

SummarySummary

Top-down monitors by themselves Top-down monitors by themselves WILL NOT result in improved care WILL NOT result in improved care deliverydelivery

Top-down monitors work best when Top-down monitors work best when combined with grass-roots efforts to combined with grass-roots efforts to improve careimprove care

Where monitors work poorly for a given Where monitors work poorly for a given purpose, it is our obligation to say so – purpose, it is our obligation to say so – and then work to make things betterand then work to make things better