4/17/2014

1

Improvement Science In Action

A Run Through the Run Chart:

Using Run Charts for Improvement

Sandra Murray

May 1st, 2014

Session Objectives

By the end of this session you will

be able to:

Identify what a run chart is

Know when to start a run chart

Be able to interpret a run chart

Identify several keys to effective ways to display data

with a run chart

Be aware of cautions and special uses for run charts.

DG p. 67

4/17/2014

2

What are we trying to accomplish?

How will we know that a change is an improvement?

What change can we make that will result in improvement?

Model for Improvement

Act Plan

Study Do

Used with permission: Improvement Guide Improvement Guide, Jossey-Bass, 2009

Going

Deeper

Run Chart

Graphical display of data plotted in some type of

order. Also has been called a time series or a

trend chart.

DG p. 67

4/17/2014

3

We Can Learn a Lot with a Simple

Tool…

What do you observe here? What would your question

be?

DG P. 68

What are Classic Uses of the Run

Chart?

Displaying data to make process performance visible.

Determining whether a change resulted in improvement

Determining whether gains made are sustained

DG p. 68

4/17/2014

4

DG p. 70

4/17/2014

5

4/17/2014

6

Determining the Median

4/17/2014

7

Analyzing a Run Chart Starts with a simple visual analysis

They are testing changes here. Do they have

improvement yet? Charts speak for themselves.

Analysis is by degree of belief

4/17/2014

8

Unplanned Returns to OR: Pilot Population

0

1

2

3

4

5

6

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26

%

Months

Pre-Procedural Briefings

Razors to Clippers

Bleeding risk assessment, DVT Proph

Beta Blocker use,

Normothermia

(N~200/Mo.) Good

Prophylactic ABX Timing

Improvement Projects Require a Family of Measures

• 2-8 measures typically -Each on a graph -All viewed on one page

DG p. 74

Fig 3.6: Improvement Evident Using a Set of Run Charts Viewed on One Page

4/17/2014

9

Small Multiples

Multiple run charts viewed on one page

All these run charts are about the same measure but for

a different location, provider or segment of the

population

Each has the same scale vertically and horizontally

Allows for rapid comparison

DG p. 75

4/17/2014

10

Average

Waiting Times:

All Primary Care

Clinics in VHA

System

Small Multiples:

Overall System

and 22 Districts

May Display More Than One

Measure on a Graph

DG p. 75

4/17/2014

11

May Use Different Measure for

Each Axis

DG p. 76

Sometimes We Don’t Have Much Data

May not be rich in data but that data may still lead to a high degree

of belief in the change(s) tested

Characterize the change by describing the before and after medians

Minimizes point-to-point variation

DG p. 77

4/17/2014

12

Probability Based Rules for Run Chart Analysis

If visual analysis leaves us uncertain that change(s)

yielded improvement we may use probability based rules

to analyze the run chart

DG p. 77

DG p. 78

4/17/2014

13

Rule 1: Shift

Six or more consecutive POINTS either all above or all below the median. Skip values on the median and continue counting points. Values on the median DO NOT make or break a shift.

Median=10 Median=11

0

5

10

15

20

25

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25Me

as

ure

or

Ch

ara

cte

ris

tic

Rule 1

Median 10

Rule 2: Trend

0

5

10

15

20

25

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25

Me

as

ure

or

Ch

ara

cte

ris

tic

Rule 2

Median 11

• Five points all going up or all going down. If the value

of two or more successive points is the same count the

first one then ignore the identical points when counting;

like values do not make or break a trend.

4/17/2014

14

Rule 3:Runs

– A run is a series of points in a row on one side of the median. Some points fall right on the median, which makes it hard to decide which run these points belong to.

– So, an easy way to determine the number of runs is to count the number of times the data line crosses the median and add one.

– Statistically significant change signaled by too few or too many runs.

DG p. 81

Rule 3: # of Runs Table for Checking for Too Many or Too Few Runs on a Run Chart

Total number of data

points on the run chart

that do not fall on the

median

Lower limit for the number of runs

(< than this number of runs is “too few”)

Upper limit for the number of runs

(> than this number of runs is “too many”)

10 3 9

11 3 10

12 3 11

13 4 11

14 4 12

15 5 12

16 5 13

17 5 13

18 6 14

19 6 15

20 6 16

21 7 16

22 7 17

23 7 17

24 8 18

25 8 18

Table is based on about a 5% risk of failing the run test for random patterns of data. Frieda S. Swed and Churchill Eisenhart,

(1943). “Tables for Testing Randomness of Grouping in a Sequence of Alternatives. Annals of Mathematical

Statistics. Vol. XIV, pp.66 and 87, Tables II and III

DG p. 80

4/17/2014

15

Rule 3: Runs

– Too many runs

– Not usually indicative of improvement

– Most likely data that needs to be stratified or another data collection or display problem

DG p. 81

RULE 4

For detecting unusually large or small numbers:

Data that is Blatantly Obvious as a different value

Everyone studying the chart agrees that it is unusual

Remember: – Every data set will have a high and a low - this does not mean the

high or low are astronomical

0

5

10

15

20

25

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

Mea

sure

men

t or

Cha

ract

eris

tic

Rule 4

4/17/2014

16

What is Our Job?

When we see signal:

– Go learn and take appropriate action

– Is the signal an improvement?

– If so how keep it, spread it?

– Is signal bad news?

– If so, how get rid of it?

– How do we prevent it?

Exercise 1: Analyze This Chart

%

Percent Incomplete Revised Trauma Score in Record14.8 13.0 13.6 14.0 13.4 13.3 13.8 13.7 14.0 14.4 13.1 14.1 14.8 13.4 13.7 14.1 12.9 9.7 10.9 11.6 10.5 10.3 9.8 10.2 9.9%

Median = 13.4

Test 1

Test 2

Test 3

Test 4

Implementation

Jun 07 J A S O N DJan 08 F M A M J J A S O N DJan 09 F M A M J

8

10

12

14

16

18

Percent Unplanned Readmissions

4/17/2014

17

Exercise 2: Analyze This Chart

0

10

20

30

40

50

60

1 2 3 4 5 6 7 8 9 10 11 12 13

% Reenrolling W/O a Gap

Median = 19

Weeks

Ad campaign

started

Exercise 3: Analyze This Chart

0

5

10

15

20

25

30

35

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

# Referrals to BH Services

#

Weeks

Median =

4/17/2014

18

Exercise 4: Analyze This Chart

Fundamental Uses of Run Charts

Display data to make process performance visible

Determine whether a change resulted in evidence of

improvement

Determine whether we are holding the gain made by our

improvement

4/17/2014

19

DG p. 86

When Do We Start a Run Chart?

DG p. 87

4/17/2014

20

Proper Use of the Median

When should we apply a median?

– Will depend on your situation

– If very little data baseline median may be only a few data points

– If want to apply probability-based rules for analysis of run chart

need 10 data points for median

– If graph shows no signals (shift, trend, runs astronomical) and

median made from 10 or more data points freeze and extend

median into the future

– This will result in earliest possible detection of signals

DG p. 87

•If median not frozen and extended will result in

delayed detection of signals

DG p. 89

4/17/2014

21

•A signal is detected utilizing both original and extended

median

DG p. 89

•If a signal is detected and sustained a new median may be

created for the new process performance

-When analyzing run chart with two separate medians rules are

must be applied separately to the data surrounding each median

DG p. 90

4/17/2014

22

Plotting Rare Events

Problems:

– Results in too many zeros

– Makes interpretation difficult and chart of little

value

– Useful alternative is to chart time or workload

between undesirable events

– Up is always good for these charts

DG p. 91

DG p. 92

4/17/2014

23

DG p. 92

DG p. 94

4/17/2014

24

Cautions with Percent or Rate Data

on Run Chart Run chart not sensitive to size of denominator

– If denominator unusually small or large compared to other

denominators used for that chart may result in misleading chart

– For an effective run cart denominators for each data point should

be +/- 25% of average denominator size

Specific types of Shewhart control charts will cope with

unequal denominator sizes

DG p. 93

DG p. 94

4/17/2014

25

Data Line Guidance

May connect data points with a line if data in

time order

If data not in time order do not connect data

points

DG p. 95

Trend Lines on Run Charts

Place only if detect signal on run chart

DG p. 97

4/17/2014

26

No signal of improvement on chart

Improper to use trend line

DG p. 98

Stratification

The separation and classification of data according to

selected variables or factors

– Unit, patient type, clinic, service, age group, diagnosis, etc.

My give us clues as to where to focus or ideas for

change to test

DG p. 99

4/17/2014

27

DG p. 100